How Do Clinical Competency Committees Assess Resident Performance on the Internal Medicine Milestones

Clinical competency committees (CCCs) play a critical role in determining the developmental progress of residents. This study examines how CCCs synthesize assessment data to make judgments about residents’ clinical performance. Beginning in 2014, CCC faculty completed questionnaires about the CCC process after each six -month reporting period to the Accreditation Council of Graduate Medical Education (ACGME). Subsequently, a focus group interview was conducted. Qualitative data was analyzed using the constant comparative method, consistent with a grounded theory approach. Milestones ratings were analyzed for two ACGME reporting cycles (n=102 categorical residents). Seven CCC members provided qualitative data for four ACGME reporting cycles. Three major themes were identified: 1) “data-input issues” (i.e., problems with the data or lack thereof); 2) “CCC data processing issues” (i.e., factors influencing ratings and decision-making processes) and 3) “data output issues” (i.e., how CCC generated milestones ratings are used). CCC members weighted rotation evaluation scores highest (weight=37%), followed by comments (weight=27%), and personal experience with residents (weight=14%) to make judgments about learner milestone levels. Generally, milestone levels improved significantly over time for second- and third-year residents; however, for first-year residents, milestone levels during the second reporting cycle were significantly lower. Milestones that received the most “not able to assess ratings” were PC4, PC5, SBP2, PBLI2 and PROF3. Identifying factors that affect assessment data at all stages of the CCC process can assist in improving assessment systems and in supporting faculty development for CCCs. In addition, recognizing challenges in synthesizing first- and second-hand assessment data is an important step toward understanding the CCC decision-making process.