Validity Evidence for a Key Features Examination in the Internal Medicine Clerkship

Key features exams (KFEs) are used to assess clinical decision making. Four forms of an online KFE were developed to assess third-year medical students’ decision-making abilities during internal medicine clerkships in the U.S. We used Messick’s unified assessment framework to gather validity evidence regarding content, response process, internal structure, relationship to other variables, and consequences. At 8 U.S. medical schools, 759 students during their internal medicine clerkship were given 75 minutes to complete one of four 15-case KFE forms. They also completed a survey regarding their experiences. Item analyses and generalizability studies were performed. KFE scores were compared with prior clinical experience and scores on the National Board of Medical Examiners Subject Examination (NBME-SE). A total of 515 (67.9%) students consented to participate. Mean form scores ranged from 54.5% to 60.3% (SD 8.4-9.6). Phi-coefficient ranged from 0.36 to 0.52; adding 5 cases to the most reliable form would increase the Phi-coefficient to 0.59. Removing the least discriminating case from the two most reliable forms would increase the alpha coefficient to 0.58 and 0.57. The main source of variance came from the interaction of students (nested in schools) and cases. Correlation between KFE and NBME-SE scores ranged from 0.24 to 0.47 (p<0.01). Three of the KFE forms had comparable reliability to other KFEs reported in the literature, despite requiring less testing time. This study provides the first validity evidence for KFEs for assessing clinical decision making in U.S. internal medicine clerkships.