Effect of Checklist Item on the Reliability of Internal Medicine Residency OSCE Scores

Objective Structured Clinical Examinations (OSCEs) are used worldwide for summative examinations but often lack acceptable reliability. Research has shown that reliability increases if OSCE checklists for medical students include only clinically relevant items. Also, checklists are often missing evidence-based items that high-achieving learners are more likely to use. The purpose of this study was to determine if limiting checklist items to clinically discriminating items and/or adding missing evidence-based items improved score reliability in an Internal Medicine residency OSCE. Six internists reviewed the traditional checklists of four OSCE stations classifying items as clinically discriminating or non-discriminating. Two independent reviewers augmented checklists with missing evidence-based items. We used generalizability theory to calculate overall reliability of faculty observer checklist scores from 45 first and second-year residents and predict how many 20-item stations would be required to reach a Phi coefficient of 0.8. Removing clinically non-discriminating items from the traditional checklist did not affect the number of stations (11) required to reach a Phi of 0.8 with 20 items. Focusing the checklist on only evidence-based, clinically discriminating items increased test score reliability, needing eight stations instead of 11 to reach 0.8; adding missing evidence-based clinically discriminating items to the traditional checklist modestly improved reliability (needing 10 instead of 11 stations). Checklists composed of evidence-based, clinically discriminating items improved the reliability of checklist scores and reduced the number of stations needed for acceptable reliability. Educators should give preference to evidence-based items over non-evidence-based items when developing OSCE checklists.