A Procedural Skills OSCE for Internal Medicine Residents to Assess Multiple Competencies

BACKGROUND: Internists are required to perform a number of procedures that require mastery of technical and non-technical skills. Formal assessment of these skills is often lacking. OBJECTIVE: To develop, implement, and gather validity evidence for a procedural objective structured clinical examination (PS-OSCE) for Internal Medicine (IM) residents, to assess their technical and non-technical skills when performing procedures. PARTICIPANTS: Thirty-five IM residents from post-graduate years one to three. METHODS: Residents participated in a five-station PS-OSCE combining task-specific models, standardized patients, and allied health professionals. Modern validity theory was used as a framework for gathering evidence for the validity of the scores. Formal blueprinting was performed and content experts were used to develop cases and the rating instruments. Examiners underwent a frame-of-reference training session to prepare them for their rater role. Reliability was calculated using a Generalizability analysis. Scores were compared by levels of training and experience and to a non-procedural OSCE (IM-OSCE). RESULTS: The g-coefficient for the technical and non-technical scores was 0.69 and 0.56, respectively. To reach a reliability of 0.80, nine technical and sixteen non-technical stations would be needed. PGY-3 residents scored significantly higher than PGY-1 residents on the technical (73.5% vs 62.2%) and non-technical (83.2 vs 75.1%) component of the PS-OSCE (p < 0.05). Residents who had performed the procedures more frequently scored higher on the central line, lumbar puncture, and thoracentesis stations (p < 0.05). There was a moderate correlation (r = 0.47) between the IM-OSCE and the technical component of the PS-OSCE scores. CONCLUSIONS: The PS-OSCE is a feasible and efficient method for assessing multiple competencies related to performing procedures and the validity evidence collected thus far supports its use for an in-training examination.