Considerations of Reusing Multiple Choice Items to Assess Medical Certification Repeat Examinees

As gatekeepers to medical specialty practice, medical boards are responsible for accurately identifying who is qualified for certification. Boards often utilize comprehensive multiple choice exams to assess minimum competence and make certification determinations. Generally, boards permit examinees who fail an exam to sit for the exam again. At the same time, boards must frequently reuse exam items due to issues with test equating or limited item availability. Given the potential memory advantages and disadvantages of prior item exposure that might interfere with the assessment of repeat examinees, the ongoing discussion of how best to assess repeat examinees has largely focused on how to minimize item exposure. The goal of this study was to contribute to the research available to guide the development of defensible retest policies in light of the reuse of multiple choice items. Through investigation of repeat examinee scores on a single medical certification exam, their response patterns on items common across exam attempts, and the measurement capabilities of the items themselves, this research helps contextualize retest score differences and pass-fail outcomes. Results suggested that repeat examinee outcomes were largely related to overall content knowledge, rather than any memory effects stemming from prior item exposure. Moreover, indications of any memory effects varied with category of item difficulty, discrimination power, content area, and cognitive complexity. The findings of this study support the need for medical boards to focus on the quality, not the number, of multiple choice items that they reuse when assessing repeat examinees in order to minimize the potential negative consequences of prior item exposure.