Reducing the Number of Options on Multiple-Choice Questions: Response Time, Psychometrics, and Standard Setting

Reducing the Number of Options on Multiple-Choice Questions: Response Time, Psychometrics, and Standard Setting Abstract Purpose Despite evidence for the use of three-option multiple-choice questions (MCQs), they are rarely used on written exams for health professions students. The purpose of this study was to examine the effect of reducing four- and five-option MCQs to three-option MCQs on response times, psychometric characteristics, and absolute standard setting judgments for a pharmacology exam administered to health professions students. Method We administered two forms of a computerized MCQ exam to second-year medical students and third-year pharmacy students. Four- and five-option MCQs were converted into three-option MCQs to create the two exam forms. Differences in response time, item difficulty and discrimination, and reliability were evaluated. Medical and pharmacy faculty judges provided three-level Angoff ratings for all MCQs for both exam forms to assess differences in cut scores. Results Students (n=77) answered three-option MCQs, on average, 5.24 seconds faster than the four- and five-option MCQs (p=.008); no significant differences in item difficulty and discrimination, or test reliability were observed. Overall, the cut scores generated for three-option MCQs were 8.32 % higher. Conclusion The use of three-option MCQs in a health professions exam resulted in time savings that could allow 16% more MCQs to be tested within a 1-hour testing period, which would increase content validity. The higher cut score may result in higher failure rates.