Validity and Feasibility Evidence for the Professionalism Mini-­Evaluation Exercise in Resident Admissions

Purpose: To provide validity and feasibility evidence for use of the Professionalism Mini-Evaluation Exercise (P-MEX) in a pediatric residency admissions process. Method: In 2012 and 2013, the OSCE-based P-MEX was administered to applicants invited for an interview at the University of Geneva Pediatrics Residency Program. Validity evidence was gathered for content (item analysis and qualitative feedback); response process (inter-rater reliability with intraclass correlation); internal structure (Generalizability); relationship to other variables (correlations); and consequences (logistic regression to predict admission). Kane’s formula was used to create composite scores using P-MEX, structured letter of reference (SLR), and structured interview (SI) scores. Differences in the applicant rank list based on composite score versus faculty global ratings were analyzed using Wilcoxon signed-rank test. Results: Seventy applicants were assessed. Moderate associations were found between pairwise correlations of P-MEX scores and SLR (r=0.25, P = .036), SI (r=0.34, P = .004), and global ratings (r=0.48, P < .001). P-MEX inter-rater reliability was moderately low (ICC=0.36). Reliability of the P-MEX was moderate (G-coefficient=0.45). Including P-MEX scores increased composite reliability from 0.51 to 0.74. P-MEX scores had the greatest correlation with acceptance (r=0.56, P < .001), were the strongest predictor of acceptance (OR=4.37, P < .001), and increased pseudo R-squared by 0.20 points. Rank lists of applicants using composite score versus global rating differed significantly (z=5.41, P < .001).