Validity and Feasibility of the Minicard Workplace Direct Observation Tool in a Single Training Program

Purpose. To collect validity and feasibility evidence for use of the Minicard direct observation tool for assessment of competence of internal medicine residents. Method. Retrospective cohort analysis of validity evidence and feasibility of the Minicard from 2006-2011 in one institution, including content (settings, observation rates, independent raters), response process (scoring distributions), internal structure (factor analysis), relationships (to time in training, OSCE and medical knowledge exams) and consequences (qualitative analysis of action plans), as well as feasibility (time and financial costs). Results. 3715 direct observations were analyzed from 80 faculty observers rating 73 residents. Residents averaged 28 (SD 8.4) observations per year from 9 (SD 4.1) independent observers. Scoring distributions used the entire rating scale. Confirmatory factor analysis identified a three-factor fit representing medical knowledge, communication and professionalism (χ2(51)= 107, p<0.05; TLI = 0.97; CFI = 0.98; RMSEA = 0.025). Individual resident scores increased significantly each month. Minicard communication scores correlated weakly with overall OSCE communication Z-scores (r=.11, p<0.001), and Minicard applied medical knowledge average score correlated weakly with in-training exam Z-score of that year (r=.07, p=.02). Action plan reviews identified action-oriented feedback in 50%, observational feedback in 11%, minimal feedback in 9% and no recorded plan in 30%. Observation times averaged 15.6 (SD: 9.5) minutes. Conclusion. This study demonstrates validity and feasibility evidence for a direct observation system for residents that can produce a broad range of observations, a wide range of scores and substantial formative feedback.