Validity Evidence for Construct-Aligned Entrustment Assessments in Vascular Surgery

OBJECTIVE Workplace-based assessments of surgical trainees historically demonstrate poor alignment between faculty and learner perception of competence and autonomy in a given clinical encounter. Furthermore, demographic differences in assessment of trainees have been demonstrated across specialties. Given that trainee perception of receiving low autonomy is associated with higher rates of burnout, depression, and attrition, it is important to better align faculty and trainee perceptions of clinical experiences. We sought to evaluate validity evidence data of vascular surgery Entrustable Professional Activities assessments (EPAs) using national pilot implementation data. METHODS A multi-institutional pilot implementation of 15 vascular surgery EPAs was open to Accreditation of Graduate Medical Education (ACGME) vascular surgery residencies and fellowships from April-June of 2024. Participating programs collected faculty, resident, and fellow assessments on entrustment for clinical encounters. Descriptive statistics were conducted on matched encounter assessments. A linear mixed-effects model was conducted to examine factors associated with the alignment. Intraclass correlation was calculated between trainee and faculty assessments. A random effects model was applied to all assessments to demonstrate sources of variance in entrustment ratings. Linear mixed effects models clustering by trainee, faculty, program, and EPA were applied to examine differences in faculty assessed and trainee self-assessed entrustment scores by faculty gender, trainee gender, race, and ethnicity, post-graduate year, phase of care, and faculty-trainee gender concordance. RESULTS Twenty-nine programs contributed 1,620 matched assessments (n=79 trainees, n=87 faculty). There were no differences in absolute alignment scores by trainee gender, race, ethnicity, faculty gender, faculty-trainee gender concordance, or EPA phase of care. ICC ranged from moderate to excellent across all EPA types (ICC=0.51-0.89), and excellent for all EPAs combined (ICC=0.76). Variance in faculty entrustment ratings was attributed primarily to the interaction between EPA and trainee (31%). Faculty entrustment ratings were not different when considering trainee gender, race, ethnicity, faculty gender, or faculty-trainee gender concordance. CONCLUSION In this retrospective analysis of national EPA pilot implementation data, response process, internal construct, and consequence validity evidence were demonstrated. Shortcomings of other workplace-based assessments used in surgical training were not evident with the use of behaviorally-anchored EPA assessments.