A Construct-aligned Assessment Tool for Internal Medicine Residents: Development and Validity Evidence

Purpose: Workplace-based entrustment-supervision scales are key components of assessment in postgraduate competency-based medical education. While the literature supports entrustment-based scales to be construct-aligned, it is unclear why construct-aligned instruments generate stronger validity evidence. Furthermore, evidence is lacking to support their use across different contexts. The goal of this study is to develop a construct-aligned entrustment-based assessment instrument for internal medicine senior residents and understand why it works. Methods: We developed a construct-aligned entrustment instrument designed to assess senior residents on the internal medicine Clinical Teaching Unit. Ten faculty members were interviewed to assess for construct-alignment. Cognitive interviews were then conducted with seven different supervisors who used the instrument to assess senior residents. The research team conducted thematic analysis to understand supervisor perspectives on its use. Results: The assessment instrument was aligned with the construct of supervision of senior residents in internal medicine. Participants found that by focusing the assessment on their own supervisory behavior (i.e. “I had to step-in more than usual”) they were willing to use the full range of the entrustment scale and they found it easy to complete. Comparing their supervision to ‘usual’ practice captured the way supervisors think of supervision and may help them account for contextual factors that lead to variability in assessment. Conclusions: This work outlines an approach for developing a construct-aligned entrustment-based assessment instrument for internal medicine senior residents with validity evidence supporting its use. This approach holds promise for improving programs of assessment within non-procedural specialties and advances the understanding of why construct-aligned assessment may be effective. These advances could improve assessment and enhance engagement among assessors and learners.