A Two-Phase Consensus Delphi to Define the EPAs for Internal Medicine in Canada

Purpose: Entrustable Professional Activities (EPAs) are a key, emerging tool for assessment in competency-based medical education. Various approaches to EPA development have been employed; none describe evidence for construct validity using the five sources of validity evidence. This study aimed to develop the stage-specific EPAs for post-graduate training in Internal Medicine in line with Royal College of Physicians and Surgeons of Canada’s Competence by Design framework. Further, it aimed to provide clear evidence for construct validity examining the methods used and EPAs generated. Method: This study employed a Delphi approach with three important modifications: inclusion of pre-existing source material; separation of the Delphi into two phases involving separate panels of clinical experts and educational experts; and inclusion of expert consensus conferences at key points. Validation of the Delphi design and the EPAs produced drew from the five sources of evidence for construct validity. Result: The Delphi generated 29 stage-specific EPAs; the EPAs achieved broad consensus across both community and academic physicians as well as the five member medical education expert panel. Analysis of the process identified validity evidence in each of content validity, response process, internal structure, relations to other variables, and consequences. Conclusion: The modified Delphi design used in this project generated EPAs for internal medicine with broad consensus across clinical and medical education experts. Analysis of the Delphi and EPAs revealed strong evidence of construct validity drawn from all five sources. This is the first study to present validity evidence for EPA development beyond evidence for content validity.