Evaluating the Validity of a National Multi-Assessment System in Postgraduate Surgical Training

Objective: To evaluate the validity evidence supporting the use and interpretation of a multi-faceted assessment system in surgical training. This was a national study using data collected from two cohorts of the entire population of postgraduate trainees in Ireland. Summary Background Data: Trainees on the National Core Surgical Training program administered by the Royal College of Surgeons in Ireland undergo multiple assessments over the first two years of postgraduate training. These assessments are collated into a high-stakes summative assessment at the end of their second year which determines their progression into higher surgical training in the specialty of their choice. Methods: This was an evaluative study of the validity evidence supporting our assessment process and results. Trainee assessments were categorised as Workplace-based (WBA), Structured assessment performed by the Royal College of Surgeons in Ireland (RCSI) and Multiple Mini Interview (MMI). Validity evidence was examined using Messick’s unified validity framework in the following domains: Content, Response process, Internal structure, Relations to other variables and Consequences of testing. Results: Best practice standards for educational testing were adhered to in the majority of steps in the assessment providing a large body of validity evidence to support the use of this process. Composite score reliability of the assessment was 0.89 which demonstrates a highly reliable process. Correlation between workplace-based assessments and standardised tests performed in the simulation setting was also very high (0.93). Conclusions: The Royal College of Surgeons in Ireland’s assessment process for Core Surgical Training is statistically highly reliable and is supported by a large body of validity evidence.