University of Illinois at Chicago
OKEEFFE-THESIS-2019.pdf (436.13 kB)

Evaluating the Validity of a National Multi-Assessment System in Postgraduate Surgical Training

Download (436.13 kB)
posted on 2019-08-01, 00:00 authored by Dara Ann O'Keeffe
Objective: To evaluate the validity evidence supporting the use and interpretation of a multi-faceted assessment system in surgical training. This was a national study using data collected from two cohorts of the entire population of postgraduate trainees in Ireland. Summary Background Data: Trainees on the National Core Surgical Training program administered by the Royal College of Surgeons in Ireland undergo multiple assessments over the first two years of postgraduate training. These assessments are collated into a high-stakes summative assessment at the end of their second year which determines their progression into higher surgical training in the specialty of their choice. Methods: This was an evaluative study of the validity evidence supporting our assessment process and results. Trainee assessments were categorised as Workplace-based (WBA), Structured assessment performed by the Royal College of Surgeons in Ireland (RCSI) and Multiple Mini Interview (MMI). Validity evidence was examined using Messick’s unified validity framework in the following domains: Content, Response process, Internal structure, Relations to other variables and Consequences of testing. Results: Best practice standards for educational testing were adhered to in the majority of steps in the assessment providing a large body of validity evidence to support the use of this process. Composite score reliability of the assessment was 0.89 which demonstrates a highly reliable process. Correlation between workplace-based assessments and standardised tests performed in the simulation setting was also very high (0.93). Conclusions: The Royal College of Surgeons in Ireland’s assessment process for Core Surgical Training is statistically highly reliable and is supported by a large body of validity evidence.



Park, Yoon Soo


Park, Yoon Soo


Medical Education

Degree Grantor

University of Illinois at Chicago

Degree Level

  • Masters

Degree name

MHPE, Master of Health Professions Education

Committee Member

Tekian, Ara Traynor, Oscar

Submitted date

August 2019

Thesis type



  • en

Issue date


Usage metrics


    No categories selected


    Ref. manager