Developing a Canadian In-Training Examination in Pediatric Emergency Medicine

Objectives: To develop, and collect initial validity evidence for, a nationally-delivered Canadian In-Training examination (ITE) in Pediatric Emergency Medicine (PEM). Methods: A rigorous process was undertaken including blueprinting, item writing, piloting and scoring. Validity evidence collected from PDs included feedback on the blueprint, draft exam, and usefulness of the ITE, as well as feedback on the feasibility of the development and scoring process. Feedback was sought from residents on the fairness and usefulness of the ITE. Cronbach’s alpha was used to assess internal consistency of the ITE. Mean exam scores were compared by year of training, and exams were double scored to assess inter-rater reliability (IRR). Results: Ten Canadian PEM training sites participated in the project; 47 residents who completed the final pilot consented and were included in data analysis. Cronbach’s alpha coefficient for the ITE across items was 0.861, indicating high internal consistency. An ANOVA with linear contrast showed a significant linear association between year of training and mean score (F = 7.53, df = 2, p = 0.009). The intraclass correlation coefficient for ITE total score between PD and subcommittee raters was 0.85 (CI = 0.30 – 0.95), indicating high IRR. On a 5-point scale from ‘Not at All Feasible’ to ‘Extremely Feasible’, mean PD rating was 4.13 (SD ± 1.13) for the exam creation and delivery process, and 3.88 (SD ± 0.99) for the scoring process. Resident mean rating when asked about usefulness of the ITE was 3.18 (SD ± 1.03) on a 5-point scale from ‘Not at All Useful’ to ‘Extremely Useful’, and 98% of residents found it fair. Conclusions: The initial process of ITE creation and delivery resulted in an examination with a high internal consistency and inter-rater reliability. The ITE was useful to both PDs and residents, and feasible for PDs to create and deliver.