Validation of a Simulation-Based Tool For Formative Assessment of Echocardiography Skill Competence

Background: New guidelines for echocardiography training recommend assessment of core competencies using simulation, direct observation and multisource evaluation. Purpose: We developed a simulation-based echocardiography competence assessment tool (ECAT) of basic echocardiography skills. We collected validity evidence to evaluate our claims that the ECAT can be used for formative assessment. Methods: We conducted a mixed methods pilot of cardiology trainees (n=5 C1; n=5 C2; n=4 C3), who were oriented to the simulator and assessed using the ECAT. We video-recorded performance and collected participants’ simulated echocardiogram videos, which were assessed by four raters in real-time or off-line. After providing participants with video footage and rater feedback, we interviewed both participants and raters. For implications evidence, we used a simple content analysis to analyze interview data. For scoring evidence, we computed internal consistency and inter-rater reliability. For extrapolation evidence, we correlated ECAT scores with participants’ expertise, diagnostic scan quality, and a summative exam score. Results: Our analysis of the interview data revealed three themes: i) feedback stimulated change, ii) how feedback was delivered impacted participants’ perceived learning, and iii) assessment credibility influenced participants’ reception of feedback. The inter-rater reliability across all items was 0.837 (95% CI 0.81-0.87) and for ECAT score, ICC was 0.913 (95% CI 0.81-0.97). An exploratory factor analysis demonstrated a two-factor model, for which coefficient were 0.96 and 0.87 respectively. ECAT scores correlated with summative exam scores (r=0.66, p=0.02), were significantly higher with diagnostic quality scans (p=0.0007), and were associated with number of echocardiograms seen (p=0.014), performed (0.012) and interpreted (0.024). ECAT score increased according to level of training (p=0.01) with significant differences seen between first and second year trainees, and first and third year trainees, but no difference between second and third year trainees. Conclusions: Trainees and raters provided specific data for refining the ECAT process, feedback delivery, and technology for improved formative assessment. We suggest the collective evidence is favorable, supporting use of the ECAT in a simulation-based formative assessment. Our study provides practical insights for echocardiography educators and those seeking to develop instruments for formative assessment.