RAMBIHAR-THESIS-2018.pdf (2.35 MB)
Download file

Validation of a Simulation-Based Tool For Formative Assessment of Echocardiography Skill Competence

Download (2.35 MB)
thesis
posted on 27.07.2018, 00:00 authored by Sherryn P Rambihar
Background: New guidelines for echocardiography training recommend assessment of core competencies using simulation, direct observation and multisource evaluation. Purpose: We developed a simulation-based echocardiography competence assessment tool (ECAT) of basic echocardiography skills. We collected validity evidence to evaluate our claims that the ECAT can be used for formative assessment. Methods: We conducted a mixed methods pilot of cardiology trainees (n=5 C1; n=5 C2; n=4 C3), who were oriented to the simulator and assessed using the ECAT. We video-recorded performance and collected participants’ simulated echocardiogram videos, which were assessed by four raters in real-time or off-line. After providing participants with video footage and rater feedback, we interviewed both participants and raters. For implications evidence, we used a simple content analysis to analyze interview data. For scoring evidence, we computed internal consistency and inter-rater reliability. For extrapolation evidence, we correlated ECAT scores with participants’ expertise, diagnostic scan quality, and a summative exam score. Results: Our analysis of the interview data revealed three themes: i) feedback stimulated change, ii) how feedback was delivered impacted participants’ perceived learning, and iii) assessment credibility influenced participants’ reception of feedback. The inter-rater reliability across all items was 0.837 (95% CI 0.81-0.87) and for ECAT score, ICC was 0.913 (95% CI 0.81-0.97). An exploratory factor analysis demonstrated a two-factor model, for which coefficient were 0.96 and 0.87 respectively. ECAT scores correlated with summative exam scores (r=0.66, p=0.02), were significantly higher with diagnostic quality scans (p=0.0007), and were associated with number of echocardiograms seen (p=0.014), performed (0.012) and interpreted (0.024). ECAT score increased according to level of training (p=0.01) with significant differences seen between first and second year trainees, and first and third year trainees, but no difference between second and third year trainees. Conclusions: Trainees and raters provided specific data for refining the ECAT process, feedback delivery, and technology for improved formative assessment. We suggest the collective evidence is favorable, supporting use of the ECAT in a simulation-based formative assessment. Our study provides practical insights for echocardiography educators and those seeking to develop instruments for formative assessment.

History

Advisor

Tekian, Ara

Chair

Tekian, Ara

Department

Medical Education

Degree Grantor

University of Illinois at Chicago

Degree Level

Masters

Committee Member

Brydges, Ryan Park, Yoon Soo Lineberry, Matthew

Submitted date

May 2018

Issue date

05/04/2018

Usage metrics

Categories

Exports