Learner Assessment for Technical Skills in Real vs. Simulated Environments: Is It Comparable?

Although simulation-based assessment and certification of technical skills is the focus of research and practice for many medical educators, there are few evidence-based reports of simulation-based assessments that compare performance to actual clinical settings. It would be useful to establish the boundaries of performance in each environment, to determine whether and why they are different. This study compared anesthesia residents’ performance of a technical task – sterile technique – completed both on a patient simulator and on a live patient. The authors hypothesized that scores would differ by less than 20%, with better performance on the simulation. Nine anesthesia residents completed videotaped epidural catheter insertions in both a live patient and an epidural part-task trainer simulator. Performance was assessed for adherence to accepted sterile technique, based on videotape review, using a 15-item checklist. Scores were compared using a matched-pairs t-test; means, standard deviations, and p-values were generated. Anesthesia residents’ sterile technique scores showed a non-significant trend toward better performance with live patients than with the simulator (mean scores±SD 18.89±4.37 vs 15.78±6.67 respectively, mean difference=3.1, 95% CI -0.5 to 6.7, p=0.0824). The worst-performed items involved operator hygiene. When checklist items concerning handwashing and OR attire were excluded, much of the observed difference in scores was eliminated (mean scores±SD 15.78±4.52 vs 15.11±5.75, mean difference 0.67, 95% CI -2.4 to 3.7, p=0.6305). Keeping in mind certain limitations of this study, the evidence is consistent with the possibility that, for the observed simple skill set requiring limited judgment, checklist scores on performance with simulated vs. live patient encounters may be comparable. Further research is needed to allow more precise estimates of this potential correspondence. Certain aspects of simulated environments may need revision in order to support desired learner performance during technical skills training.