A Mixed-Methods Validity Study Of Chart-Stimulated Recall To Review Consults With Night-Float Residents

In graduate medical education there is a need to assess specialty specific aspects of training and practice that are often underrepresented in EPA lists, while also providing formative assessment and feedback. Chart-stimulated recall (CSR) potentially meets these requirements despite being historically underused. We developed and tested the validity of a CSR assessment tool for reviewing overnight General Internal Medicine (GIM) consults with post-call Night-Float (NF) residents using a mixed methods approach in a training program at McMaster University. The CSR tool was used when supervisors reviewed consults seen by NF residents, leading to a clinical case discussion and a rating score for several individual aspects of the resident’s performance, and ending with specific feedback. Encounters were recorded and copies of the clinical notes were made. Two individual assessors later reviewed each recorded encounter, generating independent rating scores. Study participants were later interviewed about their experiences using the CSR tool. The three separate rating scores for each case-review encounter based on the CSR categories were analyzed using ANOVA to determine inter-rater reliability for different raters, items, and CSR sections. All were found to be statistically non significant. Interview transcriptions were analyzed qualitatively using a semi-inductive iterative method of constant comparison analysis associated with grounded theory, generating a framework of four themes (implementation, learning, personal experience, and the CSR tool). We found limited validity evidence based on internal structure, response processes, and consequences, for using CSR in this context. Positive aspects of this assessment included increased frequency and quality of feedback, and the idea of optimizing all other elements involved in a work-place based assessment, beyond the assessment tool itself. These elements, as a composite, generated the concept of the ‘assessment moment’. Our study found limited interrater reliability and validity evidence for the use of CSR in reviewing overnight consults with NF residents. The elements involved in work-place based assessments, beyond the tool itself, had a significant impact on the experience. Residency training programs may consider focusing on creating a local culture and workflow that enables effective and routine ‘assessment moments’.