Validity Evidence for the Use of a Discharge Observation Tool

Purpose Patients' lack of understanding of treatment plans can result in readmissions and adverse events after discharge from the hospital. This study examines validity evidence supporting the use of a workplace-based direct observation tool and aims to align assessment of medical students’ ability to provide discharge instructions with patient-reported outcomes. Method To assess content validity, faculty and residents reviewed items of the Discharge Observation Tool (DOT) for relevance to discharging patients. After rater training, study participants viewed two videos of simulated discharge instructions and scored student performances using the DOT. Response process was assessed by determining the consistency and accuracy of ratings; DOT psychometric characteristics were evaluated for internal structure validity. To examine the alignment between DOT and patient-reported understanding of discharge instructions, logistic regression was used to determine if mean score on DOT could predict patient responses in post discharge phone calls. Results The DOT was rated as being relevant to patient discharge. The overall rater-consistency agreement measured using intraclass correlation (ICC) was 0.80; rater accuracy between trained raters and experts was 0.72. Learner variance was 11%. Phi-coefficient reliability was .70, demonstrating acceptable reproducibility for formative assessments. Higher DOT scores were associated with a higher likelihood of patient-reported knowledge of what to do in urgent or emergent situations (OR = 29.52, P = 0.002) and a lower likelihood of patient report of having barriers for attendance of follow up appointments. (OR = 6.08, P = 0.006). Conclusions The results of this study suggest that there is adequate evidence of content relevance, response process and reliability for the use of this workplace-based assessment as a formative tool. Evidence also suggests alignment in the assessment of student’s ability to provide discharge instructions with patient reported outcomes of knowledge of red flags and of barriers to attending follow up appointments.