Development and Implementation of a Feedback Evaluation Tool

Purpose: We sought to develop and gather validity evidence for a concise feedback evaluation tool (FET) to assess feedback quality in a variety of clinical settings. Methods: The 9-item FET was developed through an expert consensus panel and review of existing literature; revisions were made based on pilot feedback. Nine standardized videos of feedback interactions were developed. Raters were recruited from 6 groups (expert, faculty, resident, medical student, non-physician medical, and non-medical); all raters underwent video-based rater training prior to watching and evaluating the videos using the FET. A factor analysis was performed to evaluate latent constructs. Intraclass and Pearson correlations were used to establish interrater reliability. A generalizability study with a nested design was used to examine the internal structure validity; a subsequent decision study was performed to determine the number of raters needed for a G-coefficient >0.80. Results: Each feedback video was reviewed by 18-25 raters yielding a total of 193 evaluations. A factor analysis identified two latent constructs measured. Intraclass correlations ranged from 0.89 to 0.96. Pearson correlations between expert raters and other rater groups ranged from 0.69-0.78. The generalizability study revealed a G-coefficient of 0.98 with 14.2% of variance attributed to the person delivering feedback. The decision study revealed that 2 raters of any type were needed for a G-coefficient >0.80. Discussion: This study demonstrates validity evidence in support of the use of the 9-item FET instrument when evaluating the quality of feedback delivered in a clinical setting. Future directions will include studying tool usability and implementation in a teaching setting.