Validity Evidence for Use of the Ontario Bronchoscopy Assessment Tool in Simulated Settings

The purpose of this study was to gather validity evidence for use of the Ontario Bronchoscopy Assessment Tool (OBAT) to assess the technical skills of pulmonary fellows in a high fidelity (cadaver and VR) simulated setting. The Ontario Bronchoscopy Assessment Tool was developed to assess clinical competence of trainees in bronchoscopy. Validity evidence for this tool has been demonstrated in clinical settings. The initial validation study of the OBAT did not assess interrater reliability. There is no validity data for the OBAT in simulated settings. First-year pulmonary fellows were recruited during a regional bronchoscopy workshop. All participants were assessed by two raters on the technical portion of the OBAT while performing bronchoscopy on a cadaver at three different task stations. Scores from the two raters were compared to determine interrater reliability. A G study was performed to assess sources of variance in the OBAT scores. At the end of the course fellows were asked to perform an additional bronchoscopy on a virtual reality (VR) bronchoscopy simulator. Each of the fellows was assessed by two raters using the OBAT and the BSTAT. Interrater reliability (IRR) was calculated as intraclass correlation (ICC).The relationship between OBAT and BSTAT scores were calculated using Pearson’s correlations. Internal consistency of the OBAT was calculated using Cronbach’s alpha. Sixty-nine observations were performed for 42 fellows. The IRR on the technical portion of the OBAT was 0.61, p<0.001. The generalizability study demonstrated a G-coefficient of 0.56 across the 3 stations. Variance attributed to the fellows was 11.2%, station was 25.9% and the interaction of fellows with raters nested in stations (fr:s) was 27.6%. The Pearson’s correlation between the technical portion of the OBAT and BSTAT scores on the VR simulator across 15 fellows was r = 0.62, p=0.014. The technical portion of the OBAT demonstrated good IRR in a high fidelity (cadaver) simulated setting. G-study results suggest the need for repeated observations across different tasks to obtain a stable estimate of skills.