Preliminary Experience with the Procedural Autonomy and Supervision System (PASS).

Purpose: Surgical residents are expected to be able to independently perform the core surgical procedures by the completion of their training. Existing methods for assessing resident operative performance are cumbersome, interrupt faculty workflow, and are often completed long after the end of the procedure. These issues lead to poor faculty compliance and potential for significant recall bias. To provide a more effective OR assessment system we have developed PASS (Procedural Autonomy and Supervision System). PASS is a smart phone based system that enables faculty to use the Zwisch scale to provide timely assessments of residents' operative autonomy with every case they perform. The Zwisch scale has 4-levels based on the amount of guidance the attending surgeon must provide to the resident to safely and effectively perform the procedure. Our goal with this study is to demonstrate that PASS with the Zwisch scale is a valid and feasible system for measuring resident operative autonomy without negatively impacting the OR environment. Methods: Prior to implementation, all participating general surgery residents and faculty underwent frame-of-reference training to the Zwisch scale. Once implemented, after every operation in which a resident participated, the system automatically prompted the attending to rate the resident's level of operative autonomy based on the Zwisch scale. Eight procedures were videotaped and independently rated by 2 additional surgeons. Zwisch ratings between the 3 raters were compared using an intra-class correlation (ICC) coefficient. Videotaped procedures were also scored using two alternative OR performance assessment instruments (OPRS and 0-SCORE), against which item correlations were calculated. OR times for procedures performed at least 3 times in both the 6 month periods before and during PASS implementation were compared using an independent t-test. A modified version of the OREEM survey tool (36 items) was used to assess the impact of PASS on the satisfaction of residents and faculty with the OR educational experience. Responses for faculty and residents before and during PASS implementation were compared using an independent t-test. Results: PASS was implemented for a seven month period by 27 full-time general surgery faculty to assess 1490 operative performances with 31general surgery residents. During the study period,faculty completed evaluations for 92% of all operations performed with general surgery residents. Zwisch scores were shown to correlate with PGY levels based on sequential pair-wise chi-squared tests: PGV1 vs. PGV2 (X2=76.2, df=3, p<0.001); PGY2 vs. PGY3 (X2=23.8, df=3, p<0.001); PGY3 vs. PGY4 (X2=38.3, df=3, p<0.001); Comparison of PGY4 to PGYS scores were not significantly different (X2=1.3, df=3, p=0.72). For the 8 operations reviewed for inter-rater reliability, the ICC coefficient was 0.90 (95% Cl 0.72-0.98, p<0.01). Correlation of PASS ratings with both OPRS items (each r>0.90,all p<0.01} and 0-SCORE items (each r>0.86,all p< 0.01) was high. Faculty and resident compliance with completing the modified OREEM survey was > 80%. There were no significant differences between the resident or faculty responses obtained before and during PASS implementation.OR times were compared for 40 procedures performed at least 3 times during both time periods evaluated. There were no significant differences in OR times before and during PASS implementation. Conclusions: The Zwisch scale is a reliable and valid measure of operative autonomy that does not appear to have any major disruptive effect on OR times or on residents' and faculties' perception of the OR experience. Deployed on an automated smart-phone based system PASS can be used to feasibly record evaluations for the vast majority of operations performed by residents. This information can be used to council individual residents,modify programmatic curricula, and potentially inform national training guidelines.