Mastery Skill Assessment in Hepato-Pancreato-Biliary Surgical Ultrasound

Background: A formative Hepato-Pancreato-Biliary (HPB) ultrasound (US) skills practicum is offered annually to HPB fellows in Americas Hepato-Pancreato-Biliary Association-Fellowship Council (AHPBA-FC) accredited programs, using assessment tools for open (IOUS) and laparoscopic (LAPUS) US. The aim of this work is to collect validity evidence for the IOUS and LAPUS tools, while establishing and applying Mastery standards and to a sample of graduating fellows. Methods: Faculty Survey (content; consequential validity): Expert AHPBA faculty were surveyed to evaluate the importance of each IOUS and LAPUS skill and to set Mastery Angoff probabilities. Item-level Mastery Angoff cut scores were determined as the mean rating across all judges. The overall Mastery Angoff cut score was determined as the mean cut score across all items. Mastery Angoff cut scores were converted to the 5-point entrustment scale (i.e., cut score*5). Fellow performance (response process; consequential validity): Fellow performances were evaluated using the IOUS and LAPUS tools (entrustment rating) during two annual US skills practicums, and fellow experience was collected. Mastery Angoff standards were applied. This study met criteria for exemption by the University of Illinois-Chicago Institutional Review Board (2019-0265). Results: Data from 11 faculty questionnaires were included. The mean cut scores across all items (Mastery Angoff) was 92% (equivalent entrustment 4.6) and 91% (entrustment 4.5) for IOUS and LAPUS, respectively. The mean global cut scores (Mastery Global Angoff) were 96% (SD 6.4%) and 89% (13.8%) for IOUS and LAPUS, respectively. 78.5% (29/37) fellows agreed to have their de-identified data evaluated. The mean fellow performance entrustment scores (across all skills) were 4.1 (SD 0.6, range 2.6-4.9) and 3.9 (SD 0.7, range 2.7-5), while the mean global fellow performances were 3.6 (SD 0.8, range 2-5) 3.5 (SD 1.0, range 2-5) for IOUS and LAPUS, respectively. Conclusion: AHPBA-FC HPB fellows prior to graduation are not meeting Mastery Standards for HPB US performance determined by a panel of expert faculty. A change in educational policy or curriculum may be needed to improve fellow performance.