Learner Engagement and Teaching Effectiveness in Livestreamed vs In-person Continuing Medical Education

Engaging continuing medical education (CME) learners in online environments can be challenging. Historically, CME is delivered in conference-style lectures, employing passive teaching modalities such as podium presentations. However, over the last several years, CME courses have transitioned to offering online (livestreamed) CME, with learners viewing live in-person CME courses remotely online. The authors aimed to 1) compare learner engagement and teaching effectiveness in online/livestreamed with in-person CME and 2) determine how livestream learner engagement is associated with A) livestream interactivity metrics, B) course presentation characteristics, C) medical knowledge, and D) teacher effectiveness. A three-year, non-randomized study of in-person and livestream CME was performed. The course was in-person for 2018 but transitioned to livestream for 2020 and 2021. Learners completed the Learner Engagement Inventory and CME Teaching Effectiveness Instrument after each presentation (55-57 presentations/year). Both instruments were supported by content, response process, internal structure, and relations to other variables’ validity evidence. For the livestreamed CME, interactivity metrics and presentation characteristics were collected. Interactivity metrics included learner use of audience response, questions asked by learners, and presentation views. Presentation characteristics included presentations using audience response questions, using pre/post-test format, time of day, and words per slide. Medical knowledge was assessed by audience response. A repeated measures ANOVA was used for comparisons and a mixed model approach was used for correlations. 159 learners (response rate 27%) completed questionnaires. Learner engagement did not significantly differ between in-person or livestream CME. (4.56 vs 4.53 [difference 0.04; 95% CI, -0.19, 0.11], p=0.64, maximum 5 = highly engaged). However, teacher effectiveness scores were higher for in-person compared to livestream (4.77 vs 4.71 [difference 0.06; 95% CI, 0.01, 0.09], p=0.01, maximum 5 = highly effective). For livestreamed courses, learner engagement was associated with various presentation characteristics, including presentation using of audience response (yes = 4.57, no = 4.45 [difference 0.12; 95% CI, 0.16, 0.67, ], p<.0001), use of a pre/post-test (yes = 4.62, no = 4.54 [difference 0.08; 95% CI, 0.05, 0.11], p<.0001), and time of presentation (morning = 4.58, afternoon = 4.53 [difference 0.05; 95% CI, 0.02, 0.08, p=.0002). Interactivity metrics and medical knowledge did not show significant associations with learner engagement. Learner engagement was statistically similar, and teacher effectiveness showed only small differences, comparing livestream and in-person CME. Some presentation characteristics, including presentations using audience response, using a pre/post-test format, and morning presentations were associated with increased engagement for livestream CME. Interactivity metrics and medical knowledge did not show a significant association with learner engagement.