University of Illinois at Chicago

File(s) stored somewhere else

Please note: Linked content is NOT stored on University of Illinois at Chicago and we can't guarantee its availability, quality, security or accept any liability.

External validation of an opioid misuse machine learning classifier in hospitalized adult patients

journal contribution
posted on 2022-11-15, 22:13 authored by Majid Afshar, Brihat Sharma, Sameer Bhalla, Hale M Thompson, Dmitriy Dligach, Randy A Boley, Ekta Kishen, Alan Simmons, Kathryn Perticone, Niranjan S Karnik
BACKGROUND: Opioid misuse screening in hospitals is resource-intensive and rarely done. Many hospitalized patients are never offered opioid treatment. An automated approach leveraging routinely captured electronic health record (EHR) data may be easier for hospitals to institute. We previously derived and internally validated an opioid classifier in a separate hospital setting. The aim is to externally validate our previously published and open-source machine-learning classifier at a different hospital for identifying cases of opioid misuse. METHODS: An observational cohort of 56,227 adult hospitalizations was examined between October 2017 and December 2019 during a hospital-wide substance use screening program with manual screening. Manually completed Drug Abuse Screening Test served as the reference standard to validate a convolutional neural network (CNN) classifier with coded word embedding features from the clinical notes of the EHR. The opioid classifier utilized all notes in the EHR and sensitivity analysis was also performed on the first 24 h of notes. Calibration was performed to account for the lower prevalence than in the original cohort. RESULTS: Manual screening for substance misuse was completed in 67.8% (n = 56,227) with 1.1% (n = 628) identified with opioid misuse. The data for external validation included 2,482,900 notes with 67,969 unique clinical concept features. The opioid classifier had an AUC of 0.99 (95% CI 0.99-0.99) across the encounter and 0.98 (95% CI 0.98-0.99) using only the first 24 h of notes. In the calibrated classifier, the sensitivity and positive predictive value were 0.81 (95% CI 0.77-0.84) and 0.72 (95% CI 0.68-0.75). For the first 24 h, they were 0.75 (95% CI 0.71-0.78) and 0.61 (95% CI 0.57-0.64). CONCLUSIONS: Our opioid misuse classifier had good discrimination during external validation. Our model may provide a comprehensive and automated approach to opioid misuse identification that augments current workflows and overcomes manual screening barriers.


Great Lakes Node of the Drug Abuse Clinical Trials Network | Funder: National Institutes of Health (National Institute on Drug Abuse) | Grant ID: UG1DA049467

Employing eSBI in a Community-based HIV Testing Environment for At-risk Youth | Funder: National Institute on Drug Abuse | Grant ID: R01DA041071


Publisher Statement

Copy of license:


Afshar, M., Sharma, B., Bhalla, S., Thompson, H. M., Dligach, D., Boley, R. A., Kishen, E., Simmons, A., Perticone, K.Karnik, N. S. (2021). External validation of an opioid misuse machine learning classifier in hospitalized adult patients. Addiction Science & Clinical Practice, 16(1), 19-.


Springer Science and Business Media LLC


  • en