(O-T1) Predicting High Risk Emergency Department Bouncebacks: A Natural Language Processing Approach to Provider Notes

Objectives: Emergency Department (ED) Provider Notes can be used to predict high-risk ED bouncebacks using natural language processing machine learning techniques.

Background: Emergency Department (ED) 72-hour return visits are a marker for high-risk patient visits and can be a surrogate marker for lapses in the quality of care at the index visit. 1 Published risk factors for bounce backs resulting in hospitalization include age, public insurance, and end-stage renal disease amongst others. 2 While structured data from the medical record has been analyzed for factors associated with high-risk bounce backs, there is a paucity of evidence analyzing the large volume of unstructured data that is contained in provider notes, and that lend additional insight into the features associated with high-risk bouncebacks.

Methods: The authors analyzed patient encounters in the Mayo Clinic Midwest Emergency Departments from 05/04/2018 through 09/30/2022. ED encounters were included if the repeat ED visit was within 72-hours and resulted in a disposition of admission, hospital observation, expired, send to the operating room, send to cath lab, and transfer to another healthcare facility. ED encounters were excluded if the patient was not discharged on the index visit or experienced an irregular departure (eloped, left without being seen, left against medical advice). Notes from different provider types documenting on the same encounter were combined. Machine learning methods used included text cleaning, TFIDF and Count vectorization, creation of 1 to 3 n-grams, splitting data into test and training set, model fitting using support vector classifiers and logistic regression. Model explain ability techniques include LIME and SHAP.

Results: A total of 397,125 patients among 639,693 encounters were eligible for inclusion. Of these 10,094 (2.5%) resulted in a 72-hour bounceback with a concerning disposition. Prediction accuracy on the concerning bounceback cohort achieved 67% accuracy, precision of 68%, recall of 71% and an area under the receiver operative curve (AUROC) of 71%. Figure 1. After applying SHAP (SHapley Additive exPlanations), the top ten text features associated with the concerning repeat visits include “return’, “discharged”, “significant”, “ed”, “pending”, “iv”, “mild”, “plan”, “discussed”, and “care.” Figure 2.

Conclusions: Natural Language Processing techniques applied to Emergency Department Provider notes can be utilized to predict high-risk bounce-backs as surrogate markers for gaps in quality of care. Furthermore, NLP Machine Learning explainability techniques give insight into the types of terminologies that are associated with high-risk bouncebacks. These features can be combined with previously studied risk factors for high risk bouncebacks to increased prediction accuracy, and to generate comprehensive clinical phenotypes of ed patients at risk for these serious outcomes.

References (Optional): 1. Shy, B. D., Shapiro, J. S., Shearer, P. L., Genes, N. G., Clesca, C. F., Strayer, R. J., & Richardson, L. D. (2015). A conceptual framework for improved analyses of 72-hour return cases. The American journal of emergency medicine, 33(1), 104-107.
2. Gabayan, G. Z., Asch, S. M., Hsia, R. Y., Zingmond, D., Liang, L. J., Han, W., ... & Sun, B. C. (2013). Factors associated with short-term bounce-back admissions after emergency department discharge. Annals of emergency medicine, 62(2), 136-144.

Funding (Optional): N/A