Concise Critical Appraisal: Impact of a Machine Learning Early Warning Score on Hospital Mortality

Machine learning is a branch of artificial intelligence that uses large datasets and extracts knowledge for highly constrained tasks.¹ Supervised machine learning employs multiple input variables to predict a predefined outcome² and is commonly used for research on clinical deterioration. The widespread use of electronic health records (EHRs) has provided large volumes of data points that can be used to create machine learning models that can predict deterioration in patients or early warning scores.
 
Common causes of death in hospitals, such as sepsis and respiratory failure, are treatable and benefit from early intervention. Machine learning algorithms can be used for early identification and recognition to potentially help accelerate interventions and limit morbidity and mortality. Many studies have looked at early interventions for sepsis and respiratory failure and early recognition of deterioration in general to identify patients who may need additional resources or ICU transfer. Most studies involve large retrospective datasets used to validate machine learning models. Winslow et al looked at one such validated score—the electronic cardiac arrest risk triage score (eCART)—in a multicenter prospective cohort study.³
 
The eCART scores were classified as high risk, intermediate risk, and average risk. Initially, a two-month pilot project was instituted at each of the four participating hospitals. The eCART scores were integrated into the EHRs, and a workflow was developed and implemented. New, higher scores prompted a physician-directed workflow for assessment for ICU admission. A new intermediate-risk score triggered a bedside nurse-directed workflow.
 
A total of 6681 patients were studied. Of these, 3075 (46%) had one high-risk score, and 3606 (54%) had an intermediate-risk score. Compared to baseline, overall hospital mortality was significantly lower after implementing the eCART machine learning model (8.8% vs. 13.9%, P < 0.01), with a relative risk reduction of 36.7%. The decrease in mortality was seen in both high-risk (17.9% vs. 23.9%, P = 0.001) and intermediate-risk (2% vs. 4%, P = 0.001) patients. The increased likelihood of being transferred to the ICU and the decrease in median time to transfer by 13.6 hours in the high-risk score category suggest process improvement by implementation of the eCART model. No significant differences in ICU length of stay or need for mechanical ventilation suggested that the ICU stay overall was not affected, but time to ICU interventions was shortened.
 
Heterogeneity in the presentation of disease processes such as sepsis often leads to delayed recognition and care, contributing significantly to patient morbidity and mortality. With the advent of machine learning models, which leverage extensive patient data available in EHRs, earlier recognition is possible. Artificial intelligence-based systems to predict clinical deterioration are being increasingly developed based on retrospective studies. While these technologies show promise, the pathway to affect meaningful patient outcomes in the real world is challenging. The study by Winslow et al is one of the first multicenter prospective real-world implementations of a machine learning model. Implementing the model led to improving not only mortality but process measures and metrics as well. More studies are being done that provide similar data and benefits, such as a recent study by Adams et al.⁴
 
With the advent of more artificial intelligence, machine learning-based algorithms, and predictive models, the critical care specialty needs to develop a better understanding of these models. This will allow critical care practitioners to embrace them, develop best practices around them, incorporate them into daily workflows, and leverage them for improved patient outcomes. They can be incredible assets in critical care settings.

References

1. IBM Cloud Education. What is machine learning? IBM July 15, 2020. Accessed November 15, 2022. https://www.ibm.com/cloud/learn/machine-learning
2. Malycha J, Bacchi S, Redfern O. Artificial intelligence and clinical deterioration. Curr Opin Crit Care. 2022 Jun 1;28(3):315-321.
3. Winslow CJ, Edelson DP, Churpek MM, et al. The Impact of a machine learning early warning score on hospital mortality: a multicenter clinical intervention trial. Crit Care Med. 2022 Sep 1;50(9):1339-1347.
4. Adams R, Henry KE, Sridharan A, et al. Prospective, multi-site study of patient outcomes after implementation of the TREWS machine learning-based early warning system for sepsis. Nat Med. 2022 Jul;28(7):1455-1460.