External Validation of a Machine Learning Model for Delirium Risk
Joseph I. Friedman, Prathamesh Parchure, Weijia Fu, Satyanarayana Cheertirala, Prem Timsina, Ganesh Raut, Katherine Reina, Josiane Joseph-Jimerson, Madhu Mazumdar, Robert Freeman, David L. Reich, Arash KiaImportance
Although external validation of machine learning (ML)-based delirium risk models is a key stage in the model development pipeline, few published reports exist. Moreover, ML-based delirium risk models commonly degrade in performance during external validation.
Objective
To examine the performance of a fusion multimodal with natural language processing (NLP) delirium risk stratification model in live clinical practice with cross-hospital external validation.
Design, Setting, and Participants
This quality improvement study evaluated live clinical deployment of the model among patients 60 years or older who were admitted to non–intensive care units at Mount Sinai Hospital (MSH), with external validation performed at Mount Sinai Morningside Hospital (MSM), from December 1, 2024, to December 31, 2025.
Main Outcomes and Measures
Primary outcomes were performance of the risk stratification model, assessed by area under the receiver operating characteristic curve (AUROC), and monthly delirium detection rates. A 2 × 2 confusion matrix was created, and the 4 cells were used to calculate model performance metrics and compare associated diagnostic yield between premodel deployment and active model deployment periods.
Results
The overall sample included 33 095 admissions to non–intensive care units at MSM and MSH (mean [SD] age, 74.86 (9.22) years; 15 601 women [47.1%], 17 271 men [52.2%]). A total of 9020 MSM inpatient admissions were assigned a delirium risk score by the risk stratification model during live clinical operations and were compared to 21 186 MSH inpatient admissions that were assigned a risk score by the same model. Age at admission significantly differed between patients at MSM and MSH (MSM: median [IQR] age, 75.79 [68.25-83.88] years vs MSH: 72.62 [66.64-79.65] years;
Conclusions and Relevance
In this quality improvement study, an ML-based delirium risk stratification model maintained acceptable discriminative performance during external deployment without retraining and was similar to its performance at the development site without retraining. Model deployment was also associated with increased diagnostic yield but a high false positive rate. These findings provide further evidence of the potential generalizability of the model.