Development and external validation of an AI- model to triage echocardiograms
J Strom, L Liang, N Spetko, Y Song, C Angell-James, M A Cassidy, S Horng, J P Stevens, K W Rainer, P A EliasAbstract
Background
Over 70% of US institutions report ongoing cardiac sonographer shortages, limiting necessary echocardiograms (TTEs). Whether deep learning (DL) models applied to ECG data could be used to triage TTEs with low-risk of identifying actionable pathology is uncertain.
Purpose
We aimed to derive and externally validate a DL algorithm, ASTRAL-ECG, using ECG waveform inputs to predict the composite of LVEF < 50% or wall motion abnormalities, ≥ moderate pericardial effusion, intracardiac mass, acute aortic syndrome, VSD or ASD, 2+ or greater valvular disease, ≥ moderate pulmonary hypertension, aortic aneurysm, or ≥ moderate RV dysfunction.
Methods
Data from the Medical Information Mart for Intensive Care, 4th version (MIMIC-IV) were used to identify those with same day ECG and TTE at our institution. A 2D convolutional neural network, ASTRAL-ECG, was used to classify ECGs as positive with respect to the composite outcome with internal validity assessed using 10-fold cross-validation. Gradient-weighted Class Activation Mapping (Grad-CAM) maps were generated for interpretation. Results were externally validated in a multi-center ECG-TTE dataset from the EchoNext study and cutoffs chosen to maximize sensitivity.
Results
In MIMIC-IV (N = 45,321) and EchoNext (N = 77,917) samples, 15,410 (34.0%) and 32,676 (41.9%) had the composite outcome respectively. Model discrimination and positive test performance in the MIMIC-IV leave-out sample was excellent (accuracy 78.4%, AUC = 0.847, area under the precision recall curve [AUPRC] = 0.745, sensitivity 68.3%, specificity 83.6%). Grad-CAM identified the p-wave and QRS complex as the primary model focus for classification [Figure 1]. After similar waveform reprocessing, performance in the EchoNext validation sample was excellent (accuracy = 76.8%, AUC = 0.804, AUPRC = 0.705, sensitivity = 39.9%, specificity = 95.1%) [Figure 2]. An operating threshold of 0.098 had a sensitivity of 95.6%, specificity 40.8%, PPV 45.4%, and NPV 94.7% for the composite in the derivation leave-out test sample.
Conclusions
A novel AI-ECG model had high performance for ruling out actionable pathology on paired TTE. While requiring further clinical validation, this model could be used to triage the urgency of TTE requests by deprioritizing low-risk studies.