Impact of intra-patient variability of artificial intelligence and machine learning models on the diagnosis of myocardial infarction
W Zeng, I Strebel, P Lopez-Ayala, T Zimmermann, S Knecht, L Koechlin, E Kaplan, O Miro, A T Champetier, T Stolte, K Durak, M Christ, F Mahfoud, C Mueller, J BoeddinghausAbstract
Background
Current artificial intelligence (AI) and machine learning (ML) prediction models for myocardial infarction (MI) use a single 10s ECG. Whether model performance changes across serial ECGs or longer recordings is poorly understood. We aimed to investigate the impact of intra-patient variability on model prediction for MI.
Purpose
To predict Non-ST-Elevation Myocardial Infarction (NSTEMI) type 1 using three AI and ML models, quantify prediction variability, and evaluate the impact on MI diagnosis and clinical risk assessment.
Methods
Logistic Regression (LR) and eXtreme Gradient Boosting (XGBoost) models, and a Deep Learning (DL) Residual Neural Network were derived to predict NSTEMI Type 1. The DL model was pretrained on 180,522 hospitalized patients matched with ICD-10 codes. All models were derived on a prospective cohort comprising 7,591 patients presenting with suspected MI to the Emergency Department (ED). Evaluation was conducted on a separate prospective cohort of 1,409 patients where 5-minute resting ECGs were available at presentation to the ED. Final diagnoses for the derivation and evaluation cohorts were centrally adjudicated by two independent cardiologists according to the Fourth Universal Definition of MI. Continuous 5-minute ECGs were partitioned into 10s segments, yielding 30 probabilities per patient. The intra-patient variability was defined as the 95% prediction interval across 5 minutes. The impact of intra-patient variability on MI diagnosis and clinical risk assessment was evaluated by comparing the first segment to the 5-minute min/max probabilities, quantifying reclassification across high-risk (predefined positive predictive value 70%) and low-risk (predefined sensitivity 99.5%, negative predictive value 99.0%) thresholds.
Results
Area under the curves of the models were 0.75 [95%-CI 0.74–0.75] for LR, 0.75 [0.74–0.76] for XGBoost, and 0.79 [0.78–0.80] for DL. Higher patient median probability correlated with larger intra-patient variability (Spearman’s: LR 0.75, XGBoost 0.85, DL 0.84; all p<0.001). Intra-patient variability significantly affected MI diagnosis and clinical risk assessment within the 5min ECG. Comparing the first segment to the minimum probability segment: the low-risk cohort increased from 3.0% to 5.9% for LR, 12.3% to 20.8% for XGBoost, and 14.4% to 24.4% for DL. Comparing the first segment to the maximum probability segment: the high-risk cohort increased from 0.1% to 0.4% (LR), 1.0% to 3.2% (XGBoost), and 0.6% to 2.3% (DL).
Conclusion
Model predictions for NSTEMI Type 1 exhibit significant intra-patient variability across serial ECGs. Higher patient median probability correlated with increased intra-patient variability, leading to frequent reclassification of clinical risk. These findings highlight a key limitation of current ECG-based MI-prediction models as clinicians cannot rely on a single 10s ECG prediction for diagnostic certainty.Central IllustrationResults