Rethinking false positives: can ECG-based diagnostic ML models predict future heart failure risk?
W Sun, S Kalmady, M Paganuzzi, F McAlister, J Ezekowitz, A Hindle, R Greiner, P KaulAbstract
Background
Many machine-learned diagnostic models attempt to predict whether a patient currently has some specific diseases. However, evaluation often ends at cross-sectional performance metrics, overlooking whether prediction errors, particularly the false positives (FP), encode prognostic information about future disease development and outcomes.
Purpose
This study examines whether errors (FP) from an electrocardiogram (ECG)-based machine-learned model for heart failure (HF) prediction can help forecast subsequent HF events and mortality.
Methods
We trained a deep learning model for HF detection on 964,741 ECGs (146,446 patients), then evaluated it on a test cohort of 297,773 ECGs (97,631 patients) from Alberta, Canada. Patients were stratified into true positive (TP), FP, false negative (FN), and true negative (TN) groups based on ground-truth labels of concurrent physician-confirmed HF diagnosis and then followed longitudinally for subsequent HF and all-cause mortality, both in the full test cohort and among HF-free patients with no prior or current HF diagnosis. Competing risk for HF and composite time-to-event analyses for both HF and all-cause mortality were applied. BNP/NT-proBNP levels were compared, which were not included in model training. Analyses were replicated in the independent MIMIC-IV dataset with 795,546 ECGs (159,608 patients).
Results
Competing risk analysis showed clear separation of cumulative incidence curves across prediction error strata, following a graded risk pattern TP > FP > FN > TN. FP cases showed significantly higher HF risk than TN (age & sex-adjusted Hazard Ratios in Alberta: 4.19 [3.92–4.47]; Alberta HF-free: 3.88 [3.52–4.28]; MIMIC: 2.78 [2.58–3.00]; MIMIC HF-free: 2.61 [2.41–2.83]; all p < 0.001). With respect to biomarker profiles, FP cases had substantially higher BNP or NT-proBNP levels than TN (Alberta BNP: 326 [169–613.5] vs 84 [35–208]; MIMIC NT-proBNP: 2048 [719–5664] vs 229 [76.25–742]; median [IQR], all p < 0.001). In terms of absolute risk of incident HF in HF-free cohorts, FP cases demonstrated 5.35%, 9.17%, and 11.89% (Alberta) and 10.7%, 18.51%, and 23.44% (MIMIC) higher risk compared to TN at 1, 3, and 5 years (all FDR-adj p < 0.001). Similar graded risk relationships were also identified for the composite outcome of HF occurrence and all-cause mortality (all p < 0.001). Across cohorts, FP outcomes were comparable to FN (adj HR in Alberta 1.12 [0.99–1.26]; MIMIC 0.70 [0.59–0.83]).
Conclusions
We observed a consistent FP–TN risk separation for future HF events across the overall and HF-free groups, in both Alberta and MIMIC datasets, further supported by higher BNP/NT-proBNP in FPs. These results suggest that diagnostic AI models, even when misclassifying cases, can carry prognostic signals consistent with subclinical disease. Systematic error introspection may therefore improve interpretability and clinical trust in cardiovascular AI.HF CIF at 1/3/5-yr by prediction strataHF CIF and KM by prediction strata