External validation, robustness, and clinical error profiling of a multimodal electrocardiogram model for structural heart disease reveals a dominant shared prediction component
G Riedemann, P Hempel, M Eberhardt, J Spauschus, J Fischer, C Schmidt, S Brandenburg, N SpicherAbstract
Background/Introduction
Multitask ECG models are usually evaluated per label, but whether outputs are disease-specific or dominated by a shared component and the implications for error patterns and artefact robustness, remains unclear.
Purpose
To evaluate output-space structure, external validity, artefact robustness, and clinically relevant failure modes to inform emergency department (ED) deployment.
Methods
EchoNext Mini [1], a pretrained multimodal ECG model (12-lead ECG + 7 tabular covariates; 12 probability outputs for structural heart disease), was evaluated in (i) an internal ECG-echo cohort [2](n=10,068) and (ii) an external ED cohort [3] (n=1,000; cardiologist-adjudicated labels for LVEF≤45%, aortic stenosis, and tricuspid regurgitation). Discrimination was assessed using the area under the receiver operating characteristic curve (AUROC). Prediction-space structure was quantified by inter-output correlations and principal component analysis (PCA) and related to vectorcardiographic (VCG) morphology. Error profiles were compared. Muscle artefact robustness was tested in 1,000 internal ECGs by gradually adding high-pass filtered noise.
Results
Mean internal AUROC was 0.802 (range 0.734-0.866). External AUROCs were 0.769 (95% CI 0.74-0.80) for LVEF≤45%, 0.702 (0.64-0.76) for aortic stenosis, and 0.745 (0.7-0.79) for tricuspid regurgitation, corresponding to performance decrements of ΔAUROC 0.083, 0.157, and 0.088, respectively. PCA showed that PC1 explained 85.5% of prediction variance internally and 79.9% externally. Ground-truth labels were weakly correlated (median |r|=0.10), whereas predicted probabilities were strongly correlated (median |r|=0.89), suggesting a dominant shared prediction component. Externally PC1 separated disease strongly for LVEF≤45% and tricuspid regurgitation (p<1×10⁻¹⁸) but modestly for aortic stenosis (p=0.032). The QRS–T angle was the top single vectorcardiographic feature (e.g., AUROC 0.744 for LVEF≤45%; Cohen’s d=0.92). Robustness to muscle artefact was high overall (mean ΔAUROC −0.005 at 157 µV RMS), but across the 12 internal outputs pericardial effusion was most sensitive (ΔAUROC 0.159), while aortic stenosis was minimally affected (ΔAUROC 0.010). False positives were enriched in older patients with electrical remodelling (wider QRS, prolonged QTc, higher QRS-T angle), whereas false negatives clustered in younger patients with narrower QRS and lower QRS-T angle.
Conclusion(s)
The model generalised to external ED data with moderate, label-dependent performance loss. Multitask outputs were dominated by a shared prediction component aligned with VCG morphology, supporting broad screening/triage but limiting disease-specific inference from individual heads. Error profiling suggests likely false alarms in older remodelling phenotypes and missed disease in younger patients with minimal electrical changes. Muscle artefact was a minor threat overall, except for pericardial effusion.Figure 1- Collapse of predictions spaceFigure 2- external validation and corruption