Electrocardiogram complexity determines large language model diagnostic accuracy: a pathology-stratified analysis of 599 cases across 22 diagnostic categories
H Kamalanathan, P Sahai, D Downes, K Kaleeny, A R Wilson-Smith, V Premkumar, A Nielssen, M Mo, T Chauraya, K Rajesh, S Christoforou, W Ho, J MillhouseAbstract
Background
Clinicians increasingly use multimodal large language models to interpret 12-lead electrocardiograms (ECGs) from images, without formal validation of their diagnostic accuracy. Benchmark studies report overall accuracy but not which pathologies these models interpret well or poorly. This gap prevents targeted model improvement and makes it impossible to set pathology-specific safety thresholds for clinical use.
Purpose
We hypothesised that large language model accuracy for ECG interpretation degrades with increasing ECG complexity, particularly when multiple pathologies require integrative reasoning. We aimed to quantify performance across 22 diagnostic categories and identify which interpretive domains drive errors in each.
Methods
Three multimodal large language models (ChatGPT-5.1, Claude 4.5, Gemini 3) each interpreted 599 real-world 12-lead ECGs from a published five-volume teaching case series under minimalist and structured prompt conditions (3,594 reports total). We classified ECGs into 22 predominant pathology categories and scored each report using a prespecified, clinically weighted rubric (0-100) across ten interpretive domains. We fitted mixed-effects models with pathology, model identity, and prompt style as fixed effects, ECG case as a random intercept, and a domain-by-pathology interaction term to test whether specific domains varied in difficulty across diagnoses.
Results
Performance differed significantly across pathology categories (F=7.15) but was uniformly low (Figure 1). Normal ECGs scored highest yet reached only 63.6/100. Combined rhythm and conduction abnormalities scored lowest: narrow complex tachycardia with heart block 38.5 and atrial fibrillation/flutter with heart block 39.2 - a 25-point gap. Single-pathology ECGs outperformed multi-pathology ECGs (48.4 vs 42.1). A significant domain-by-pathology interaction (F=18.82) showed that errors were domain-specific: chamber hypertrophy recognition exceeded 70% across most pathologies, but final diagnostic synthesis fell below 5% in nearly all categories. Even for normal ECGs, only 20% of reports achieved a correct final diagnosis.
Conclusion
Large language model accuracy for ECG interpretation varies substantially by pathology, but no diagnostic category achieved clinically acceptable performance. Models reliably recognise isolated waveform features such as chamber hypertrophy but fail at integrative diagnostic reasoning across all pathology categories, with accuracy declining predictably as ECG complexity increases. Pathology-specific benchmarks are needed to identify which diagnostic categories require targeted model improvement and to define domain-specific safety thresholds before these tools are integrated into clinical ECG interpretation workflows.