Multimodal large language models miss two-thirds of STEMIs on electrocardiograms and concentrate false-positive diagnoses in STEMI mimics and normal tracings
H Kamalanathan, K Kaleeny, P Sahai, D Downes, A R Wilson-Smith, T Chauraya, M Mo, K Rajesh, S Christoforou, V Premkumar, A Nielssen, D Ng, J MillhouseAbstract
Background
Clinicians increasingly use commercially available multimodal large language models (LLMs), without formal validation, to interpret 12-lead electrocardiograms (ECGs). ST-elevation myocardial infarction (STEMI) represents the highest-acuity ECG diagnosis: missed STEMIs delay reperfusion, whilst false positives trigger unnecessary catheterisation laboratory activation. Although recent studies have examined individual LLMs for myocardial infarction detection, no head-to-head comparison of current-generation models exists. Additionally, none has characterised which ECG pathologies most frequently trigger false-positive STEMI diagnoses.
Purpose
We hypothesised that current-generation multimodal LLMs lack sufficient diagnostic accuracy for STEMI detection to support safe clinical use. This study aimed to compare three LLMs head-to-head and identify which ECG pathologies most frequently trigger false STEMI diagnoses.
Methods
We presented 599 real-world 12-lead ECGs sourced from a published five-volume teaching case series to three multimodal LLMs (ChatGPT-5.1, Claude 4.5, Gemini 3) under minimalist and structured prompt conditions, generating 3,995 interpretive reports. Textbook-assigned expert interpretations served as the reference standard. Two authors independently classified each report as STEMI-positive or STEMI-negative; 87 of 599 unique ECGs (14.5%) were STEMI-positive. We evaluated sensitivity, specificity, positive predictive value (PPV), and false-positive and false-negative rates by model and prompt condition, then cross-referenced false-positive reports against the true ECG pathology.
Results
All three LLMs demonstrated critically low STEMI sensitivity: 28.1% for ChatGPT, 30.4% for Claude, and 40.5% for Gemini (overall 33.2%), corresponding to a two-thirds miss rate (237/355 STEMI reports undetected). The overall false-positive rate reached 21.9% (798/3,640 non-STEMI reports), with a PPV of only 12.9%. Structured prompting shifted rather than resolved errors: for ChatGPT, it tripled the false-positive rate (12.6% to 38.3%) while raising sensitivity from 18.0% to 39.6%. False-positive STEMI calls clustered in normal ECGs (26.1%) and in recognised STEMI mimics: pericarditis (34.3%), old myocardial infarction (28.9%), ventricular strain (28.4%), and early repolarisation (27.1%) (Figure 1). No model achieved both sensitivity and specificity above 50% under any condition (Figure 2).
Conclusion
Multimodal LLMs missed two-thirds of STEMIs and fabricated the diagnosis in one-fifth of non-STEMI ECGs, with false positives concentrated in the very STEMI mimics and normal tracings where clinical discrimination matters most. These findings confirm our hypothesis and demonstrate that general-purpose LLMs cannot safely support STEMI triage. As these tools are readily accessible to clinicians at all training levels, prospective studies are urgently needed to define domain-specific safety thresholds.Figure 1Figure 2