An explainable calibrated concept bottleneck model with evidence-constrained large language model reporting for multi-label valvular heart disease screening from 12-lead ECGs
H Anjum, N Perwaiz, S sultana, T Anwar Lashari, Z AhmadAbstract
Background/Introduction
Valvular heart disease (VHD) affects over 100 million people worldwide, but up to half of cases remain undiagnosed because echocardiography access is limited. Deep learning can detect VHD from 12-lead electrocardiograms (ECGs), but all published models lack explainability. When large language models (LLMs) generate reports from unconstrained neural embeddings, they risk fabricating ECG findings absent from the input signal.
Purpose
We aimed to build an explainable screening pipeline that a cardiologist can audit at every step. We developed a concept bottleneck model (CBM) that routes each prediction through 15 named ECG concepts. We applied post-hoc calibration and designed an evidence token mechanism that limits LLM input to verified concept activations alone.
Methods
We trained on 100,643 paired 12-lead ECGs and echocardiography reports from a single centre. The CBM combined a ResNet1d-18 encoder (8.8 M parameters) with seven tabular features via late fusion. We used three training phases: concept stabilisation, valve alignment, and joint fine-tuning (20, 30, and 10 epochs). The concept layer predicted 15 ECG abnormalities (e.g., left ventricular hypertrophy and atrial fibrillation). A learned attention head inferred four valve targets: aortic stenosis (AS), aortic regurgitation (AR), mitral regurgitation (MR), and tricuspid regurgitation (TR). We fitted post-hoc calibration (Platt for AS, AR, and MR; isotonic for TR) on a held-out set (n = 4,626). We discretised concept activations into structured evidence tokens. A domain-adapted LLM (BioMistral-7B, 4-bit quantised) generated template-constrained reports from these tokens alone.
Results
We evaluated on a patient-level test set (n = 5,442; no patient overlap with training). We report area under the receiver operating characteristic curve (AUROC) with 95% confidence intervals (CIs). The CBM reached AUROC: AS 0.867 (0.848–0.884), TR 0.836 (0.814–0.856), MR 0.807 (0.784–0.828), and AR 0.757 (0.709–0.801); mean 0.817. Post-hoc calibration reduced mean expected calibration error (ECE) from 0.242 to 0.006 (97% reduction; mean slope 0.90). At 90% specificity, sensitivity reached 0.57 for AS, 0.57 for TR, 0.46 for MR, and 0.30 for AR. The explainable concept layer reached mean AUROC 0.866. Across 5,442 reports, vocabulary compliance reached 100% (0 violations; Wilson 95% CI: 0.000–0.001).
Conclusion(s)
A CBM for multi-label VHD screening from ECGs delivers explainable predictions with calibrated probabilities. The 15-concept layer lets a cardiologist trace each valve prediction back to named ECG findings. Post-hoc calibration cuts ECE by 97% without changing discrimination. The evidence token architecture restricts LLM input to a closed vocabulary; test-set validation confirms zero hallucinations across 5,442 LLM-generated reports. This single-centre proof-of-concept warrants prospective multi-centre validation with clinician review of generated reports.system architectureEcho CBM Gap Analysis and result table