Accuracy of atrial fibrillation labels in the MIMIC-IV ECG database: implications for artificial intelligence model development
I Valasakis, D Connelly, P MacfarlaneAbstract
Methods
We randomly sampled 700 consecutive ECGs (633 unique patients) labelled as “Atrial Fibrillation” (AF) from the MIMIC-IV database and had them independently reviewed by two experts in ECG and arrhythmia analysis. One batch of 100 ECGs was dual-reviewed for inter-rater agreement assessment. The remaining 600 ECGs were divided into two batches and each reviewer assessed one batch of 300. 121/600 difficult cases underwent cross-review by the alternate reviewer. Diagnoses were categorised as AF, atrial flutter (AFL), sinus (SR), or other arrhythmia.
Results
Of 700 ECGs labelled as atrial fibrillation, only 458 (65.4%) showed AF on expert review. The remaining 242 (34.6%) were mislabelled: 176 (25.1%) showed SR (some with extrasystoles or AV block), 37 (5.3%) showed AFL, and 29 (4.1%) showed other arrhythmias or were unclassifiable. The mislabel rate was consistent across batches (range 26–39%), regardless of reviewer. Initial inter-rater agreement on broad diagnostic category was 84% (Cohen’s kappa 0.63); initial agreement on the binary AF versus non-AF classification was 92%. The predominant mislabel pattern - SR classified as AF - accounted for 72.7% of all errors, suggesting systematic failure of the automated interpretation algorithm rather than borderline diagnostic cases (see examples).
Conclusions
Over one-third of ECGs labelled as atrial fibrillation in this sample from MIMIC-IV do not show AF on expert review, with the majority of the mislabelled ECGs showing SR. This substantial mislabel rate has critical implications for the many published AI studies, often written by non-medical authors, that use such databases as ground truth. Models trained on contaminated labels may learn spurious associations, while performance metrics derived from validation against these labels are unreliable. Our findings argue that curated, expert-verified labels are essential for credible AI-based ECG research, and that published performance claims based on non-curated database labels should be interpreted with considerable caution.ECG SR with PACs mislabelled as AF.jpgECG AV block mislabelled as AF.jpg