Beyond model accuracy: uncertainty-driven quality control for population-scale CMR segmentation
M Ferrari, K K Mohammed, M Urtis, C K Cassidy, N Narula, E Arbustini, D Zhi, J NarulaAbstract
Background
Large-scale Cardiac Magnetic Resonance (CMR) repositories are widely used for population health phenotyping. While automated cardiac segmentation enables the extraction of interpretable functional markers, current pipelines implicitly assume image validity and standard deep learning models may attempt to segment non-diagnostic scans with high confidence.
Purpose
We aim to develop a scalable quality control (QC) mechanism that can identify invalid cases and prevent their propagation through automated pipelines, avoiding invalid measurements and biasing downstream statistical analyses.
Methods
Expert manual annotations on 2-, 3-, and 4-chamber Long-Axis Cine-CMR views from a subset of randomly selected 100 UK Biobank subjects were used to train (N=80) and test (N=20) a whole-heart binary segmentation model. An automated QC pipeline was developed for population-scale deployment. Test-Time Augmentation was applied at inference to generate voxel-wise entropy maps as a measure of uncertainty. A multidimensional feature vector was computed across the cardiac cycle, capturing multiple aspects of segmentation consistency including motion consistency and shape regularity. These features were analyzed using an unsupervised Isolation Forest algorithm; the anomaly score threshold was determined empirically its distribution in the full cohort. Manual validation was performed on 107 randomly selected CMR studies (55 flagged, 52 non-flagged). Each study underwent expert review grading segmentation quality and clinical impact on a three-point scale (0: no impact; 1: minor; 2: major); studies with any detected impact were considered significant failures.
Results
Across different segmentation models, mean Dice scores always reached 0.95, showing performance saturation under conventional accuracy metrics. At population scale (N = 64,628), the QC system identified 2,915 cases as statistical outliers (4.5%). Manual validation revealed that 38.2% of flagged studies corresponded to intrinsically non-diagnostic or degraded acquisitions. Clinically significant segmentation errors were present in 72.7% of flagged cases, compared to only 3.8% of non-flagged cases. The pipeline achieved a sensitivity of 95.2% and a negative predictive value of 96.2% for detecting clinically relevant failures.
Conclusions
These findings demonstrate that high segmentation accuracy on controlled test sets may be insufficient to guarantee pipeline reliability at population scale. Therefore, we developed an automatic, uncertainty-driven QC system that can identify degraded acquisitions and model errors when manual labeling is unfeasible. While future work should address the restricted segmentation training set and single-rater design, establishing a scalable QC is essential to ensure the integrity of derived phenotypes and the reliable integration of AI-driven pipelines into research.Graphical Abstract