Clinicians need to review only half of LLM outputs to preserve diagnostic accuracy: the SCOUT randomised crossover trial
Z Q Ba, M He, H R He, Q A Fu, J R Lai, R Q Zhang, X J Gao, Y J WuAbstract
Introduction
Large language models (LLMs) are increasingly deployed in clinical workflows, yet requiring clinician review of every AI output negates the efficiency gains. Conversely, unverified predictions risk diagnostic errors. No validated framework exists to selectively route only unreliable LLM outputs for human review.
Purpose
We developed and validated SCOUT (Scalable Clinical Oversight via Uncertainty Triangulation), a model-agnostic meta-verification framework, and assessed its clinical utility in a prospective multi-reader randomised crossover trial.
Methods
SCOUT triangulates three orthogonal signals to identify unreliable LLM predictions: (1) model heterogeneity, whether an architecturally distinct auxiliary model agrees; (2) stochastic inconsistency, whether repeated inference yields divergent outputs; and (3) reasoning critique, whether a checker model detects logical flaws in the chain-of-thought. High-uncertainty cases are deferred to clinicians; low-uncertainty cases are accepted automatically. The optimal configuration was identified via Pareto optimisation on a retrospective discovery cohort (n=405). The locked system was validated on coronary heart disease (CHD) subtyping using MIMIC-IV (n=1,866) and a prospective multicenter cohort (n=110), and on diseased vessel counting (n=286). Clinical utility was assessed in a prospective randomised crossover trial: seven clinicians (2 junior residents, 3 senior residents, 2 attending physicians) each reviewed all 110 CHD cases under both standard manual review and SCOUT-assisted selective review. The primary endpoint was review time; the secondary endpoint was diagnostic accuracy.
Results
Across validation cohorts, SCOUT reduced cases requiring human review to 17.8-54.5% while achieving final accuracies of 99.1-100.0% (efficiency ratio 1.6-5.6x). In the crossover trial, SCOUT-assisted review reduced mean time per case by 59.3% (0.37 vs 0.92 min; mean difference -0.54 min, 95% CI -0.76 to -0.36; P=0.016). Diagnostic accuracy was comparable to full manual review (90.8% vs 89.4%; P=0.596). Stratified analysis revealed a skill-level equaliser effect: for junior residents, SCOUT functioned as a safety net, improving accuracy from 78.7% to 85.7%; for attending physicians, accuracy was stable (97.2% vs 96.4%) while computational return on investment reached 8.5-fold. Cost per case decreased by 46-52% across all strata.
Conclusion
SCOUT enables clinicians to review only a subset of LLM outputs while preserving diagnostic accuracy. In a randomised crossover trial, the framework reduced review time by 59% and served as a safety net for junior clinicians and an efficiency multiplier for senior physicians, offering a scalable blueprint for responsible clinical AI deployment.Study designKey Results