Uncertainty-Aware AI-Assisted Diabetic Retinopathy Grading from Fundus Images with Ordinal Conformal Prediction

Background: Artificial intelligence (AI) systems for diabetic retinopathy (DR) grading require reliable uncertainty estimates when applied to fundus images outside the development dataset. We evaluated whether conformal prediction can provide structured set-valued outputs and whether internal uncertainty calibration remains reliable during external evaluation. Methods: EfficientNet-B0, ResNet-50, and Vision Transformer (ViT-Base) classifiers were trained on APTOS 2019. Split-conformal predictors were calibrated exclusively on held-out APTOS images using three categorical scores, LAC, APS, and RAPS, and an ordinal score restricted to adjacent severity grades. Performance was assessed internally on APTOS and externally on IDRiD, with an independent five-seed ViT replication extending evaluation to Messidor-2, at target coverages of 90% and 95%. Results: Coverage was approximately nominal internally but decreased on both external datasets. On IDRiD, the largest deficit occurred for severe DR (grade 3). The ordinal method produced contiguous intervals in 100% of cases and achieved the highest grade-3 coverage in every tested backbone–risk configuration. For ResNet-50 at 95% target coverage, grade-3 coverage increased from 0.750 with APS to 0.945 with the ordinal method, while average set size increased from 2.89 to 3.03. In the independent ViT replication on Messidor-2, ordinal marginal coverage exceeded APS at both targets (0.676 versus 0.632 and 0.741 versus 0.714). Conclusions: Internal calibration did not ensure reliable class-specific uncertainty during external evaluation. Ordinal prediction sets improved structural coherence and mitigated severe-grade undercoverage, but did not restore formal coverage guarantees after dataset shift.