How to read a study of a diagnostic artificial intelligence tool: a narrative review and checklist for clinicians

Diagnostic tools based on artificial intelligence (AI) now read mammograms, colonoscopy video, chest radiographs, electrocardiograms and retinal photographs in routine care, yet most published evaluations are retrospective, single-centre comparisons on curated data; in one systematic review, only 2 of 83 published studies comparing deep learning with clinicians were randomised trials. This narrative review, based on a PubMed search to September 2026, gives clinicians a checklist for appraising such studies. Six questions expose most weaknesses: whether the patients resemble the intended population; whether the reference standard was sound and training and test data were kept apart; whether the tool was validated externally; whether discrimination is reported with calibration and subgroup performance; whether clinicians using the tool were compared with clinicians without it in real workflow; and whether a patient outcome was measured. Worked examples show the range of evidence. In the Mammography Screening with Artificial Intelligence (MASAI) trial of 105,915 women, AI-supported reading detected 29% more cancers, reduced screen-reading workload by 44% and gave a non-inferior interval cancer rate (proportion ratio 0.88, 95% confidence interval 0.65 to 1.18). By contrast, a widely deployed sepsis prediction model had an area under the receiver operating characteristic curve of 0.63 (95% confidence interval 0.62 to 0.64) on external validation. Computer-aided polyp detection increases adenoma detection in trials, but guideline panels disagree because no reduction in cancers has been shown. Clinicians should treat a diagnostic AI tool without external, preferably prospective, validation and evidence of patient benefit as experimental.