Accuracy of Large Language Model Chatbots in Retrieving
MRI
Safety Conditions for Medical Implants
Ariel Kerpel, Shir Tal, Mark Waserman, Christine Dan Lansman, Anat Berkovitch, Eyal Klang, Ronen Spierer, Shai Shrot, Emanuel Kanal, Gahl Greenberg ABSTRACT
Background
Retrieving MRI safety conditions for implanted devices is increasingly complex and time‐consuming.
Purpose
To evaluate the accuracy of internet‐enabled large language model (LLM) chatbots in retrieving implant‐specific MRI safety conditions, using manufacturer documentation as the reference standard.
Study Type
Prospective comparative evaluation in a constructed device cohort, followed by a retrospective clinical pilot.
Population
Fifty purposively selected device configurations and 20 patients (10 with cardiac and 10 with noncardiac devices) referred to the MRI Safety Committee.
Field Strength
No field‐strength or sequence restrictions.
Assessment
Standardized single‐shot prompts requested eight parameters from ChatGPT‐5.0, Gemini 2.5 Pro, and Claude Sonnet 4.5: MRI safety label, field strengths, eligible regions/centering, transmit‐coil restrictions, RF exposure limits, maximum spatial gradient, maximum gradient slew rate, and scan duration. Two radiologists (10 and 25 years' experience) and an MRI technologist (32 years) independently scored 1200 parameter‐level outputs against manufacturer Instructions for use and technical manuals using a 5‐point scale (5, fully correct; 1, substantively incorrect or fabricated); scores of 4–5 indicated accuracy. In the pilot, 160 outputs from ChatGPT‐5.6, a newer version of the best‐performing chatbot, were scored identically.
Statistical Tests
Intraclass correlation coefficient; Gwet's first‐order agreement coefficient (AC1); Friedman test; Wilcoxon signed‐rank tests; device‐clustered bootstrap 95% confidence intervals (CIs); two‐sided significance threshold, p < 0.05.
Results
Mean ordinal scores were 4.76 (ChatGPT), 4.20 (Gemini), and 4.17 (Claude); ChatGPT scored significantly higher. Respective pooled parameter‐level accuracies were 0.94 (95% CI, 0.89–0.98), 0.79 (0.71–0.86), and 0.78 (0.69–0.87). Pilot accuracy with ChatGPT‐5.6 was 0.98 (95% CI, 0.94–1.00).
Data Conclusion
Internet‐enabled LLM chatbots correctly retrieved up to 94% of MRI safety parameters in the constructed cohort and 98% in the clinical pilot. Errors occurred even with the best‐performing chatbot, requiring direct verification against current manufacturer documentation before clinical use.
Evidence Level
Stage 2.
Technical Efficacy
Stage 2.