Evaluating Frontier Large Language Models for AI-Assisted Differential Diagnosis in Gastroenterology: A Comparative Study of Diagnostic Performance and Repeated-Run Consistency

Background: Large language models (LLMs) have shown potential for clinical decision support. However, the diagnostic performance of newly released frontier models—including GPT-5 and Gemini 2.5 Pro—in gastroenterology remains insufficiently evaluated. This study evaluated the diagnostic performance and repeated-run consistency of four frontier LLMs—DeepSeek-V3.1, GPT-5, GPT-4.1, and Gemini 2.5 Pro—in challenging gastroenterology cases. Methods: This cross-sectional comparative study evaluated four LLMs (DeepSeek-V3.1, GPT-5, GPT-4.1, and Gemini 2.5 Pro) using 50 challenging gastroenterology cases retrieved from PubMed. Cases published after February 2025 were selected to minimize the potential risk of training-data contamination and prior model exposure. Each case was structured into a standardized prompt comprising clinical history, examination findings, laboratory results, and imaging data. Three independent runs were performed for each case. First-run outputs were used to assess ordinal primary diagnostic scores and differential-diagnosis quality, whereas all three runs were used to assess repeated-run consistency. Outputs were anonymized and scored by a gastroenterologist blinded to model identity. Results: Gemini 2.5 Pro achieved significantly higher ordinal primary diagnostic scores than the other three models (all Holm-adjusted pairwise p < 0.01). Exact-match rates were 46% for Gemini 2.5 Pro, 22% for GPT-5, and 18% each for GPT-4.1 and DeepSeek-V3.1. All four models had a median differential-diagnosis quality score of 4; the Friedman test did not detect a statistically significant difference among the models (p = 0.725). Complete agreement in the primary diagnosis across all three runs was observed in 68–78% of cases, depending on the model. No statistically significant differences in repeated-run consistency scores were detected among the models. Conclusions: In this investigator-selected benchmark of 50 published gastroenterology cases, Gemini 2.5 Pro achieved higher ordinal primary diagnostic scores than the other three models. No statistically significant differences were detected in differential-diagnosis quality or repeated-run consistency. These nonsignificant findings do not establish equivalence. Prospective evaluation in clinical settings is required before drawing conclusions about clinical effectiveness.