OpenEvidence versus Six Leading Large Language Models in Refractive Surgery: Does Curated Medical Evidence Confer an Advantage?
Avi Wallerstein, Mathilde Gallant, Mathieu GauvinPurpose:
To assess the accuracy of OpenEvidence relative to six leading large language models (LLMs) in answering refractive surgery multiple-choice questions, and to compare their performance with users of the American Academy of Ophthalmology (AAO)
Setting:
Not applicable.
Design:
Randomized questionnaire-based study using 225 questions from the refractive surgery section of the AAO
Methods:
OpenEvidence and 6 LLMs (ChatGPT-4o, ChatGPT-5, Claude, Copilot, Gemini, and Grok) answered the same set of multiple-choice questions. Their accuracy was defined as the proportion of correct responses. Performance from 1,983
Results:
OpenEvidence achieved the lowest accuracy (76%) and performed significantly worse (P<0.05) than all six general-purpose LLMs. Claude achieved the highest accuracy (92%), followed by Grok (89.8%). ChatGPT-5, Copilot, and Gemini each scored an accuracy of 88%, and ChatGPT-4o achieved 86.2%. All models’ lower 95% confidence interval bounds exceeded the 60% passing threshold. The average score of
Conclusions:
OpenEvidence demonstrated significantly lower accuracy than the six evaluated general-purpose LLMs and did not exceed the BCSC Self-Assessment program user benchmark on this standardized refractive surgery multiple-choice assessment. This task-specific difference was not fully explained by response-format noncompliance and should not be interpreted as direct evidence of globally inferior medical knowledge or performance in open-ended clinical applications.