Accuracy and Quality Assessment of Artificial Intelligence Platforms and a Hybrid Evidence-Assisted Model in Responding to Drug Information Questions
Nasr Issa, Evan Colmenares, Andrew Douglas, Gregory Heindel, Lacey M Lee, Erin Maxwell, Erin Mazur, Zach Pedretti, John Killian Rodgers, Kirsten Smith, Ina LiuAbstract
Purpose
This descriptive report evaluated the accuracy and response quality of ChatGPT 5.1, OpenEvidence, Clair, and a hybrid evidence-assisted ChatGPT 5.1 model in responding to drug information questions, using drug information specialist–developed reference responses as the comparator.
Summary
Twenty drug information questions were selected from the UNC Health Drug Information Center archive from July 2021 through July 2025, stratified across 10 categories, and classified as simple or complex. Responses were generated in November 2025 using ChatGPT 5.1, OpenEvidence, Clair, and a hybrid evidence-assisted model in which drug information specialist–curated evidence packets were provided to ChatGPT 5.1. Drug information specialist–developed reference responses included a definitive answer, evidence synthesis, a recommendation when applicable, and verifiable references. Deidentified and randomized responses were evaluated through a blinded two-phase review process assessing accuracy and response quality. ChatGPT 5.1 had the numerically highest accuracy at 70%, followed by the hybrid evidence-assisted model at 60%, OpenEvidence at 40%, and Clair at 15%. Drug information specialist–developed reference responses and OpenEvidence had the highest mean overall quality performance at 82.5% each.
Conclusion
AI platforms demonstrated variable accuracy and quality when responding to drug information questions. In this limited sample, ChatGPT 5.1 had the numerically highest accuracy, while OpenEvidence had the highest mean overall quality performance among AI-generated responses. Despite strengths in organization and readability, limitations in accuracy, reference identifiability, and evidence synthesis support the continued need for drug information specialist oversight.