Evaluation of Large Language Model Concordance With AAOS Clinical Practice Guidelines for Carpal Tunnel Syndrome
Jonathan Spaan, John T. Streepy, Alexander L. Hornung, Kyleen Jan, Michael Murray, Xavier SimcockBackground:
Large language models (LLMs) such as ChatGPT, Gemini, Claude, and Mistral are increasingly used to obtain medical information, but their accuracy in carpal tunnel syndrome (CTS) management is uncertain. This study evaluated whether LLM-generated recommendations align with American Academy of Orthopaedic Surgeons (AAOS) clinical practice guidelines (CPGs).
Methods:
Four LLMs (ChatGPT-4, Gemini, Claude-3, Mistral-7B) were queried with standardized prompts derived from AAOS CTS CPGs. Each prompt was asked 3 times per model, and responses were synthesized. Two blinded reviewers graded outputs as concordant, discordant, or indeterminate, with disagreements resolved by consensus. Concordance distributions were compared using χ 2 testing with Bonferroni correction. Inter-rater reliability was measured with Cohen’s κ.
Results:
Across 252 recommendations, 72.6% were concordant, 11.5% discordant, and 15.9% indeterminate. ChatGPT achieved the highest concordance (92.1%) and lowest discordance (1.6%), while Mistral performed poorest (52.4% concordance, 17.5% discordance). Reviewer agreement was excellent (κ = 0.91).
Conclusions:
Over 1 in 4 LLM-generated CTS recommendations were not concordant with AAOS CPGs. ChatGPT outperformed other models, though variable performance underscores the need for clinician oversight and careful validation before clinical use.