Diagnostic and Triage Performance of Large Language Models in Hand and Wrist Complaints: A Standardised Vignette Evaluation
Callum M. Counihan, Rory J. D. Duncan, Max G. Bullock, Ravi MallinaBackground:
Large language model (LLM) chatbots are widely accessible and increasingly used by the public for health information and as a tool to diagnose health conditions. While this may improve access to information, incorrect responses may lead to false reassurance and treatment delays. Evidence specific to ‘patient-voice’ descriptions in orthopaedic hand and wrist presentations is limited.
Methods:
Fictitious first-person ‘patient-voice’ vignettes were created for common elective hand conditions (carpal tunnel syndrome, trigger finger, de Quervain tenosynovitis, thumb carpometacarpal osteoarthritis), common traumatic hand/wrist injuries (mallet finger, scaphoid, little finger metacarpal and distal radius fractures), and more minor injuries. Five vignettes were created per condition (20 elective; 20 trauma), and 10 minor-injury vignettes (total 50). Each vignette was entered into 3 freely available consumer LLMs (ChatGPT, Google Gemini, Microsoft Copilot). The first response was assessed for primary diagnostic and triage accuracy.
Results:
Overall diagnostic accuracy was 84.7%. Diagnostic accuracy differed by presentation type: 96.7% for elective, 65.0% for trauma, and 100% for minor injuries (
Conclusions:
Free-to-use LLM chatbots demonstrated high diagnostic and triage accuracy for minor injuries and elective hand conditions, but lower performance for traumatic presentations, with under-triage increasing with acuity. Given the potential for false reassurance and delayed care in time-critical injuries, LLM outputs should not be relied upon for triage decisions. Prospective evaluation using real-world patient queries is warranted.