Evaluation of Large Language Models for Emergency Triage in Urology: A Multi-Model Study

Objective: To evaluate the triage accuracy, response quality, safety, and repeatability of widely used large language models (LLMs) in urologic emergencies and to examine their potential clinical utility and risks. Methods: We developed 50 standardized clinical vignettes covering 10 categories of urologic emergencies, including testicular torsion, acute urinary retention, and Fournier gangrene. The set comprised high-risk vignettes requiring immediate emergency care, borderline vignettes, and atypical vignettes containing misleading cues. Six commercially available LLMs—Kimi 3.0, DeepSeek V4, Qwen 3.8-MAX, GPT-5.6 Sol (ChatGPT), Claude Opus 4.8, and Ant Afu (AQ)—were evaluated in Chinese-language interactions. Each vignette was submitted three times in independent conversations. Two urologists with more than 10 years of clinical experience assessed triage accuracy using a three-level reference standard (immediate emergency department [ED] evaluation, routine outpatient evaluation, or home observation). They also rated response quality across eight dimensions using a 5-point Likert scale and evaluated medical-error severity (0–3), response stability, and Chinese readability. Overall differences were assessed using the Friedman test, followed by pairwise Wilcoxon signed-rank tests with Holm adjustment. Results: A total of 900 responses were generated. Overall three-class accuracy was 89.11%, and the macro-averaged F1 score was 85.73%. Qwen achieved the highest accuracy (94.00%), whereas Claude achieved the lowest (80.67%). However, none of the pairwise comparisons between models reached statistical significance after Holm adjustment. All models generally recognized scenarios requiring immediate ED evaluation, but their ability to distinguish routine outpatient evaluation from home observation varied substantially. Mean content-quality scores ranged from 3.68 to 4.15; AQ achieved the highest overall score (4.15 ± 0.49), whereas DeepSeek achieved the lowest (3.68 ± 0.48). Clinically relevant errors occurred in 12.3% of responses, including major errors in 6.2% and dangerous errors in 3.1%. Three-run exact triage agreement ranged from 78% to 96%. Readability performance varied by metric, with no single model consistently outperforming the others. Conclusions: The six LLMs demonstrated some ability to perform three-level triage of urologic emergencies, supporting further investigation of their potential clinical applications. Nevertheless, important limitations remained in the management of borderline cases, response stability, and safety; some responses included potentially dangerous self-management instructions or could delay care for high-risk patients. Integration with authoritative guidelines, structured risk warnings, and human review will be necessary to improve clinical safety and reliability.