Comparative Evaluation of Three Large Language Models in Systemic Lupus Erythematosus Using a Clinical Vignette-Based Rubric

Background: Large language models (LLMs) can generate clinically relevant information, but safety-related performance in systemic lupus erythematosus (SLE) remains insufficiently characterised. We compared GPT-5.5 Instant (ChatGPT), Claude Sonnet 4.6, and Gemini 3.5 Flash through their free-tier web interfaces across diagnostic acuity, guideline adherence, patient safety/red-flag recognition and hallucination-related reliability. Methods: Twelve SLE clinical vignettes were submitted to all three interfaces in May 2026 (36 single-pass responses). Two blinded specialists independently assessed the outputs using a study-developed four-domain rubric (composite range, 4–20). Inter-rater reliability was evaluated using quadratic-weighted Cohen’s κ. Paired comparisons across the shared vignettes used Friedman tests with Kendall’s W and post hoc Wilcoxon signed-rank tests with Holm adjustment. Results: Inter-rater κ ranged from 0.754 to 0.893. Friedman tests showed between-interface differences for patient safety/red-flag recognition (χ2(2) = 9.571, p = 0.008, W = 0.399), hallucination-related reliability (χ2(2) = 6.500, p = 0.039, W = 0.271) and composite score (χ2(2) = 8.140, p = 0.017, W = 0.339), but not diagnostic acuity (p = 0.926) or guideline adherence (p = 0.175). Claude scored higher than ChatGPT and Gemini for patient safety (Holm-adjusted p = 0.0176 for both) and composite score (Holm-adjusted p = 0.0469 for both). No hallucination-related pairwise comparison remained significant after Holm adjustment. Conclusions: Within this exploratory, 12-vignette paired evaluation, the Claude interface received higher safety and composite ratings, but these observations do not establish generalisable model superiority or real-world clinical safety. The findings are specific to the tested May 2026 model versions and require independent replication.