Medical content quality and radiologist-rated patient understandability of three large language model systems answering patient questions about renal cystic lesions: A blinded, Paired, multi-reader study
Zhenghan Liu, Qianqian Chen, Tianxiao Hu, Mingyu Li, Haomei Yu, Jie Wang, Xiangming Fang, Xiaoyun HuObjective
To compare three large language model (LLM) systems answering patient questions about renal cystic lesions in terms of medical content quality and radiologist-rated patient understandability. We also assessed inter-rater reliability and, exploratorily, the readability characteristics of English translations.
Methods
This cross-sectional, blinded, paired, multi-reader evaluation included 40 questions derived from authoritative patient education resources and web searches. Each question was submitted to OpenAI GPT-5.5, DeepSeek V4 Pro, and Ant A-Fu. Three radiologists independently rated every response on six 1–5 dimensions: medical accuracy, completeness, safety/risk guidance, radiologist-rated patient understandability, actionability/care advice, and overall quality. Overall comparisons used the Friedman test. Pairwise comparisons used paired Wilcoxon signed-rank tests with Holm correction. Intraclass correlation coefficients (ICCs) quantified inter-rater reliability. Sensitivity analyses evaluated all three responses and included 10,000 simulated random selections.
Results
No statistically significant overall differences were found in medical accuracy (4.65/4.68/4.65, P=0.790) or completeness (4.58/4.67/4.68, P=0.362). Safety/risk guidance also did not differ significantly (4.65/4.50/4.57, P=0.651). Differences were nonsignificant for actionability/care advice (4.34/4.42/4.38, P=0.120) and overall quality (4.30/4.38/4.40, P=0.134). Radiologist-rated patient understandability was the only dimension with an overall between-system difference (4.05/4.12/4.41, P<0.001; Kendall’s W=0.23). Ant A-Fu scored higher than OpenAI GPT-5.5 (Holm-adjusted P=0.0039; rank-biserial r=−0.768) and DeepSeek V4 Pro (Holm-adjusted P=0.0138; r=−0.638). This dimension showed very high inter-rater reliability [ICC(2,1)=0.947; ICC(2,3)=0.982]. The patient-understandability difference remained significant when all three responses were included and in all 10,000 random-selection simulations; the other five dimensions remained nonsignificant.
Conclusion
The primary analysis did not detect clear between-system differences in most medical-content dimensions. However, pronounced high-score clustering limited the discriminatory ability of the five-point rubric, and these nonsignificant findings do not establish similar or equivalent medical-content performance. Radiologist-rated patient understandability differed between systems and showed very high inter-rater reliability. Prospective studies involving patients are needed to determine whether this professional proxy predicts actual patient comprehension.