A comparative evaluation of large language models in hemodialysis health communication: A cross-sectional study
Shenao Yu, Xinling Zhou, Yanru Wang, Liuyan XuBackground
This study aimed to systematically evaluate and compare the performance of three LLMs (ChatGPT, Gemini, and AQ) in addressing daily inquiries from dialysis patients. To assess their feasibility as digital health education tools for answering real-world questions from HD patients.
Methods
A cross-sectional, evaluator-blinded comparative design was employed. We constructed 32 multimodal (text and image) test scenarios encompassing four core domains: dietary management, vascular access, symptom burden, and report interpretation. These scenarios were categorized into four progressive levels based on interaction complexity and modality to simulate real world clinical patient-physician interactions. Responses from the three multimodal LLMs were generated in parallel. Twelve dialysis healthcare professionals, using the latest clinical guidelines as the evidence-informed reference standard, conducted independent blinded evaluations of the de-identified responses across four dimensions: accuracy, comprehensiveness, applicability, and readability.
Results
The evaluation revealed that Gemini significantly outperformed ChatGPT in overall accuracy. Regarding comprehensiveness, both Gemini and AQ demonstrated significantly better overall performance than ChatGPT; specifically, Gemini showed greater advantages in dietary management and vascular access, while AQ performed better in addressing symptom burden. In terms of practical applicability, the AQ model achieved the highest overall score, significantly outperforming both Gemini and ChatGPT. This advantage was particularly pronounced in complex scenarios involving follow-up inquiries and symptom burden-related consultations. Furthermore, the responses generated by the AQ model were significantly superior in readability compared to the other two models, proving to be more accessible and easier to understand.
Conclusion
The general-purpose LLM, Gemini, demonstrates high accuracy and comprehensiveness of clinical knowledge, performing comparably to the AQ model when addressing dialysis-related medical inquiries. Conversely, the AQ model, specifically fine-tuned for the Chinese context, exhibited distinct statistical advantages in practical applicability and linguistic readability, highlighting its enhanced potential for localized patient communication.