Clinical accuracy, information quality and readability of chatbot responses to patient questions about lower extremity amputation
Kamil Balaban, Mehmet Batu Ertan, Mahmut KalemObjective
To compare the clinical accuracy, information quality and readability of chatbot responses to common patient questions about lower extremity amputation.
Methods
Twenty frequently asked patient questions were submitted once to each of ChatGPT, Gemini and DeepSeek in separate sessions. Two reviewers, blinded to chatbot identity, independently assessed responses using a four-category clinical accuracy scale and the DISCERN instrument. Readability was quantified as the mean grade level across the Flesch-Kincaid Grade Level, Gunning Fog, Coleman-Liau, and Simple Measure of Gobbledygook indices. Comparisons used Friedman tests and Holm-adjusted Wilcoxon signed-rank tests.
Results
Sixty responses were analysed. Mean (±standard deviation) clinical accuracy scores were 2.20 ± 0.57, 1.55 ± 0.65 and 1.20 ± 0.38 for ChatGPT, Gemini and DeepSeek, respectively (lower scores indicate greater accuracy). Gemini and DeepSeek received lower scores than ChatGPT (adjusted p=0.012 and p=0.002, respectively), whereas the Gemini–DeepSeek difference was not significant (p=0.090). Mean DISCERN scores were 50.05 ± 5.78, 53.33 ± 5.43 and 58.23 ± 3.39, respectively, with significant differences in all pairwise comparisons. Mean reading grades were 12.93 ± 1.28, 13.17 ± 1.36 and 11.74 ± 1.15, respectively; DeepSeek had lower grades than both comparators (both adjusted p<0.001).
Conclusion
DeepSeek responses received higher information quality scores and had lower reading grade levels, although clinical accuracy did not differ significantly between DeepSeek and Gemini. All three chatbots exceeded commonly recommended reading levels for patient education. Clinician review and language adaptation are warranted before chatbot-generated information is used to supplement individualised counselling on amputation and rehabilitation.