Four large language models in parent-oriented education for pediatric flexible flatfoot: Quality, readability, hallucination, and reproducibility
Erkan Servet, Beytullah Unat, Musa Alperen Bilgin, Çağrı Karabulut, Nevzat GönderPurpose
Parents increasingly consult large language model (LLM) chatbots about pediatric flexible flatfoot, a common condition that rarely requires treatment and in which the clinical task is chiefly communication. We compared four LLMs on the quality, readability, and reproducibility of parent-oriented answers.
Methods
This was a cross-sectional, comparative content analysis of AI-generated text. No patients, clinical records, or human participants were involved; the units of analysis were parent-oriented questions and the chatbot responses they generated. Twenty parent-oriented questions were submitted in triplicate to ChatGPT, Claude, DeepSeek, and Gemini in independent, memory-disabled sessions (240 responses). Two blinded pediatric orthopedic surgeons scored responses (Mika grade, four-point Likert, and a hallucination taxonomy; PEMAT-P, DISCERN, and QUEST on pooled material). Readability used five indices. Between-model differences were tested with Friedman and Dunn–Bonferroni post-hoc tests; agreement and reproducibility used intraclass correlation coefficients (ICC).
Results
All five readability indices differed between models (P < 0.001); none reached the recommended sixth-grade level. DeepSeek was most accessible (FRES 66.3) and ChatGPT least (FRES 17.7). On pooled-material instruments (inter-rater ICC 0.63–0.74), Claude scored highest for understandability (91.6%) and actionability (88.3%). Per-response clinical-quality measures were closely clustered, showed poor inter-rater agreement (ICC −0.01 to +0.07), and did not discriminate. Readability reproducibility across identical queries was moderate (FRES ICC 0.49–0.72).
Conclusions
Clinical-quality scales did not reliably distinguish the four LLMs, whereas readability differed substantially; none met recommended readability standards, and output was only moderately reproducible. Model choice should be guided by readability and actionability, under clinician oversight.
Significance of Study
For a condition managed mainly through parent education, LLM selection should rest on readability and actionability rather than presumed clinical accuracy, with output used under clinician oversight.
Level of Evidence
V (cross-sectional comparative content analysis of AI-generated text; no human participants).