Quality and readability of responses generated by popular large language models to frequently asked questions about menopause: A comparative study
Çağlasu Sancaktar, Burak ÇakmakObjective
Patients increasingly use large language models (LLMs) for menopause-related health information, yet the quality and readability of their responses have not been systematically compared across widely used platforms. This study compared the information quality and readability of responses from popular LLMs to frequently asked menopause questions.
Methods
In this comparative study, 50 menopause questions were compiled from four sources (ChatGPT, Google Gemini, Microsoft Copilot, and Google’s “People also ask”) after removing duplicates and less relevant items. Each was entered into the three tools, and responses were categorized with a modified Rothwell system and independently rated by two obstetrics and gynecology specialists using DISCERN, a Likert scale, PEMAT, Flesch Reading Ease, and Flesch-Kincaid Grade Level.
Results
Of the 50 questions, 70% were fact, 20% policy, and 10% value. Total DISCERN scores and overall information quality did not differ among platforms. Gemini scored higher on Likert and PEMAT actionability than ChatGPT and Copilot; ChatGPT and Gemini had higher Flesch Reading Ease than Copilot, which had a higher grade level. Overall readability differed (p = 0.022), with Gemini most often sufficient. Final overall sufficiency, meeting both information quality and readability, was low across all platforms (24.7%).
Conclusion
Popular LLMs produced menopause responses generally consistent with the literature but often limited in reliability structure and reading level, with low overall sufficiency across all three tools. Performance varied by platform and question type. Current consumer LLMs may complement rather than substitute for clinician-guided counseling; whether they can safely substitute requires patient-centered evaluation.