Quality, Readability, and Reliability of Artificial Intelligence Chatbot Responses to Parent‐Centered Questions on Newborn Hearing Screening: A Cross‐Sectional Comparative Study
Arif Ozses, Beyza Demirtas Yilmaz, Merve OzsesABSTRACT
Aims and Objectives
Parents increasingly use large language models (LLMs) to obtain health‐related information; however, the quality, readability, and reliability of AI‐generated patient education remain uncertain. This study evaluated the performance of three LLM‐based chatbots in providing parent‐centred information regarding newborn hearing screening.
Methods
Thirty‐eight parent‐centred questions regarding newborn hearing screening were submitted to ChatGPT‐5.1, Gemini 2.5 Flash, and DeepSeek‐V3. Responses were assessed using a five‐point Likert scale for accuracy, comprehensiveness, clarity, and credibility. Responses were assessed using the modified EQIP for quality, and the Flesch Reading Ease Score (FRES) and Flesch–Kincaid Grade Level (FKGL) for readability.
Results
DeepSeek‐V3 outperformed both ChatGPT‐5.1 and Gemini 2.5 Flash in accuracy and comprehensiveness ( p < 0.05), whereas its clarity scores were lower ( p < 0.05). The FRES scores (51.5–59.3) indicated difficult readability, whereas the FKGL scores (8.4–8.9) corresponded to an eighth‐grade reading level. Gemini 2.5 Flash demonstrated higher FRES scores ( p = 0.001). mEQIP scores were not significant ( p = 0.148) and indicated good quality.
Conclusions
LLM‐based chatbots generally perform acceptably on newborn hearing screening questions, although substantial variability was observed across models. Despite acceptable readability levels, chatbot‐generated content remains relatively difficult to read. LLMs may serve as complementary information resources for parents seeking information about newborn hearing screening, although further research is needed before their routine clinical use.
Rationale
This study addresses the JECP journal's aim to evaluate the clinical applications of emerging artificial intelligence technologies in healthcare. Evaluating the quality, accuracy, readability, and reliability of chatbot‐generated information for newborn hearing screening provides evidence to support person‐centred care, patient education, and the responsible integration of AI into clinical practice.