Evaluating the Quality of Artificial Intelligence Chatbot Responses in Audiology Case Interpretation and Management: A Blinded Comparative Study
Nicole Labuschagne, Faheema Mahomed-Asmail, Vinaya Manchaiah, Jan-Willem Wasmann, De Wet SwanepoelPurpose:
This study aimed to compare the quality of audiological case interpretation and management recommendations generated by two freely-accessible large language model (LLM) chatbots (ChatGPT 3.5 and Gemini 2.0 Flash) with those of qualified audiologists, using an expert-rated quality assessment tool.
Method:
This study followed a two-phased, cross-sectional design with comparative analysis: Phase 1: 15 anonymized adult audiological cases were interpreted by 10 qualified audiologists (> 2 years' experience) and two LLM chatbots, producing 60 interpretations; each interpretation answered two questions (diagnostic interpretation and management), giving 120 responses. Phase 2: 30 qualified audiologists independently rated all 120 responses using the CLEAR (Completeness, Lack of false information, Evidence, Appropriateness, Relevance) tool to assess the quality of AI-generated health information (five raters per response; 600 total ratings), blinded to responder identity.
Results:
There were no statistically significant differences in CLEAR scores between artificial intelligence–generated and audiologist-generated responses (all
Conclusions:
Under controlled conditions, no statistically significant differences in CLEAR scores were detected between LLM-generated and audiologist-generated responses. These findings do not establish formal equivalence but represent a meaningful empirical benchmark for this emerging area of research.