P-156: Accuracy of AI-generated nutrition advice in diabetes: comparing the reliability of two methodologies
R Kleine, R Manoel, L Aragão, C Moraes, V Golfieri, N LZ de Oliveira, T Otranto, C Sennersten, E SonestedtAbstract
Introduction
Type 2 diabetes (T2D) is increasingly recognized in pediatric populations worldwide. Children, adolescents, and their caregivers are progressively using large language models (LLMs) to obtain nutritional advice, raising concerns about the accuracy of information in this vulnerable group.
Objectives
This study aimed to assess the reliability of a new methodology compared to current methods to assess AI generated nutrition advice in diabetes.
Methods
A large language model was prompted to generate 45 frequently asked questions about diabetes. Each question was answered with a maximum of 50 words. Responses were then fragmented into individual claims, with each sentence containing a single piece of information and rewritten for standalone clarity. Six registered dietitians (RDs) with ≥3 years of clinical experience in chronic disease were divided into two groups (n=3) to assess response accuracy using the Standards of Care in Diabetes 2026 as the primary reference. Group 1 evaluated full responses using a 4-point Likert scale (1 = not at all accurate; 4 = fully accurate), representing the reference method. Group 2 assessed the responses which have been fragmented using a binary scale based on the accuracy of the statement (Yes/No), representing the proposed method. All assessors received prior training. Inter-rater reliability was evaluated using Fleiss’ kappa (p < 0.05). To enable comparison, claim-level scores were converted to equivalent Likert-scale values.
Results
Fleiss’ kappa coefficients were 0.07 (p > 0.05) for the method of reference and −0.01 (p > 0.05) for the proposed methodology, indicating very low inter-rater agreement in both approaches. Therefore, the proposed method did not improve reliability of assessing accuracy of AI generated advice compared with conventional method.
Conclusions
Further research is needed to develop reliable, systematic, and resource-efficient method for evaluating the accuracy of AI-generated nutrition advice.