Reliability and readability assessment of appendicitis information from AI language models (ChatGPT, gemini and perplexity) in English and Spanish
Mabel Lucero Olarte Jurado, Natalia Pimiento Blanco, María Jose Prieto Otero, Miguel Enrique Ochoa Vera, Jaime Fernando Barajas, Cristian Orlando Porras BuenoBackground and aims
Appendicitis is one of the most frequent conditions in emergency departments. Artificial intelligence (AI)-based language models represent a novel source of medical information; however, their reliability and readability across languages remain uncertain. This study aimed to evaluate the reliability and readability of AI-generated appendicitis information.
Methods
Seven standardised questions were formulated in Spanish and English. Two independent reviewers assessed response reliability using a validated tool. Spanish readability was measured using the Szigriszt-Pazos formula, while English readability was assessed via the Flesch Reading Ease (FRE) and Flesch–Kincaid scores. Results were compared across tools and languages.
Results
For Spanish reliability, Perplexity® produced the highest proportion of complete responses (71.43%), followed by ChatGPT® (57.14%) and Gemini® (28.57%). In English, Perplexity® and Gemini® both achieved 71.43%, followed by ChatGPT® (57.14%). Readability analysis via FRE formula in English revealed significant differences between the models (
Conclusion
Perplexity® emerged as the most reliable model in both languages, although readability remains a concern. There is a clear need for improvements to optimize the accuracy and accessibility of AI-generated medical information.