DOI: 10.7126/cumudj.2049044 ISSN: 1302-5805

EVALUATING THE ACCURACY AND RELIABILITY OF ARTIFICIAL INTELLIGENCE CHATBOTS IN ANSWERING DENTAL SPECIALTY EXAMINATION PERIODONTOLOGY QUESTIONS

Aysun Akpınar
Show PDF Cite
Background/Aim: Large language models (LLMs) are increasingly utilized by dental students and specialty candidates as supplementary tools for examination preparation. However, their reliability in specialized clinical domains requiring analytical reasoning remains underexplored. This study aimed to comparatively evaluate the accuracy, reliability, and inter-model agreement of three contemporary reasoning-focused LLMs in answering Periodontology questions from the Turkish Dental Specialty Examination (DUS).Materials and Methods: In this cross-sectional in silico study, 100 multiple-choice Periodontology questions administered in past DUS sessions (2012–2026) were selected. The questions were presented in isolated sessions without prior contextual prompting to three models: OpenAI GPT-5.5 Thinking, Anthropic Claude Opus 4.7 (adaptive thinking), and Google Gemini 3.1 Pro. Responses were scored dichotomously against official answer keys. Overall accuracy rates were compared using Cochran's Q test, pairwise agreement was assessed via Cohen's Kappa (k) coefficient, and shared error distributions were analyzed using a two-tailed Binomial test (p 0.05), ChatGPT and Gemini selected the identical incorrect distractor in all 5 questions they mutually failed (100% shared error rate; p = 0.0625).Conclusion: Contemporary reasoning-capable LLMs demonstrate high accuracy and substantial inter-model consistency on specialty-level Periodontology questions. Nevertheless, the presence of concordant errors indicates shared algorithmic blind spots, underscoring that AI outputs must be supplemented by specialist oversight and evidence-based verification in dental education.

More from our Archive