Accuracy of Artificial Intelligence based chatbots in reporting jaw lesions from multimodal radiographic images: A cross-sectional study
Noha M Elkersh, Aly Mohamed Atteya, Ahmed Abdelghany ShararahAbstract
Objectives
The current study aimed to quantify the diagnostic accuracy of commonly utilized chatbots including Gemini, Copilot, Claude, and specialized architectures like Manus in the detection and differential diagnosis of various jaw lesions, while concurrently evaluating the clinical safety and fidelity of the information they provide.
Materials and methods
Cone beam computed tomography (CBCT) dataset from 97 patients presented with jaw lesions were collected and anonymized. Panoramic 2D views were reconstructed from Digital Imaging and Communication in Medicine (DICOM) of all cases using Bluesky Plan software and provided to 4 chatbots (Gemini 2.5 Pro, Copilot, Claude and Manus). Moreover, the DICOM data was provided to Manus followed by prompting. The reports generated were evaluated for accuracy, relevance and feasibility.
Results
Statistically significant differences were detected between the chatbots in all measured parameters. In all evaluated parameters Manus CBCT showed the most accurate results (95% of lesions were detected and correctly diagnosed,). Gemini 2.5 pro ranked second where 80% of lesions were detected and 56% were correctly diagnosed. Manus Pan showed less accurate results. The least accurate results were detected in Copilot and Claude.
Conclusion
Significant discrepancies exist among artificial intelligence (AI) chatbots regarding their diagnostic accuracy in reporting jaw lesions. Notably, the integration of raw 3-dimensional CBCT data substantially optimizes chatbot performance in lesion detection and diagnosis, as demonstrated by Manus architecture.