Challenges in applying generative artificial intelligence for ankle fracture classification: Insights from AO/OTA and Lauge-Hansen systems
Zhiyong Cui, Bibo Wang, Bing Li, Chonglin Yang, Yuan Zhu, Jinhao Liu, Yunfeng YangAim
To evaluate the diagnostic performance and radiology report quality of generative artificial intelligence (AI) models in classifying acute ankle fractures using the AO/OTA and Lauge-Hansen systems.
Methods
A study of 529 acute ankle fracture cases was conducted. Expert consensus was used as the reference for AO/OTA and Lauge-Hansen classifications. Global multi-class and subtype-level metrics were used to quantify diagnostic performance, and five-dimensional blinded scoring was performed to evaluate AI-generated report quality.
Results
ChatGPT-5.1 achieved the highest accuracy (62.0% for AO/OTA, 64.1% for Lauge-Hansen). All models performed below the majority-class baseline for both classification tasks (73.5% for AO/OTA and 76.8% for Lauge-Hansen). Higher weighted-averaged F1 (weighted_F1) scores overestimated model performance. Near-zero Matthews correlation coefficient (MCC) and low macro-averaged F1 (macro_F1) reflected poor global multi-class classification performance, while one-versus-rest subtype-level balanced accuracy around 0.5 confirmed random-level discrimination for each subtype in binary one-versus-rest testing. All models tended to misclassify rare fractures as common AO/OTA and Lauge-Hansen subtypes, showing high specificity but poor sensitivity for rare and high-risk fractures. Quality evaluation revealed significant inter-model differences (all p<0.001), with ChatGPT-5.1 generating the most complete and clinically qualified radiology reports, while Gemini and Claude yielded inferior textual quality.
Conclusion
All models demonstrated limited capability in ankle fracture classification, highlighting the need for specialized orthopaedic AI tools.