No significant differences between large language model prompting strategies and human experts in summarisation of ‘the formal EU–US meniscus rehabilitation consensus: An ESSKA‐AOSSM‐AASPT Initiative’
Jakob Lindskog, Felix Öttl, Yinan Yu, Mark V. Paterno, Airelle Giordano, Alexandre Rambaud, Robert Prill, Aleksandra Królikowska, Kristian Samuelsson, Eric Hamrin SenorskiAbstract
Purpose
This study aimed to evaluate the interpretive and applicative performance of three conditions applied to the formal EU‐US, ESSKA‐AOSSM‐AASPT consensus on meniscus rehabilitation: (1) single‐pass prompting, (2) sequential chain‐of‐thought prompting and (3) human expert‐written output.
Methods
This exploratory comparative study evaluated three conditions applied to two parts of the formal EU‐US ESSKA‐AOSSM‐AASPT meniscus rehabilitation consensus: (1) single‐pass prompting, (2) sequential chain‐of‐thought prompting using ChatGPT (GPT‐5.1), Claude (Sonnet 4.5), and Gemini (3.0), and (3) human expert‐written output from three collaborating human experts. For each part, three tasks were assessed: generation of summaries, application of summaries to three predefined clinical scenarios, and resistance to three predefined misinterpretation queries. Three blinded senior sports physiotherapists evaluated summary quality using five QUEST categories and rated the clinical application and misinterpretation resistance responses. Median ratings were compared using Friedman and post‐hoc Wilcoxon signed‐rank tests. Gwet's AC2 coefficient with quadratic weights was used to assess inter‐rater agreement for each QUEST category and for the clinical application and misinterpretation resistance items.
Results
Across consensus parts, no statistically significant differences were identified between large language model prompting strategies and human expert outputs for QUEST categories, clinical application, and misinterpretation resistance queries. Sequential chain‐of‐thought prompting received higher numerical reviewer ratings than single‐pass prompting for ChatGPT and Gemini, particularly for quality of information and trust and confidence, although differences were not statistically significant. Inter‐rater agreement ranged from substantial to almost perfect (Gwet's AC2 = 0.79–0.99).
Conclusion
No statistically significant differences in reviewer ratings were observed between ChatGPT, Claude, Gemini and human expert output, with generally high ratings for summaries and responses to clinical scenarios. Large language models may therefore represent a potential adjunct for summarisation and interpretation of sports medicine consensus recommendations to inform rehabilitation decision‐making.
Level of Evidence
Level V.