Evaluation of ChatGPT-Generated Simplifications of Shoulder MRI Reports: Assessments by Orthopedic Surgeons and Non-Healthcare Volunteers

Background/Objectives: To evaluate ChatGPT-generated (GPT-5) simplifications of shoulder MRI reports from the perspectives of orthopedic surgeons and non-healthcare volunteers. Methods: In this single-center retrospective study, 100 de-identified shoulder MRI reports issued between June 2024 and June 2025 were simplified with ChatGPT in February 2026 using a standardized prompt. Between 10 March and 10 April 2026, three orthopedic surgeons independently rated each simplified text against the original report for accuracy, completeness, understandability, and misleading potential on a 5-point Likert scale (higher scores indicating better ratings). In a second arm, 100 adult non-healthcare volunteers each reviewed one simplified report and completed a six-item questionnaire on perceived clarity, comprehensibility, usefulness, and appropriateness. Results: Median expert scores were 5.0 in all domains, and all three raters assigned scores ≥ 4 in 86–98% of reports. Gwet’s AC2 ranged from 0.93 to 0.98, whereas Krippendorff’s α ranged from −0.02 to 0.05. Among volunteers, agreement ranged from 47% for understanding report terms to 80% for the appropriateness of the text. Health literacy correlated positively with most items (r = 0.329–0.429, p ≤ 0.001), and age correlated negatively with understanding report terms (r = −0.352, p < 0.001). Conclusions: The simplified reports were rated favorably by orthopedic surgeons and were perceived as useful by non-healthcare volunteers. However, the study was limited by its single-center design, the use of a single model and prompt, the absence of a comparison with the original reports in the volunteer arm, and the reliance on self-reported rather than objectively tested comprehension; therefore, it does not demonstrate improved patient comprehension or clinical safety. Such texts should supplement, not replace, the original report and physician consultation.