Evaluation of ChatGPT responses to common questions about rTMS: can ChatGPT support patient education about rTMS?
Büşra Kalem, Hatice Gülhan Gülel, Theokritos Veskoukis, Eva Meisenzahl, Milenko Kujovic, Christian TheisenAbstract
Objective:
In recent years, artificial intelligence (AI) and its applications have become a compelling topic in psychiatry. This study aimed to investigate whether the AI chatbot ChatGPT is useful for patient education by evaluating the quality, reliability, and readability of its responses to patient questions about repetitive transcranial magnetic stimulation (rTMS).
Methods:
Twenty-five frequently asked questions about rTMS were separately queried to ChatGPT-4 with no follow-up questions. Responses were reviewed by three psychiatrists by using the Global Quality Scale (GQS), DISCERN tool, and Journal of the American Medical Association (JAMA) Benchmark Criteria. Readability was evaluated by the Flesch-Kincaid Grade Level and the Gunning Fog, Coleman-Liau, and Simple Measure of Gobbledygook Indexes.
Results:
Overall, the quality of ChatGPT responses was moderate to good, with a mean GQS score of 3.69. The mean DISCERN score was 35.33, which is considered poor, and the JAMA Benchmark Criteria score was 0 for all questions. The mean readability score was high (mean grade level, 10.98).
Conclusion:
Our results show that ChatGPT responses to patient questions about rTMS are of acceptable quality and usefulness. However, the reliability and accuracy of the responses are limited, and a high reading level is required to understand them. Although ChatGPT can support patient education about rTMS, it should be used with caution and under guidance from healthcare professionals.