Can ChatGPT Aid Clinicians in Educating Patients on the Surgical Management of Glaucoma?
Reza Kianian, Deyu Sun, JoAnn Giaconi- Ophthalmology
Précis:
ChatGPT can help healthcare providers automate the quality assessment of online health information, but it does not produce easier to understand responses compared to existing online health information.
Purpose:
To compare the readability of ChatGPT-generated health information about glaucoma surgery to existing material online, and to evaluate ChatGPT’s ability to analyze the quality of information found online about glaucoma surgery.
Methods:
ChatGPT was asked to create patient handouts on glaucoma surgery using 7 independent prompts aiming to generate 6th grade level reading material. Existing patient-targeted online health information about glaucoma surgery was selected from the top 50 search results of three search engines, excluding advertisements, blog posts, information intended for health professionals, irrelevant content, and duplicate links. Four validated tools were used to assess readability, and the readability of the ChatGPT-generated material was compared with the readability of existing online information. The DISCERN Instrument was used for quality assessment of online materials. The DISCERN instrument was also programmed to ChatGPT to evaluate its ability to analyze quality. The R software and descriptive statistics were used for data analysis.
Results:
35 webpages were included. There was no difference between the reading level of online webpages (12th grade) and the reading level of ChatGPT-generated responses (11th grade), despite the ChatGPT prompts asking for simple language and a 6th grade reading level. The quality of health content was “fair” with only 5 resources receiving an “excellent” score. ChatGPT scored the quality of health resources with high precision (r=0.725).
Conclusions:
Patient-targeted information on glaucoma surgery is beyond the reading level of the average patient, and therefore at risk of not being understood, and is of subpar quality, per DISCERN tool scoring. ChatGPT did not generate documents at a lower reading level as prompted, but this tool can aid in automating the time-consuming and subjective process of quality assessment.