Assessing the Reliability, Quality, and Readability of ChatGPT-Generated HPV Vaccination Responses

Background

Human papillomavirus (HPV) causes cervical and oropharyngeal cancers. Despite vaccine availability, vaccination rates remain suboptimal due to misinformation and safety concerns. As people increasingly seek health information from AI tools such as ChatGPT, evaluating the reliability, quality, and readability of these responses is essential. This study assesses ChatGPT-generated responses to frequently asked questions about HPV vaccination from the Turkish population perspective.

Methods

This descriptive study evaluated ChatGPT-generated responses to 21 HPV vaccine-related questions. Questions were identified via Google Trends (Türkiye) and refined using official guidelines. All questions were submitted to ChatGPT-5.3 in separate new chat sessions in April 2026. Two independent reviewers assessed reliability and quality using the DISCERN instrument and the Global Quality Score (GQS). Inter-rater reliability was measured using the Intraclass Correlation Coefficient (ICC) for DISCERN and percent agreement for GQS. Readability was measured using the Ateşman Readability Index.

Results

DISCERN analysis revealed an ICC of 0.754 (95% CI: 0.395–0.900, p = 0.001) and a Cronbach’s alpha of 0.754. GQS percent agreement was 81.0%. Mean scores were 4.00 ± 0.22 for GQS and 51.88 ± 7.81 for DISCERN. The mean readability score was 80.76 ± 10.75 (range: 62.4–100). Responses were at the 5th–6th grade level (47.6%), 7th–8th and 9th–10th grade levels (19.0% each), and below the 4th-grade level (14.3%).

Conclusions

ChatGPT’s responses are generally understandable for individuals with secondary and high school education levels and provide reliable information. However, limitations in addressing complex queries may reduce comprehensibility for those with lower health literacy. Optimizing prompts and quality assurance frameworks can enhance AI’s role in public health communication.

Key messages

• ChatGPT provides understandable HPV vaccine information for individuals with at least a secondary school education, but reliability may vary for complex health topics.

• ChatGPT-generated HPV vaccination information may not be suitable for individuals with low health literacy, highlighting the need for quality assurance in AI-based health communication.