Artificial Intelligence for vaccine communication: comparing main available Large Language Models

Background

Vaccination is one of the most powerful public health interventions. Large language models (LLMs) have demonstrated strong potential in delivering health-related information. This study compared the performance of ChatGPT 3.5 (G3E), ChatGPT 4o in English (G4E), Claude 3.0 (CDE), Gemini 1.5 (GME), and ChatGPT 4o in Italian (G4I) regarding vaccination communication.

Methods

A mixed-method study was performed. 26 realistic healthcare scenarios were created, then rated via a modified DISCERN questionnaire by 4 independent experts as 1-5 Likert scores. Information Reliability, Information Quality, Medical Appropriateness, Impact on Vaccine Hesitancy, Potential for Behavioral Influence, and Overall Rating were evaluated.

Results

Inter-rater agreement was substantial (Gwet’s AC2 = 0.061). Effects were reported as log(OR), with value 0 indicating no difference. G4E demonstrated the highest performance, with log(OR) = 0.77 (95% CI: 0.18–1.37, p = 0.010) for Overall Rating and log(OR) = 0.62 (95% CI: 0.04–1.20, p = 0.036) for Medical Appropriateness, indicating higher odds of better ratings (log(OR) = 0 means no difference). G4I showed comparatively improved performance in Information Quality (log(OR) = 0.57, 95% CI: 0.06–1.08, p = 0.030), although it performed less favorably in domains related to vaccine hesitancy and behavioral influence. GME exhibited weaker performance across qualitative dimensions, whereas CDE and G3E produced intermediate and relatively consistent results.

Conclusions

Differences in LLM performance likely depend on model architecture, training data, and language-specific optimization. Overall, LLMs display heterogeneous performance in delivering accurate and behaviorally impactful vaccination information, highlighting the need for careful evaluation prior to health communication strategies implementation. These findings highlight the importance of domain-specific adaptations, culturally-sensitive data curation and human expert supervision.

Key messages

• LLMs differ markedly in vaccine communication, depending on the evaluation domain, highlighting the need for careful model selection in public health messaging.

• AI-enabled vaccine communication requires language adaptation, cultural sensitivity, and expert oversight to ensure accuracy and positive behavioral impact.