Accelerating infodemic surveillance: An evaluation of GenAI vs. human social media thematic analysis

Background

Social listening is essential for generating infodemic insights through the systematic collection and analysis of public narratives. However, traditional manual thematic analysis is too labor-intensive for real-time insights from large social media datasets. Generative AI (GenAI) offers a scalable alternative, yet its methodological robustness remains under-evaluated. This study assesses the thematic consistency and coding reliability of GenAI compared to a human analyst for inductive thematic analysis.

Methods

The dataset comprised 60 publicly available posts on hormonal contraception from TikTok, Instagram, and YouTube in Serbia. Multimedia content was converted to text using Whisper (large-v3) and EasyOCR (1.7.2). Inductive thematic analysis was performed by an experienced qualitative researcher and Google Gemini (v2.5 Pro) across three iterations. Thematic consistency was measured as the percentage of themes shared by human analyst and GenAI, while intercoder reliability (ICR) was calculated as the ratio of total agreements to total coding instances (the sum of agreements and disagreements) for those shared themes.

Results

The themes generated by GenAI were consistent with 83,3% (15/18), 66,7% (12/18) and 72,2% (13/18) of themes identified by human analyst across three iterations, occasionally merging specific human analyst’s themes into broader categories. Intercoder reliability (ICR) for shared themes varied from fair (35%) in the first iteration to substantial (68.9%) in the second and moderate (55%) in the third.

Conclusions

GenAI demonstrates an acceptable ability to identify human-defined narratives, with both thematic consistency and coding reliability varying across iterations. Our findings suggest that GenAI currently lacks the stochastic stability required for autonomous thematic mapping, and should be utilized as an iterative “human-in-the-loop” tool to ensure precision in infodemic surveillance.

Key messages

• GenAI’s coding reliability fluctuates significantly across iterations (35% to 68,9%), suggesting that a single-run GenAI approach is too unstable for reliable public health decision-making.

• Multiple runs and human oversight are essential for stable GenAI use in social listening and infodemic insights.