LLM-based content analysis: evaluating consistency, relevance, and interpretation

Background

Traditional social listening based on manual content analysis is time-consuming. Large language models (LLMs) offer a potential solution by enabling rapid qualitative analysis of large volumes of content. However, their reliability as analytical tools remains insufficiently explored. This study evaluates LLM performance in inductive thematic analysis across three dimensions: consistency, relevance, and interpretation.

Methods

The study analyzed three Gemini 2.5 Pro inductive content analysis outputs applied to 60 transcripts of social media posts on hormonal contraception. Consistency (C), defined as the level of agreement between repeated outputs on the same dataset, was assessed using Jaccard similarity after harmonization of narratives across outputs. Relevance (R) was assessed by mapping narratives from each output onto a defined reference set of literature-based themes, with relevance calculated as the proportion of aligned narratives. Interpretation (I) was assessed as the degree to which AI-generated narratives accurately reflected the source text. All metrics were calculated per output and aggregated as mean values.

Results

Consistency was moderate (C = 0.595), indicating partial overlap in themes across analyses, with some variability in narrative selection and formulation. Relevance was high (R = 0.908), showing strong alignment with established thematic patterns in literature. Interpretation was also high (I = 0.899), indicating that narratives closely reflected the analyzed transcripts, with no hallucinations identified during evaluation.

Conclusions

LLMs can support rapid qualitative analysis of health-related content, demonstrating high relevance and accurate reflection of source material. However, moderate consistency highlights the need for methodological safeguards when using LLMs in research. These findings support the use of LLM-assisted approaches in social listening while emphasizing the importance of validation frameworks.

Key messages

• LLMs can reliably identify and interpret health-related narratives in social media data, with high relevance and strong grounding in source material.

• Moderate consistency across repeated analyses highlights the need for validation frameworks when integrating LLMs into public health research.