Human versus large language models in thematic analysis of infectious disease surveillance in Cyprus

Background

Limited evidence exists on the use of large language models (LLMs) in the analysis of real-world qualitative data, particularly in public health settings. This study compares LLM-generated and human thematic analyses using focus group transcripts from a national baseline assessment of the infectious disease surveillance system. The assessment was part of the CY-ESM project, which aims to strengthen and modernise the surveillance system in Cyprus under a One Health approach.

Methods

Two focus group discussions, with a total duration of three hours, were analysed. Participants were stakeholders from human, animal and environmental health. Human coding in NVivo served as the reference standard. Structured prompts were applied to ChatGPT-4 and DeepSeek-V3 to evaluate theme capture, consistency across runs, hallucination rates and accuracy of categorisation within a predefined framework. Outputs were compared with the human codebook and between models using percentage-based indicators.

Results

ChatGPT captured 64.3% of human-identified themes, while DeepSeek achieved 71.4%. Consistency across runs was 77.7% for ChatGPT and 95.0% for DeepSeek. Hallucination rates were lower for DeepSeek (9.5%) compared to ChatGPT (12.5%). When applying a predefined framework, ChatGPT correctly categorised 46.4% of themes, compared to 68.4% for DeepSeek. However, DeepSeek showed reduced adherence to instructions and a tendency towards more negative framing, focusing on system weaknesses over strengths.

Conclusions

LLMs demonstrate considerable potential as supportive tools in real-world qualitative analysis, with DeepSeek performing slightly better. When combined with human oversight, they can substantially reduce initial coding time, allowing researchers to focus on refining and synthesising results in applied settings. Recognising the strengths and limitations of different LLMs is essential for responsible and effective integration into public health research and practice.

Key messages

• LLMs can accelerate qualitative thematic analysis, but rigorous human oversight is essential to ensure accuracy, reliability, and contextual validity of public health research results.

• LLM performance varies substantially between models. Researchers should validate accuracy, consistency, instruction adherence and framing bias for responsible public health practice integration.