Accuracy of aTrain vs human transcription in infectious disease surveillance focus groups in Cyprus

Background

Accurate transcription is essential for qualitative public health research, particularly in infectious disease surveillance, where discussions involve specialized terminology. Automated speech-to-text tools such as aTrain are increasingly used to accelerate transcription; however, their reliability in complex multi-speaker discussions remains uncertain. This study assessed the accuracy of aTrain compared with human transcription in public health focus groups, as part of the CY-ESM project, which aims to strengthen and digitise the surveillance in Cyprus through One Health approach.

Methods

A comparative qualitative assessment was conducted using three focus groups with 45 participants. Participants represented multiple sectors, including human health and veterinary services, reflecting a multisectoral One Health approach. Recordings were transcribed manually and with aTrain v1.4.1 (Haberl et al., 2024). Transcripts were compared per segment against the human transcript, as the reference, with segments defined as speaker turns. Comparison criteria included lexical accuracy, domain-specific terminology, speaker identification and structural coherence.

Results

Of 4048 segments analyzed, 20-30% required correction due to at least one meaningful error. Errors were most frequent in specialized terminology (15%), overlapping speech (12%) and speaker identification (10%). Generic speaker labels were identified in 10% of segments and fragmented sentence structure in 30%, compared with 5% and 0% in the human transcript.

Conclusions

aTrain-based transcription improves efficiency but has limitations in accuracy, particularly in terminology and multi-speaker discussions. This is key in disease surveillance, where misinterpretation of terms may affect analysis and lead to misleading conclusions in policy and decision-making. A hybrid approach combining automated transcription with human validation can improve efficiency while ensuring high-quality, reliable data.

Key messages

• aTrain transcription improves efficiency in qualitative public health research but introduces errors in terminology and speaker labels, reducing accuracy in infectious disease surveillance settings.

• Human validation remains essential to ensure accurate transcripts and prevent misinterpretation of qualitative findings that may affect analysis, policy decisions and public health responses.