Reproducible Workflows for LLM-Assisted Qualitative Coding and Text Classification in Systematic Literature Reviews

Systematic literature reviews (SLRs) are constrained by two persistent challenges: the need to engage with large bodies of literature and the time- and resource-intensive nature of qualitative interpretation. Large language models (LLMs) can offer a means to mitigate these constraints by supporting selected stages of qualitative literature analyses. In this study, we demonstrate how LLMs can be integrated and evaluated in SLR workflows while maintaining transparency and human oversight. Across four case studies, we specify and compare LLM-supported tasks with varying degrees of complexity, spanning classification, automated deductive coding, and inductive codebook development based on different premises. Key methodological contributions include a test-driven cycle for prompt and codebook refinement, robust reliability measurement including a cross-disciplinary comparison of reliability indicators, and mitigation strategies for frequently occurring issues and inherent weaknesses of LLM-coders (such as biases, hallucinations, sycophancy, context decay). An open-source, modular Python pipeline automates our workflows with logging, metric computation, and evidence-line checks. It can be readily reused and adapted to different qualitative coding tasks. Our results suggest that LLMs display higher reliability in low-complexity tasks, such as classification. For more complex and interpretative coding tasks, more time and resources were required to reliably integrate the LLM in the process, and its performance still continuously plateaued below human–human agreement even after intensive alignment efforts. The practical value of LLMs in qualitative coding may thus primarily lie in supporting less interpretive coding tasks, that require less alignment efforts. Their application to complex coding tasks may be most worthwhile when processing large datasets, where the substantial investment of time and resources in alignment can be justified by the scale of the analysis.