One Prompt Does Not Fit All: Comparing Uniform and Adaptive Code‐Specific Prompting for Deductive Qualitative Coding With Large Language Models
María Jesús Rodríguez‐Triana, Luis P. Prieto, Mohamed Saban, Juan I. Asensio‐Pérez, Cristina Villa‐Torrano, Inmaculada Haba‐Ortuño, Denis GilletABSTRACT
Background
Large language models (LLMs) have demonstrated potential for (semi‐)automating the qualitative analysis of unstructured data, particularly in deductive qualitative coding using codebooks. While prior research has shown the feasibility of this technology, model performance seems to vary depending on the nature of the constructs being coded. However, existing approaches typically apply a single LLM prompting strategy across entire datasets, often of discourse transcripts or questionnaire data, using a single coding scheme or theoretical frame.
Objectives
We aim to explore whether code‐specific adaptive prompting, where LLM prompts are customised based on expert coders' or data‐driven rules for specific codes, outperforms uniform prompting in terms of agreement with a negotiated human coding and, secondarily, reduced computational cost when agreement is maintained.
Methods
We apply this approach to analyse the description of 35 multimedia learning designs (comprising 758 items/activities) created by teachers in an inquiry‐based learning digital platform. Two human coders and two open‐weights LLMs (Llama 3.3 and GPT‐OSS) coded the dataset, attending to three different pedagogical frameworks.
Results and Conclusions
Our results indicate that, with both LLMs, the data‐driven adaptive prompting reached similar or higher agreement with the reference point (i.e., the coding agreed by humans) than uniform prompting strategies (namely, zero‐shot and few‐shot with and without context), while reducing costs. These results may be especially relevant for the coding of large‐scale datasets, highlighting opportunities for human–AI collaboration in deductive qualitative coding of educational data.