Comparing LLM automated grading with IELTS writing essays: evidence from rubric prompting and model stability
Jaber Jemai, Nourchène Elleuch Ben AyedPurpose
This study assesses the extent to which large language models (LLMs) reproduce the IELTS Writing Task 2 band scores assigned by professional IELTS examiners. It also evaluates whether incorporating the official IELTS band descriptors into the grading prompt improves scoring accuracy and agreement.
Design/methodology/approach
Six LLM routes (GPT-4o, incept5/llama3.1-claude, Gemma 3, Aya, deepseek-local and deepseek-remote) graded a stratified 100-essay subset from a public IELTS benchmark containing dataset-provided band labels. Each essay was graded 10 times under No Rubrics (NR) and With Rubrics (WR). Scores were treated as single-label multi-class ordinal classifications over the 4.0–9.0 half-band scale and evaluated using accuracy, precision, recall, F1-score, Cohen's kappa, Fleiss' kappa, grading gaps, directional bias and run-to-run stability.
Findings
Rubric guidance improved average performance and most model-level classification metrics, with cross-model mean gains of 8.8% in accuracy, 14.9% in precision, 23.2% in recall and 18.7% in F1-score. Within the parseable-response subset, GPT-4o achieved the highest reported accuracy and the strongest agreement with the dataset-provided labels; however, its results rest on a substantially smaller set of valid responses than the other routes, and agreement remained moderate at best. Inter-model agreement was negligible; most models inflated scores, and in this model pool, the local/Ollama routes showed larger gaps and instability. The results do not support unsupervised high-stakes marking.
Originality/value
The study provides an applied, risk-aware evaluation protocol for LLM-assisted writing assessment by jointly reporting correctness, agreement, directional bias and run-to-run stability rather than accuracy alone. It supports rubric-anchored, human-in-the-loop LLM use for formative feedback, triage and moderation, not examiner replacement.