PROSE: Rubric-Aware Multi-Objective Fine-Tuning of Small Language Models for Automated Assessment of Second-Language Writing

Automated assessment of second-language (L2) writing supports language teaching and testing by providing rapid, consistent scores across multiple rubric dimensions. We study whether small, openly available language models can be turned into reliable analytic raters, and we introduce PROSE, a rubric-aware, multi-objective fine-tuning method built on a compact DeBERTa-v3 encoder with low-rank adaptation. PROSE trains a regression read-out together with a soft-ordinal auxiliary head, and combines a squared-error term for calibration, a distance-aware soft-ordinal term that respects the ordered rubric bands, a concordance term that restores the spread of human ratings and improves agreement with human ratings, and a trait-relational coherence term. On the ELLIPSE corpus of L2 English essays, PROSE raises the quadratic weighted kappa (QWK) from 0.646 for a strong regression baseline to 0.661—a small but statistically reliable gain under a paired test across folds (mean difference +0.015—95% CI [0.007,0.023], and paired t-test p=0.007; p=0.030 after correcting for the overlap between cross-validation training sets), with comparable error and improvements on all seven scoring dimensions, four of which remain significant after Holm correction, and on the Write & Improve (BEA-2019) proficiency data, it matches the strongest baseline on agreement and accuracy and clearly outperforms classification and pure ordinal training. Zero-shot prompting of three strong large language models (GPT-5.4, Claude-Sonnet-4-6, DeepSeek-Chat) reaches a kappa of only 0.13 to 0.21 on the same essays, and a distributional analysis shows that the tested large models produce compressed score distributions with means below the human mean, while the compact fine-tuned model reproduces the human score distribution. Further analyses show that the gain is stable under reasonable changes of the loss weights and holds for truncated and non-truncated essays and across gender, ethnicity, socioeconomic, and grade groups, although these analyses also reveal a shared grade-related bias and a reliance on length and lexical-diversity cues that call for caution in deployment. The results indicate that, for this task, a task-specific objective applied to a small supervised model is far more effective than general-purpose models used zero-shot; we are careful to attribute the gap to task-specific supervision and objective design rather than to model scale alone.