Scoring Accuracy and Differential Rater Functioning of Large Language Models (LLMs) in Evaluating Short Student Essays under Different Conditions: A Simulation Study

This study aimed to evaluate the accuracy and differential rater functioning (DRF) of LLMs in scoring short essays, based on synthetically generated data of 270 essays with varying quality. The LLMs scored the essays under three prompting conditions: without a rubric, with a brief rubric, and with a detailed rubric. Human benchmark ratings were also generated in comparison with the automated scores. The results showed that using detailed rubric improved accuracy and consistency with human raters. Regression analysis further revealed differential rater functioning: the LLMs were more likely to award higher scores to longer essays, favored indirect writing styles, and displayed different outcomes depending on student group membership. These findings suggest that LLMs could serve as useful assessment tools in educational settings, provided they are used within well-defined frameworks and safeguards that reduce DRF while maintaining human judgment in high-stakes or sensitive situations.