DOI: 10.21449/ijate.1807948 ISSN: 2148-7456

The impact of rubric layout on L2 English writing scores and raters’ decision-making

Özgür Şahan, Turgay Han
Show PDF Cite
This study examined the influence of rubric layout on ratings assigned to L2 English essays and raters’ decision-making process in the context of Turkish higher education. Raters scored 24 essays in two rounds with the same analytic rubric presented in standard and reverse layouts, responded to a criteria importance survey at six time points throughout the study, and provided written score explanations to justify their ratings. The two scoring rounds were conducted eight weeks apart, with the rubric categories reversed in the second round. A two-way repeated measures ANOVA revealed significant differences in scores between the two rounds. Multivariate generalizability analyses indicated that the rubric layout influenced variability in scores across raters. Content analysis of the written score explanations revealed a primacy effect, with raters placing greater emphasis on organization and grammar when these categories appeared earlier. However, raters consistently prioritized content and organization when justifying their scores. These findings suggest that the placement of rubric categories can affect raters’ focus and scoring, emphasizing the importance of careful rubric design to ensure consistent assessment outcomes.

More from our Archive