Can Artificial Intelligence (AI) Grade Like a Teacher? Evaluating Skill-Based Open-Ended Questions in Mathematics

This study examines the scoring performance of artificial intelligence (AI) tools compared with human raters for grading skill-based, open-ended mathematics questions. 196 secondary school students’ responses to 10 open-ended items were assessed by three expert mathematics teachers and four LLM-based AI systems: ChatGPT, Gemini Flash, Microsoft Copilot, and DeepSeek Think, using a zero-shot learning approach with a holistic rubric. Scores were compared using Friedman and Wilcoxon signed-rank tests. Interrater agreement was examined through Krippendorff’s Alpha, exact agreement rates, and Bland-Altman analyses, while interrater reliability was assessed using Intraclass Correlation Coefficient and Cronbach’s Alpha Coefficient. A subsample of 20 scripts was scored three times by each AI tool to examine intra-rater reliability. Results revealed significant differences among raters, with ChatGPT and Microsoft Copilot assigning significantly higher scores than human raters. Interrater agreement was moderate and fell below the level teachers reached among themselves on every item, with Bland-Altman analyses showing that DeepSeek Think had the lowest bias but the highest inconsistency. In contrast, Gemini Flash showed the most consistent scoring behavior. Reliability estimated for the seven-rater average was excellent, whereas the corresponding single-rater estimate was moderate. Across repeated sessions, none of the tools closely reproduced their own marks. Hallucination phenomena were observed, with AI tools inventing non-existent student responses. The findings suggest that while AI tools show promise for automating assessment, they lack both agreement with human judgment and the self-consistency required for independent use in high-stakes educational evaluation, highlighting the continued need for human oversight in assessing higher-order mathematical thinking.