Beyond AUC: Evaluating Deep Knowledge Tracing Through Temporal Coherence and Post Hoc Explanations

Deep knowledge tracing (KT) models estimate students’ evolving skill mastery from sequences of question–response events and predict whether the next response will be correct. They are often compared using AUC, a score of how well predictions distinguish correct from incorrect answers. We evaluate two complementary aspects beyond AUC: temporal prediction behavior for 11 KT models on four educational datasets, and attribution diagnostics on ASSISTments 2017 data. The relationship between AUC and response-direction consistency varies by dataset: models with higher AUC sometimes show more, and sometimes fewer, prediction drops after correct answers or prediction rises after incorrect answers on repeated skills. Reference baselines further show that stability must be interpreted together with responsiveness, because a model that never updates its prediction also appears perfectly stable. In an exploratory ASSISTments 2017 analysis of 50 histories from 33 learners per model, gradient-based selected question–response events produce modest, model-dependent signed probability changes (original minus post-replacement probability) relative to recent-history, same-skill, and random controls. For practitioners comparing KT models, the paper contributes two practical checks to report alongside AUC: whether predictions respond coherently after student answers, and how explanation-selected history events compare with simple recent-history or same-skill alternatives under a shared intervention.