CMITM: A Multimodal Interaction and Local Temporal Modeling Network for Classroom Interactive Behavior Recognition

Accurate recognition of classroom interaction behaviors helps characterize the teaching process, analyze the state of teacher–student interaction, and provide a basis for teaching quality evaluation. Existing methods mostly rely on classroom transcription texts for behavior classification. These methods can capture semantic information in discourse content but have difficulty fully utilizing acoustic cues in speech, such as intonation, pauses, and emotional intensity. Meanwhile, classroom interaction behaviors exhibit obvious contextual dependence, and the current behavior is often associated with preceding behaviors. Modeling only the current utterance can easily overlook the temporal information contained in historical behaviors. To address the above problems, this paper proposes Classroom Multimodal Interaction and Temporal Modeling (CMITM), a multimodal interaction and local temporal modeling method for classroom interactive behavior recognition. CMITM models classroom interactive behaviors from two perspectives: one is the cross-modal feature interaction between text and audio information contained in the current utterance, and the other is the category transition relationships and state evolution processes contained in local historical behaviors. This paper designs cross-modal feature interaction, local historical behavior transition modeling, and adaptive context correction mechanisms to enhance the information complementarity between text and audio and utilize historical contextual information related to the current behavior. Experiments were conducted on a self-built multimodal dataset of classroom interaction behaviors. The results show that CMITM outperforms various unimodal models and multimodal fusion baseline models and achieves better performance on the classroom interaction behavior recognition task.