Cross-Modal Adaptive Feedback Knowledge Distillation for Enhanced sEMG-Based Semantic Recognition

In semantic recognition based on surface electromyography (sEMG), low signal-to-noise ratios and physiological differences across subjects limit the ability of unimodal sEMG features to fully represent speech semantics, constraining recognition accuracy. Existing multimodal fusion methods can improve accuracy but often require large models and high computational complexity. Mismatches in feature spaces and learning capacity between multimodal teachers and unimodal students further limit cross-modal knowledge transfer. To address these issues, this paper proposed cross-modal adaptive feedback knowledge distillation (CAFKD). The multimodal teacher model uses Bi-GRU to fuse temporal features from synchronously acquired audio, lip image sequences, and sEMG signals, generating high-level representations that encode complementary semantic information. This multimodal knowledge was transferred to an sEMG-only student through a joint loss combining classification, output distribution distillation, and intermediate feature distillation. In addition, the cross-modal adaptive feedback mechanism (CAF) periodically adjusted the teacher using stagewise output feedback from a temporary student, enabling teacher guidance to better match the student’s learning state. In subject-independent evaluation on the AVE-Speech dataset, CAFKD achieved 78.13% accuracy on the 101-class closed-set short-utterance classification task, exceeding the unimodal Bi-GRU baseline by 4.82 percentage points. Ablation experiments showed that adding intermediate feature distillation to output distribution distillation increased test accuracy by 1.49 percentage points. CAF further improved accuracy by 1.46 percentage points over joint distillation. The student model also retains the same parameter count and FLOPs as the Bi-GRU baseline, at 4.42 M and 0.29 G, respectively. These results indicate that CAFKD improves recognition performance on unseen subjects while preserving a lightweight architecture for sEMG-only inference.