MS²FL: Modality-shared and Modality-specific Feature Learning for Multimodal Emotion Recognition

Multimodal emotion recognition based on complementary physiological signals such as electroencephalogram (EEG) and eye movements can effectively reflect human emotional states, demonstrating significant potential in fields such as rehabilitation monitoring and driving safety. However, existing multimodal emotion recognition approaches often focus solely on either feature fusion or feature alignment strategies, lacking a unified framework capturing both complementary and modality-specific representations. To address this limitation, we propose a modality-shared and modality-specific feature learning framework (MS \({}^{2}\) FL) for multimodal emotion recognition. Specifically, a dual-path encoder is employed first to extract the spatiotemporal frequency features of EEG signals and dynamic response features of eye movement signals, respectively. Then, a specificity-preserving feature distribution alignment mechanism is introduced to alleviate distribution discrepancies between modalities while reinforcing modality-specific modeling of distinctive features. Finally, a feature distribution enhancement fusion strategy is utilized to effectively integrate the shared features across modalities, thus improving the representational capacity of the fused features. Experimental results demonstrate that MS \({}^{2}\) FL achieves recognition accuracies of 96.99% and 89.83% on the SEED and SEED-V datasets, respectively, significantly improving emotion recognition performance. This study provides a concise and effective solution for emotion perception neurotechnology and offers a practical solution for emotion perception health monitoring and rehabilitation.