A multiscale and multilevel fusion network based on ResNet and MobileFaceNet for facial expression recognition

Abstract

There are complex correlations between facial expression and facial landmarks in facial images. The facial landmarks detection technology is more mature than the facial expression recognition methods. Considering this, in order to better address the problem of interclass similarity and intraclass discrepancy in facial expressions recognition (FER), facial landmarks are used to supervise the learning of facial expression features in our work, and a multiscale and multilevel fusion network based on ResNet and MobileFaceNet (MMFRM) is proposed for FER. Specifically, the authors designed a triple CBAM feature fusion module (TCFFM) that characterises the correlation between facial expression and facial landmarks to better guide the learning of expression features. Furthermore, the proposed loss function of removing facial residual features (RFLoss) can suppress facial features and highlight expression features. We extensively validate our proposed MMFRM on two public facial expression datasets, demonstrating the effectiveness of our method.