SignFAU: Enhancing Sign Language Production by Integrating Facial Action Units

Sign language production (SLP) plays a significant role in facilitating communication with the deaf community. Significant advancements have been made in modeling the manual aspects of sign language (SL), while non-manual features like facial expressions, particularly through the explicit representation of facial action units (FAUs), remain insufficiently explored. We therefore propose SignFAU, an SLP framework that integrates the 18 FAUs directly into pose representation, together with the body, hand, and facial landmarks. We also introduce PHOENIX14T-FAU, an extension of the PHOENIX14T dataset including FAUs extracted from each video frame, and an improved Sign-VQ model that exploits this richer representation. In a controlled comparison, using the same training configurations and three random seeds, between two versions of the improved Sign-VQ model, one with FAUs and one without, the results show that adding the FAUs raised the BLEU-4 test score from 8.37 to 10.91 when each model was evaluated by the sign language transformer (SLT) evaluator trained on its own dataset. When both models were evaluated by the same landmarks-only SLT evaluator, the gains in BLEU-1 (26.95 to 28.60) and ROUGE (28.13 to 29.32) were statistically significant, whereas the BLEU-4 gain (8.37 to 8.85) was not. Under this evaluator, SignFAU also outperformed the state-of-the-art SLP models, i.e., Sign-VQ baseline and Progressive Transformer, achieving a BLEU-4 score of 8.85 versus 4.11 and 6.76, respectively.