Real-Time Enhancement of Facial Expression Recognition with Lightweight Divide-and-Conquer Algorithm

Facial Expression Recognition (FER) plays a pivotal role in human–machine interaction (HMI), supporting applications such as driver monitoring, healthcare, and intelligent systems. Despite significant progress, real-time FER systems still face critical challenges, particularly reduced accuracy in distinguishing visually similar expressions and high computational complexity that limits deployment on resource-constrained platforms. In this work, we propose a novel hierarchical FER framework that integrates a lightweight PAtt-Lite MobileNetV1 backbone with a divide-and-conquer learning strategy. The proposed architecture adopts a two-stage hierarchical design: a primary lightweight model performs initial emotion classification, while low-confidence predictions are selectively forwarded to specialized binary expert models for refinement. These expert models are trained on commonly confused emotion pairs, enabling enhanced discrimination where it is most needed. By activating expert models only on demand, the framework improves classification performance without incurring substantial computational overhead. Experimental evaluation on the RAF-DB dataset demonstrates strong performance, achieving a validation accuracy of 93.75% ± 0.35% (mean ± standard deviation over five random seeds). On the held-out test set, the primary model achieves 70.00% ± 1.60%, while the hybrid framework improves performance to 70.72% ± 1.37%, all while maintaining real-time operation at approximately 101.6 FPS on CPU-only hardware. Unlike many FER studies that evaluate performance using data distributions involved in training or model selection, this work employs a completely independent external test set that remains isolated from all training, validation, and tuning stages. Consequently, neither the main model nor the expert classifiers encounter these samples during learning, ensuring that the reported results reflect genuine generalization to unseen data and preserving the integrity of the evaluation process, including confidence-threshold and expert-routing optimization. These results highlight the proposed framework’s effectiveness in balancing accuracy, efficiency, and reliable real-world generalization, making it a promising solution for real-time edge deployment in practical HMI scenarios.