Speech-Driven Facial Expression Synthesis Using Emotion-Aware Parametric Facial Descriptors

Speech-driven facial expression synthesis aims to generate realistic and emotionally consistent facial motion from speech signals for natural human–computer interaction and expressive virtual agents. This paper presents an emotion-aware framework based on parametric facial descriptors that explicitly models the relationship among speech, emotion, and facial motion. The framework consists of three components. First, a Speech Emotion Recognition (SER) module predicts frame-wise emotion probabilities over six categories: neutral, happy, anger, surprise, sad, and fear. Second, an emotion-conditioned facial descriptor represents interpretable facial expression parameters through a lip descriptor and a head–eye descriptor. The lip descriptor models speech-synchronous mouth articulation using a lookup table of canonical lip geometries, while the head–eye descriptor captures emotion-dependent head pose, eyebrow motion, eyelid activity, and gaze direction. Third, an envelope-driven temporal modulation mechanism generates Animated Facial Expression Parameters (AFP) according to the temporal energy structure of speech. The resulting AFP sequence drives a LivePortrait-based portrait synthesis model to generate emotionally expressive facial images. Experimental results demonstrate visually coherent and temporally consistent facial motion aligned with both speech articulation and emotional state.