Manifold‐Aligned VQ Convolutional Adapters for Parameter‐Efficient Cross‐Lingual Speech‐Driven 3D Facial Animation

Abstract

Speech‐driven 3D facial animation has achieved remarkable realism, yet existing models suffer from severe performance degradation on unseen languages due to discrepancies in articulation patterns and motion distributions. We attribute this limitation to a distribution shift on the latent motion manifold, where pretrained vector‐quantized (VQ) codebooks fail to adequately cover target‐language articulation regions. To address this, we propose Manifold‐Aligned VQ Convolutional Adapters for parameter‐efficient cross‐lingual adaptation. Instead of retraining the entire backbone, we insert lightweight 1D convolutional adapters into the post‐quantization and decoder stages, formulating adaptation as a localized geometric correction on the motion manifold. Theoretically, we demonstrate that under the local bi‐Lipschitz continuity of the pretrained decoder, minimizing reconstruction loss effectively aligns the latent motion distribution with the target language. Unlike conventional linear adapters, our convolutional design explicitly captures local temporal dependencies and higher‐order motion dynamics critical for language‐specific articulation. To support systematic evaluation, we construct CMLR‐3D, a paired Mandarin speech–3D facial motion dataset. Extensive experiments on CodeTalker and ARTalk show that our method significantly improves lip synchronization and motion naturalness, consistently outperforming full and partial fine‐tuning baselines in both quantitative metrics (LVE, AVLR) and perceptual user studies. Notably, the framework updates only 0.46%–1.26% of parameters while preserving pretrained motion priors. This work provides a theoretically grounded, highly efficient paradigm for multilingual virtual avatar deployment. (Anonymous link http:github.com/GreenTomato‐hugo/MAVQ‐Adapter )