Visual–Language Joint Modelling and Conditional Flow Matching for High‐Fidelity Cross‐Modal Animation Synthesis
Siyu Song, Chuan LiABSTRACT
Synthesising high‐fidelity animation from textual descriptions requires coordinated modelling of non‐rigid motion, semantic alignment and character appearance across video sequences. This study proposes a visual–language joint modelling framework that combines a multimodal large language model (MLLM)‐based semantic encoder, conditional flow matching (CFM) and a dual‐stream consistency module. The Q‐former adapter maps semantic features into the video latent space, whereas CFM parameterises continuous transport from noise towards encoded video data. Frequency‐domain filtering, reference attention, and an optical‐flow consistency loss are used to constrain appearance drift and inter‐frame artefacts. The framework is evaluated on AnimeShooter, with a curated WebVid‐10M subset incorporated during pretraining. The proposed model obtains an FVD of 82.16, a CLIPSIM of 0.348 and an adjacent‐frame ICS of 0.885.