HoloHand: Bidirectional Motion-Language Modeling for Semantic Hand Interaction in Immersive Environments

Mixed-reality (MR) systems can observe increasingly detailed hand motion, but they still have limited ability to communicate the semantic meaning of continuous hand actions to users. This paper explores language-mediated hand-motion feedback, where natural language serves as an interpretable layer between fine-grained hand motion and MR system response. We present H

H , a bidirectional hand motion-language framework that connects MANO-based hand-motion sequences with natural language. Given observed hand motion, H H generates semantic descriptions that expose action intent, hand roles, and bimanual coordination. Given a language-level intent, it generates corresponding hand-motion sequences that can be rendered as ghost-hand feedback. The framework learns language-compatible hand-motion representations through discrete motion tokenization, latent query alignment, and stable bidirectional training. We evaluate H H on a finegrained hand motion-language dataset covering everyday hand activities and compare it with representative motion-language baselines. Results show improved motion-to-text semantic interpretation, text-to-motion alignment, and motion smoothness, while ablation studies validate the importance of bidirectional integration and the latent motion-language interface. We further implement an MR prototype with runtime scenario probes, illustrating how language-mediated hand-motion feedback can make system interpretation more visible and support visual preview of intended actions in immersive interaction.