An Immersive Closed‐Loop Human Machine Interaction System Enabled by Dual‐Modal On‐Skin Sensors

ABSTRACT

Human‐machine interaction (HMI) as the critical bridge between human and digital world, has garnered significant attention in embodied intelligence (EI). However, current HMI systems are plagued by several key challenges, including narrow perception modality, insufficient recognition precision of human intent, and unidirectional communication. Here, an immersive closed‐loop HMI system (ICL‐HMIS) is developed, integrating breathable dual‐modal on‐skin (DMO) sensors, a multimodal fusion model (MF model), and a haptic feedback unit. By the localized electrical modification on the soft porous scaffold, the DMO sensor can simultaneously measure bioelectrical and biomechanical information. Meanwhile, the composite signals measured by the DMO sensor can be effectively separated into surface electromyography (sEMG) and triboelectric (TE) signals without spatiotemporal mismatch by wavelet analysis. A MF model is proposed to achieve an accuracy of 93.5% for fine‐grained gesture recognition across 15 classes, which encompasses the classification of basic gesture categories and precise motion intensities. Moreover, by integrating a haptic feedback unit that monitors grip force with programmable vibrotactile stimuli, the ICL‐HMIS enables non‐destructive robotic remote manipulation based on bi‐directional information loop of dual‐modal biosignals and somatosensory feedback.