A Multi‐View and Multimodal Framework for Human Action Recognition in Human–Robot Collaborative Assembly

ABSTRACT

With the development of flexible manufacturing and smart factories, human–robot collaborative assembly is moving towards large‐scale deployment. However, lighting variations, occlusions, and the complex motion characteristics of collaborative assembly actions often challenge reliable perception and fine‐grained action recognition. To address these challenges, this paper proposes a task‐oriented multi‐view and multimodal framework for collaborative assembly action recognition. First, an assembly‐oriented whole‐body pose estimator termed HigherPose is developed to emphasise manipulation‐related keypoints and improve pose localization under complex industrial conditions. On COCO‐WholeBody, HigherPose reaches 67.1 AP on whole‐body keypoints, exceeding ZoomNAS by 1.7 AP while reducing computation by 0.5 GFLOPs. Furthermore, a viewpoint–modality coordination network, termed MMARN (multi‐view multimodal action recognition network), is designed to exploit complementary workspace‐context and human‐motion information from red–green–blue (RGB) observations and skeleton sequences. Evaluated on the industrial collaborative assembly dataset InHard65, MMARN achieves the best validation performance among the evaluated configurations, with 99.39% Top‐1 accuracy, and qualitative results show reliable discrimination of fine‐grained operational intentions. Overall, the results demonstrate the effectiveness of the proposed task‐oriented two‐stage framework for fine‐grained human–robot collaborative assembly action recognition.