Real-time Recognition of Contextual Composite Activities from Asynchronous Multi-IMU Sensor Streams

Human Activity Recognition (HAR) is foundational to intelligent systems delivering personalized healthcare and wellbeing services. Wearable devices equipped with inertial measurement units (IMUs), such as smartwatches and earables, have enabled non-intrusive recognition of motion-driven activities. However, real-time recognition of contextual composite activities – temporally overlapping actions initiated by distinct body parts – remains challenging due to asynchronous multi-IMU data streams, where data collected from multiple IMU-enabled wearable devices may exhibit heterogeneous sampling rates, independent device clocks, and communication delays, resulting in temporally misaligned sensor readings, variability in sensor quality, and limited contextual cues from low-resolution streaming data. To address these challenges, we propose Multisource Self-Supervised Learning (MSSL), a lightweight framework that exploits unlabeled multisource-IMU data to enhance composite activity recognition in real time. MSSL incorporates (1) a dynamic source selection mechanism leveraging probability density estimation to adaptively identify primary and auxiliary sensor sources, and (2) a multimodal contrastive learning strategy to improve feature representations across heterogeneous sensor modalities. Evaluations on an in-house dataset and four benchmark HAR datasets demonstrate that MSSL achieves (i) 95.26% accuracy on our in-house composite activity dataset, (ii) 85.93% on PAMAP2-ADL, (iii) 80.08% on Opportunity, (iv) 91.18% on RealDisp, and (v) 69.41% on PAMAP2-Locomotions. Notably, MSSL maintains robust performance even with only 10% labeled data (78.05% accuracy) while achieving low inference latency of 0.6328 milliseconds. These results underscore MSSL’s potential for reliable, context-aware composite activity recognition in real-time applications, paving the way for more responsive and personalized services.