A Human-in-the-Loop Reference-Label Curation Framework for Task Classification During Free-Living, Using 3D Animation of IMU Data

Reference-labelled data for training activity-recognition models from wearable inertial measurement units (IMUs) are typically collected in laboratories, which poorly represent real-world movement, and often use video, which raises privacy concerns and does not scale to long recordings. This study evaluated a novel human-in-the-loop annotation framework in which IMU-derived skeletal animations are reviewed by human raters to generate activity labels. Eight participants performed 72 functional tasks in a simulated living environment while wearing IMUs, with synchronized video serving as a reference standard. Data from five participants were analyzed. Two reviewers labelled both IMU animations and video recordings, and intra- and inter-reviewer agreement were compared across a detailed label set and a simplified four-class taxonomy. Using the video annotations as the reference standard, the intra-reviewer macro-averaged F1 for the detailed label set was 61.9 ± 13.5% (Cohen’s κ = 0.67 ± 0.12), and 78.0 ± 11.5% (κ = 0.74 ± 0.13) for the simplified taxonomy. In a sensitivity analysis excluding samples labelled as transitions, macro-F1 was 69.4 ± 10.9% and 92.8 ± 5.3%, respectively. Inter-reviewer agreement for the detailed label set was lower when labelling animations (κ = 0.51 ± 0.06) than video (κ = 0.72 ± 0.02). A main source of disagreement between reviewers was in the labelling of transitions of state, reflecting in part the ambiguous nature of defining their endpoints. These findings support skeletal-animation review as a potential privacy-preserving alternative to video annotation for generating reference labels in simulated living; validation in free-living recordings, and of the reconstructed kinematics, is still required.