Driver Action Recognition Based on a Multi‐Factor Fusion Network

ABSTRACT

Driver action recognition technology significantly enhances the interaction experience between drivers and vehicles and improves driving safety by fusing multiple sensor modalities such as infrared and depth. Current methods mainly consider the driver's skeleton points and vectors, overlooking that different human body joints can function independently. Therefore, it is crucial to construct connections that reflect the semantic relationships between different parts of the human body. However, extracting such relationships remains a challenging task. We propose a novel multi‐factor fusion network (MFFN) for driver action recognition to tackle this issue. Specifically, a group‐connected graph convolutional network (GC‐GCN) is proposed to build a group‐connected graph to hierarchically represent the human skeleton and extract spatiotemporal skeleton features through group‐connected graph convolutions and temporal convolutions. Afterward, MFFN fuses skeleton and context features with an attention‐based fusion method, dubbed the intergroup attention aggregation (IAA) module. The IAA module integrates scene context associated with each joint set to global context and calculates channel interactions per joint according to its spatial location. Experiments are conducted on the publicly available Drive&Act and Driver Monitoring Dataset (DMD) datasets. MFFN achieves 44.33% mean per‐class accuracy on Drive&Act and 95.63% overall classification accuracy on DMD, showing competitive performance compared with mainstream driver action recognition methods. Both qualitative and quantitative analyses demonstrate the effectiveness of the proposed model.