DHA-eGCN: Differential Hyperedge Attention-Enhanced Graph Convolution Network for Skeleton-Based Human Action Recognition

Skeleton-based human action recognition (HAR) requires models that preserve the local kinematic structure of the human body while capturing long-range spatiotemporal dependencies under noisy or incomplete joint observations. Traditional Graph Convolutional Networks (GCNs) provide topology-aligned inductive bias but are often limited by local information aggregation from neighboring joints. In contrast, attention-based mechanisms capture global interactions, yet they may attend to spurious correlations when skeletal constraints are weakly enforced. This paper proposes Differential Hyperedge Attention-enhanced GCN (DHA-eGCN), a hybrid architecture that couples structure-aware Differential Hyperedge Attention with multi-scale temporal convolution for spatiotemporal skeleton sequence processing. DHA injects skeletal structure into attention via hop-distance relative positional encoding and hyperedge context tokens generated via joint-to-part pooling. It further employs differential attention to suppress shared noisy correlations and enhance interaction selectivity. To strengthen spatial grounding, an explicit GCN branch is added under partial- or full-depth configurations, where the first four or all ten layers are applied with graph convolutions. The model further employs an ensemble strategy that combines predictions from multiple complementary model instances. Our experiments on NTU RGB+D 60 under the X-Sub and X-View protocols, NTU RGB+D 120 under the X-Sub and X-Set protocols, and Northwestern-UCLA demonstrate that DHA-eGCN consistently outperforms or remains competitive with strong graph-based, transformer-based, and hybrid state-of-the-art methods based on the same four-stream architecture. The best configuration achieves 93.7% and 97.0% on NTU RGB+D 60 X-Sub and X-View, respectively; 90.9% and 91.9% on NTU RGB+D 120 X-Sub and X-Set, respectively; and 97.6% on Northwestern-UCLA.