BHM-IDS: Behavior-Driven Hierarchy and Multi-Dataset Training for Cross-Dataset Generalization

Digital infrastructures are increasingly exposed to diverse and evolving cyber threats, highlighting the need for robust intrusion detection systems (IDSs). Although machine learning (ML)-based IDSs have achieved strong performance, most existing frameworks are still developed and evaluated mainly under intra-dataset settings, providing limited evidence of their ability to generalize across unseen environments. Moreover, few studies go beyond simply reporting cross-dataset performance to propose dedicated mechanisms for improving generalization. To address this limitation, we propose BHM-IDS, a three-stage hierarchical intrusion detection framework that combines behavior-driven hierarchy with multi-dataset training to improve generalization. The first stage performs binary detection of benign versus malicious traffic, while the second stage classifies malicious traffic into two behaviorally distinct groups: the first corresponding to flood and exhaustion attacks and the second to infiltration and exploitation attacks. The final stage performs fine-grained attack classification through two specialized multi-class classifiers. To expose the framework to more diverse attacks, CIC-IDS2017 is enriched with CIC-DDoS2019 during training, while CSE-CIC-IDS2018 is used as an external test dataset to evaluate generalization. The cross-dataset validation results yielded stage-wise accuracies of 0.93, 0.96, and 0.99, respectively, while the complete end-to-end framework achieved a weighted recall of 0.93. Recall values ranging from 0.76 to 1.00 were obtained for several major classes, including benign traffic, Patator, DoS, and DDoS, although limitations remained for certain attack categories, particularly Web Attack. Overall, the proposed framework demonstrated promising and competitive performance compared with simpler frameworks and existing state-of-the-art approaches. These findings highlight the potential of combining behavior-driven hierarchical classification with multi-dataset training to improve cross-dataset generalization in IDSs.