A Distributed Island-Based Feature Selection Framework for IoT Intrusion Detection Systems

The widespread deployment of Internet of Things (IoT) environments has led to an increasing number of cyberattacks, highlighting the need for efficient and accurate intrusion detection systems. Over the last few decades, Intrusion Detection Systems (IDSs) have been proposed to tackle this challenge. However, IDSs face challenges when dealing with high-dimensional IoT data that include redundant or irrelevant features, which can lead to increased false positives and decreased detection performance. An optimization-based IDS framework is one of the solutions for reducing data dimensionality and improving detection accuracy. However, the serial implementation of this type of IDS suffers from high computational time as the volume of data and its dimensionality increase. In this paper, we propose a scalable distributed island-based feature selection IDS using Apache Spark (version 3.5.6), called DISFS-IDS. DISFS-IDS follows a two-level partitioning strategy—data and population—to distribute the workload across worker nodes in order to identify the most informative features while achieving high detection accuracy. Using binary and multiclass IoT datasets, the experimental results demonstrate that DISFS-IDS achieves statistically comparable detection performance to the serial SFOA-based IDS while selecting a smaller subset of features. Moreover, DISFS-IDS provides effective feature reduction and competitive or superior performance compared with Spark-based filter feature selection methods. In the scalability analysis, DISFS-IDS achieves significant speedup as the number of islands increases while maintaining high parallel efficiency.