MaGOS-IDS: A Mahalanobis-Enhanced OpenMax Method for Graph Neural Network-Based Intrusion Detection

Graph Neural Networks achieve strong closed-set accuracy in network intrusion detection but cannot flag zero-day attacks, because the closed-world assumption forces every input into a known class. OpenMax adds an Extreme Value Theory reject option, yet its Euclidean distance ignores the class-conditional covariance that encodes attack-specific structure, which produces unreliable tail models and rejection thresholds. We propose MaGOS-IDS, which extracts topology-aware embeddings with an edge-aware GCN that fuses flow-level edge features directly into message passing, whitens each class with a regularized Mahalanobis distance so the reject decision respects per-class variance and correlation, and calibrates a per-class EVT tail on these distances to set an attack-pattern-aware rejection boundary without a hand-tuned cutoff. We provide a theoretical justification via a peaks-over-threshold argument: whitening removes the per-class covariance dependence of the distance tail, so a single extreme-value tail model calibrates consistently across classes. On three benchmarks (NF-BoT-IoT, CIC-IDS-2017, UNSW-NB15) under withheld zero-day families, MaGOS-IDS raises open-set AU-PR over the Euclidean OpenMax baseline (0.932 vs. 0.848 on UNSW-NB15) while adding negligible inference cost.