A Robust LightGBM-Based Machine Learning Framework for Network Outlier Detection

The growing volume and complexity of network traffic have made manual and signature-based monitoring insufficient for detecting malicious activity. This paper presents a robust machine learning framework, based on the Light Gradient Boosting Machine (LightGBM), for detecting outliers in network traffic data. The framework integrates data cleaning, one-hot encoding, logarithmic transformation, mutual-information-based feature selection, standardisation and a regularised LightGBM classifier trained with early stopping. It was evaluated on the NSL-KDD benchmark and compared with Logistic Regression, Random Forest and a Support Vector Machine under identical conditions. On a held-out test set of 25,193 connections, LightGBM achieved an accuracy of 99.933%, a precision of 99.957%, a recall of 99.898%, an F1-score of 99.927% and a ROC-AUC of 0.99999, making only 17 errors. It outperformed all baseline models while training about four times faster than Random Forest. SHAP-based interpretation showed that byte volume, connection rate, service concentration and SYN-error features drive detection. Robustness experiments confirmed stable performance across data splits and random seeds, and resilience to severe class imbalance, with an F1-score of 97.74% when outliers formed only 1% of traffic. On an independent test set containing unseen attack types, accuracy fell to 78.54%, identifying generalisation to novel attacks as the main limitation. The results show that LightGBM offers an accurate, efficient and interpretable foundation for network outlier detection.