A Two-Stage Selective Deep Learning Framework for Malware Family Classification in Encrypted Network Traffic

Encrypted network traffic limits traditional payload-based inspection and makes malware family attribution more difficult. This study proposes a two-stage selective deep learning framework for malware family classification in encrypted traffic. The framework was evaluated on two datasets: USTC-TFC2016 as a 20-class task containing 10 malware families and 10 benign applications, and MCFP-14-M2 as a 14-class malware classification task. Stage 1 uses a lightweight 1D convolutional neural network for fast screening, while uncertain samples are escalated to Stage 2, where NetMamba performs refined classification. The models were evaluated using accuracy, macro-F1, weighted F1-score, per-class results, inference latency, escalation rate, and edge-oriented deployment measurements. On USTC-TFC2016, the standalone 1D-CNN achieved 99.02% accuracy, while standalone NetMamba achieved 99.51%. At the selected threshold of τ = 0.80, the two-stage framework achieved 99.34% accuracy, 0.9664 macro-F1, and 0.9934 weighted F1-score while escalating only 11.03% of samples and reaching an expected latency of 0.6494 ms. On MCFP-14-M2, the corresponding results were 93.02% accuracy, 0.9311 macro-F1, a 28.71% escalation rate, and 2.25 ms expected latency. The results show that selective escalation improves over the lightweight model and approaches NetMamba-level accuracy while reducing average computational cost.