Image-Based Malware Detection Using Deep Convolutional Autoencoder and Random Forest
Shajjia Mazhar, Sadia Fatima, Irfanud Din, Muzamil Dilawar, Ayesha Afzal, Gafur Namazov, Mirjalol AshurovIntroduction:
Image-based analysis has been extensively used for malware classification by converting binary files into grayscale images to extract structural and behavioral patterns. However, common deep learning approaches generally suffer from problems such as non-interpretability, class imbalance, and computational overhead. To overcome these problems, this work proposes a hybrid framework that combines Deep Convolutional Autoencoder (DCAE)- based feature extraction with a Random Forest classifier.
Methods:
We normalized the malware binaries of the Malimg dataset to produce gray-scale images. An unsupervised DCAE was trained to learn compact latent representations, and the encoder was fine-tuned to extract further features. We trained a Random Forest model on these embeddings. We split the data into 70/15/15 for training, validation, and testing. The performance was evaluated with accuracy, precision, recall, F1-score, and a confusion matrix.
Results:
The hybrid DCAE RF model achieved high accuracy and good generalization across malware families. The latent features encoded by the encoder show clearer decision boundaries and improved detection of the minority classes compared to classical CNN baselines. The Random Forest classifier has the advantages of fast training time, interpretability, and robustness to imbalanced data.
Discussion:
The results demonstrate that deep unsupervised feature learning combined with classical ensemble classification offers a scalable solution for malware classification. The model was successful without the use of synthetic oversampling techniques, making it more applicable to real-world datasets.
Conclusion:
The proposed framework is an efficient and interpretable solution for malware classification and demonstrates excellent classification performance, which also opens the door for future research in hybrid and explainable deep learning techniques.