Feature Misalignment: A Critical Issue in Cross‐Architecture KD

ABSTRACT

This article addresses the challenge of deploying complex deep learning models on resource‐constrained edge devices for real‐time security applications. The core objective is to transfer the capabilities of a high‐computation model to a lightweight model through feature‐based KD (KD) and optimization techniques like quantization and pruning. The teacher model used for experiments was DeiT‐base, pretrained on ImageNet. It was then fine‐tuned to custom datasets. Student model, MobileNetV3 Large was trained from scratch using KD on custom datasets tailored for security tasks, including Eyegaze, Emotions, and Weapons. Through KD, the student model learned to replicate the teacher model's performance efficiently. Experiments demonstrated that the optimized student model performs effectively in real‐time analysis of images and videos on edge devices. This provides a practical tool for security forces in threat detection and surveillance. The findings highlight the potential of KD and quantization to make advanced machine learning models deployable in real‐world security applications, offering enhanced performance in resource‐limited environments. Future work will explore multitask learning, quantization‐aware training, MobileNetV4 applications, custom dataset development, and innovative loss functions for improved cross‐architecture distillation.