Robust backdoor attacks against fine-tuning-based transfer learning via self-attention distillation

Backdoor attacks inject hidden behaviors into deep neural networks, causing them to behave normally on benign inputs but produce attacker-specified results under trigger conditions, which poses significant threats to model integrity. However, transfer learning, as a prevalent paradigm for training neural networks, often compromises backdoors during the fine-tuning process. To address this issue, we propose Robust Backdoor Attacks against Fine-tuning-based Transfer Learning via Self-Attention Distillation (RBA-SAD), a robust backdoor attack scheme that manipulates model’s ability to recognize the trigger via layer-wise and top-down self-attention distillation, effectively withstanding the impact of fine-tuning. Additionally, imperceptible noise is employed as a trigger to evade defenses based on input purification. To enhance the effectiveness of self-attention distillation, we optimize trigger generation to achieve multi-layer feature collisions, ensuring that the representations of poisoned samples at each corresponding layer are similar to those of clean samples. Extensive experiments on three benchmark transfer-learning tasks show that RBA-SAD achieves over 97% attack success rate while remaining stealthy against seven state-of-the-art backdoor defenses. Moreover, it generalizes well to additional settings, including Vision Transformers, ImageNet-scale transfers, and parameter-efficient fine-tuning with Low-Rank Adaptation.