Shuyi Li, Hongchao Hu, Shumin Huo, Hao Liang

Clean, performance‐robust, and performance‐sensitive historical information based adversarial self‐distillation

  • Computer Vision and Pattern Recognition
  • Software

AbstractAdversarial training suffers from poor effectiveness due to the challenging optimisation of loss with hard labels. To address this issue, adversarial distillation has emerged as a potential solution, encouraging target models to mimic the output of the teachers. However, reliance on pre‐training teachers leads to additional training costs and raises concerns about the reliability of their knowledge. Furthermore, existing methods fail to consider the significant differences in unconfident samples between early and late stages, potentially resulting in robust overfitting. An adversarial defence method named Clean, Performance‐robust, and Performance‐sensitive Historical Information based Adversarial Self‐Distillation (CPr & PsHI‐ASD) is presented. Firstly, an adversarial self‐distillation replacement method based on clean, performance‐robust, and performance‐sensitive historical information is developed to eliminate pre‐training costs and enhance guidance reliability for the target model. Secondly, adversarial self‐distillation algorithms that leverage knowledge distilled from the previous iteration are introduced to facilitate the self‐distillation of adversarial knowledge and mitigate the problem of robust overfitting. Experiments are conducted to evaluate the performance of the proposed method on CIFAR‐10, CIFAR‐100, and Tiny‐ImageNet datasets. The results demonstrate that the CPr&PsHI‐ASD method is more effective than existing adversarial distillation methods in enhancing adversarial robustness and mitigating robust overfitting issues against various adversarial attacks.

Need a simple solution for managing your BibTeX entries? Explore CiteDrive!

  • Web-based, modern reference management
  • Collaborate and share with fellow researchers
  • Integration with Overleaf
  • Comprehensive BibTeX/BibLaTeX support
  • Save articles and websites directly from your browser
  • Search for new articles from a database of tens of millions of references
Try out CiteDrive

More from our Archive