AdvDPO: Adversarial Negative-Augmented Preference Optimization for Text-to-Image Diffusion Models

Diffusion Direct Preference Optimization (DPO) provides a reward-free route for aligning text-to-image diffusion models with human preferences, but its supervision is commonly restricted to fixed winner–loser pairs. When dispreferred examples cover only a limited subset of failure modes, optimization can over-emphasize local dataset preferences and provide insufficient corrective contrasts. We present AdvDPO, an adversarial candidate-negative preference optimization framework for diffusion models. AdvDPO couples the policy with an auxiliary adversarial diffusion network that produces a local perturbation direction from the preferred diffusion state. The resulting candidate augments the original preference pair while preserving the prompt, timestep, and noise target. The policy learns from both real and synthesized preference pairs, whereas the adversarial network is alternately trained with reversed preference supervision and an attack objective that keeps synthesized candidates challenging for the current policy. On three test sets and two diffusion backbones, AdvDPO achieves the strongest automatic scores among the compared methods and the largest preference share in the anonymous user study.