HEADHUNTER: Training-Free Annotated Dataset Synthesis via Self-Guided Diffusion Transformer Attention Head Selection

Pixel-level annotation remains a major bottleneck for semantic segmentation, motivating methods that synthesize image–label pairs directly from generative models. Prior synthetic dataset generators typically obtain pseudo-labels from cross-attention maps or learned decoders over generative features; however, recent text-to-image (T2I) models increasingly use multimodal diffusion transformers (MM-DiTs), where concept localization is no longer exposed through a single cross-attention pathway but instead distributed across many layers and attention heads. Existing MM-DiT localization methods address this by aggregating saliency across heads, but we observe that this averaging can dilute clean target localizers due to attention head heterogeneity. We introduce HEADHUNTER, a training-free segmentation framework that uses aggregate concept saliency as a self-guided proxy to select the single attention head that best localizes a queried textual concept, yielding cleaner segmentation masks. We then use HEADHUNTER to turn target classes into training data automatically: a large language model (LLM) diversifies prompts, an MM-DiT generates images, HEADHUNTER produces pseudo-labels, and a vision language model (VLM) verifies each image–mask pair before acceptance. HEADHUNTER achieves strong zero-shot segmentation performance (81.2 mIoU on PASCAL VOC2012 and 73.2 mIoU on ImageNet-Segmentation), outperforming head aggregation and other interpretability methods. Using only our generated image–label pairs, we train segmentation models which reach 64.9 mIoU on VOC2012 validation, matching or outperforming comparable synthetic dataset generators and showing that the proposed pipeline produces effective dense labels without human intervention.