Enhancing Human Body Generation in Diffusion Models with Dual-Level Prior Knowledge

The development of diffusion models (DMs) has greatly enhanced text-to-image generation, outperforming previous methods like generative adversarial networks (GANs) in terms of image quality and text alignment. However, accurately generating human body images remains challenging, often resulting in disproportionate figures and anatomical errors, which limits their practical applications in areas such as portrait generation. While previous methods such as HcP have shown promising results, limitations, including the retention of noisy priors, limited understanding of human representation, and restriction of generalization power, still exist due to the specific design of fully supervised learning with only pose-related information. In this study, we introduce a novel method to enhance pretrained diffusion models for realistic human body generation by incorporating dual-level human prior knowledge. Our approach involves learning shape-level details with the human-related tokens in the original prompts, and learning pose-level priors by adding a learnable pose-aware token to each text prompt. We use a two-stage training strategy to rectify the cross-attentions with a bind-then-generalize process, leveraging multiple novel objectives along with adversarial training. Paired comparisons on SD1.5 and SDXL, under identical prompts and generation seeds, show more coherent human bodies than the pretrained models and a LoRA baseline trained on the same images. Image-level metrics improve on SDXL. Those metrics measure image quality, distributional similarity, and preference. The anatomical comparison is carried by the paired images and the attention maps.