Reinforcement Learning for Aligning Structure-Based Molecular Generation toward Make-on-Demand Chemical Spaces

Abstract

Generative models for structure-based drug design have strong potential to accelerate early-stage drug discovery by enabling the conditional generation of target-specific molecules. However, a major bottleneck is that generated compounds are frequently difficult or impossible to synthesize, limiting their direct experimental validation and utility. We address this by optimizing for membership in make-on-demand chemical libraries, ensuring that generated molecules can be directly purchased and tested. To this end, we apply reinforcement learning to a pocket-conditioned generative molecular model, steering generated molecules toward two large make-on-demand spaces: Enamine REAL and Freedom 4.0. We compare an offline, preference-based strategy (direct preference optimization, DPO) and an online RL strategy (DiffusionNFT) across single-target and multi-target settings. In single-target optimization for KRAS, the fraction of unique purchasable candidates rises from 10.0% to 25.5% for Enamine REAL and from 4.3% to 12.1% for Freedom 4.0, while drug-likeness and docking-based affinity proxies are preserved. In multi-target optimization, DPO achieves consistent improvements across diverse protein targets, including a 23-fold increase for the challenging CDK4 pocket. These results demonstrate that RL alignment can substantially improve the experimental accessibility of SBDD candidates without sacrificing structural or chemical quality.