Ratio-Constrained Offline Reinforcement Learning: Consensus-Gated Policy Improvement via Critic Gradient Agreement

Offline reinforcement learning aims to learn effective policies from fixed datasets without further interaction with the environment. However, limited behavior-policy coverage can lead to value-estimation errors and unreliable policy improvement, particularly when multiple critics provide conflicting update directions. To address these challenges, we propose Probability-Ratio-Constrained Q-Learning (PCR-QL), which integrates behavior-density modeling, conservative ensemble-based Q-learning, and probability-ratio-regularized policy extraction to control distributional shift. Building on this framework, we further introduce Consensus-Gated PCR-QL (CG-PCR-QL), which evaluates critic-specific actor gradients using shared states and policy noise and adaptively attenuates the ensemble-mean policy-improvement direction according to the weakest critic-gradient agreement. Theoretically, we establish results connecting ratio-constrained Bellman approximation, behavior-density estimation errors, policy-feasibility repair, and multi-critic gradient geometry, together with conditional guarantees for completed optimizer updates. Experiments on eleven Datasets for Deep Data-Driven Reinforcement Learning (D4RL) benchmark tasks compare nine policy-extraction rules across ten matched runs. CG-PCR-QL achieves a mean performance improvement of +6.32 (95% CI [5.14, 7.50]) over PCR-QL and a pooled interquartile-mean (IQM) improvement of +7.19 [5.63, 8.69]. Furthermore, an analysis of 2400 paired Adam-update events shows an increase of +13.0 [11.1, 14.9] percentage points in the frequency of simultaneous improvement across all critics. Additional ablations and diagnostics examine the effects of gradient consensus, behavior-density estimation, parameter sensitivity, and computational cost. These results highlight the effectiveness of integrating probability-ratio constraints with critic-gradient consensus for more reliable policy improvement in offline reinforcement learning.