Imagine to Ensure Safety in Hierarchical Reinforcement Learning

This work investigates the safe exploration problem in reinforcement learning, where an agent must maximize cumulative performance while simultaneously satisfying safety constraints. This challenge becomes particularly difficult in long-horizon tasks because safe reinforcement learning methods often become overly conservative, restricting exploration and preventing agents from reaching distant goals, while estimation errors accumulate over extended horizons. We propose Imagine To Ensure Safety in Hierarchical Reinforcement Learning (ITES), which combines a learnable world model with high-level and low-level policies to promote safety at both hierarchical levels. The novelty of ITES lies in jointly ensuring safe subgoal generation and safe subgoal execution within a hierarchical framework. The high-level policy generates intermediate subgoals that guide exploration toward safe regions, while the low-level policy uses imagined rollouts in the learned world model to reduce unsafe behavior during subgoal execution. We evaluate ITES on the long-horizon SafeAntMaze C-shape, SafeAntMaze W-shape, and SafePusher benchmarks, as well as on short-horizon Safety Gym tasks, using task performance and episodic cost under prescribed safety budgets. The results show that ITES achieves substantially higher task success and constraint-compliant success rates on the long-horizon benchmarks while maintaining mean episodic costs below the prescribed budgets.