An RL-Inspired Probabilistic Updating Heuristic for Mode Selection in Modular Construction Scheduling

Multi-mode project scheduling, in which each activity can be executed in one of several alternative modes with distinct duration and resource requirements, represents a challenging combinatorial optimization problem prevalent in modular construction and manufacturing. While metaheuristic algorithms excel at exploring the global space of activity sequences, the selection of appropriate execution modes remains a critical sub-problem that demands efficient and effective decision-making. This paper conducts a systematic empirical evaluation of an RL-inspired probabilistic updating heuristic for mode selection in multi-mode resource-constrained project scheduling. Unlike traditional co-evolutionary approaches that optimize sequences and modes jointly, we decouple mode selection from sequence optimization and formulate it as an independent probabilistic learning task. The proposed heuristic maintains a categorical distribution over execution modes for each activity, iteratively updated by reinforcing modes appearing in high-performing schedules, using a normalized reward signal derived from schedule makespan. We implement five representative learning algorithms (REINFORCE, PPO, SAC, DDPG, TD3) as mode selectors and pair them with a Learning-Enhanced Differential Evolution (LEDE) sequence optimizer. Among these, the tabular probability updating (TPU) variant, which does not rely on log-policy gradients or a full MDP formulation, achieves the lowest makespan and the smallest variance, owing to its simple update rule, reward normalization, and a minimum probability threshold that preserves exploration. The learned mode preferences, although conditioned only on the activity index, consistently improve schedule quality when integrated with three different sequence optimizers (Genetic Algorithm, canonical Differential Evolution, and LEDE), reducing makespan by 6–12% compared with a static fastest-mode heuristic. Our results demonstrate that a lightweight, heuristic-driven probability adaptation can serve as an effective, plug-and-play enhancement for mode selection, offering a practical alternative to more complex neural-network-based approaches in resource-constrained modular construction scheduling.