Selective Residual Action Chunking for Vision-Guided Multi-Task Robotic Manipulation

Vision-guided robotic manipulation requires control actions that reflect visual observations, robot state, and task goals. Action Chunking with Transformers (ACT) performs well on individual tasks, but shared multi-task policies must accommodate different action distributions and state-dependent control requirements. We propose an ACT-based selective residual action-chunking policy using global and wrist RGB images together with proprioceptive feedback. A task-conditioned ACT model is first trained on multi-task demonstrations. Its parameters are then frozen while bounded residual experts learn candidate action corrections. A gain predictor uses observation-conditioned decoder features, robot state, task information, and base-action statistics to select a correction or retain the base prediction. Gain supervision is derived from reductions in imitation error. The robot executes an action prefix and updates its prediction from new observations, incorporating selective correction into closed-loop manipulation. Across all ten LIBERO-Goal tasks, the method achieves a mean success rate of 91.2%, exceeding MTACT and BAKU by 11.4 and 4.2 percentage points. On three jointly learned real-world tasks, it achieves 78.7%, exceeding MTACT by 10.0 percentage points. These results support selective residual action generation for vision-guided multi-task robot control.