PILOT: A Replay-Free Continual Learning Approach for Real-Time Semantic Segmentation via Boundary Guidance

Real-time semantic segmentation models offer an excellent balance between accuracy and inference speed. However, deploying these models in dynamic real-world environments often requires the ability to learn novel classes incrementally without retraining on the entire dataset. This capability is known as continual learning. In this regard, standard fine-tuning methods often suffer from catastrophic forgetting, where the model learns new information but loses accuracy on previously learned classes. The severity of this effect depends on the incremental setup, the available data, and the fine-tuning strategy. Contributing to this crucial domain, this paper proposes a novel continual learning framework tailored for PIDNet, which is a widely cited state-of-the-art real-time semantic segmentation model. Our method, PILOT (Parallel Incremental Learning Over Time), introduces a real-time and lightweight strategy by implementing a parallel Derivative branch (D-branch) designed to capture the high-frequency boundary information of novel classes while freezing the trained parameters of the original segmentation network. This novel setup allows the model to adapt to new semantic categories while preserving the knowledge of previously learned classes. By using only data associated with the new class, our model significantly reduces training overhead. Experimental results demonstrate that our approach successfully segments new classes while maintaining a high mean Intersection over Union (mIoU) on the original base classes, thereby outperforming prior continual learning approaches in this real-time segmentation setting. Overall, PILOT is shown to effectively mitigate catastrophic forgetting with minimal impact on inference latency, adding fewer than 5% additional parameters and reducing the frame rate by only about 9%, thus maintaining real-time performance.