Prompt‐Based Continual Learning via Task‐Adaptive Knowledge Sharing

ABSTRACT

Prompt‐based continual learning (PCL) has gained significant attention as a rehearsal‐free and parameter‐efficient framework built upon the generalisable knowledge of pre‐trained models. Its core idea is to adapt to new tasks by effectively exploiting pre‐trained knowledge while preserving previously learned information. However, existing PCL methods overlook where task‐relevant pre‐trained knowledge actually resides. They insert prompts into manually selected layers for all tasks, regardless of whether those layers contain the representations required by each task. This uniform prompting limits task‐specific utilisation of the rich and diverse representations within the pre‐trained model. In this work, we propose an adaptive prompting approach that investigates how pre‐trained knowledge should be utilised for each task. To achieve this, we introduce task‐specific layer‐wise learnable gates that identify where prompts are required and insert them into layers that contain task‐relevant pre‐trained knowledge. Such an insertion pattern enables the individual task to adaptively utilise pre‐trained knowledge according to its requirements. Moreover, to reinforce representations across related tasks by transferring their knowledge, we estimate inter‐task relevance using feature‐distance analysis and regulate knowledge sharing across tasks accordingly. Our method differs from existing works by addressing both how pre‐trained knowledge should be utilised for each task and how knowledge should be shared across related tasks. We evaluate our method on three image classification benchmarks in class‐incremental learning and achieve superior performance compared to existing methods.