SkipOPU: An FPGA-based Overlay Processor for Large Language Models with Dynamically Allocated Computation

Large language models (LLMs), particularly decoder-only models with auto-regressive decoding, have achieved remarkable performance across a wide range of tasks, but their inference efficiency remains a critical bottleneck due to rapidly growing parameters and memory-bound generation phases. Recent advances in dynamic computation allocation address this challenge by exploiting the highly uneven contributions of different tokens and layers, enabling selective execution that significantly reduces redundant computation while preserving model accuracy. However, existing hardware platforms and accelerators are primarily optimized for uniform, static execution, limiting their ability to efficiently support such dynamic inference patterns. In this work, we propose SkipOPU , an FPGA-based overlay processor that dynamically allocates computation across tokens and layers with high flexibility through a lightweight routing mechanism. First, we decouple reduction operations from element-wise computation in nonlinear modules and perform reductions incrementally. This design enables both stages to be fused with adjacent linear operations (router or matrix multiplication), effectively hiding nonlinear latency within the pipeline. Second, motivated by asymmetric sensitivity to numerical precision between activation and weight, we design a processing element (PE) array that efficiently supports float–fixed hybrid execution. A novel DSP overpacking technique is introduced to maximize hardware utilization while minimizing resource overhead. Finally, we develop a proactive on-chip KV (Key-Value) history buffer that exploits cross-layer KV invariance of pruned tokens, eliminating irregular HBM (High Bandwidth Memory) accesses during decoding and supplementing off-chip bandwidth through high-locality on-chip reuse. Experimental results demonstrate that SkipOPU on an AMD U280 FPGA outperforms GPU and other FPGA-based accelerators by 1.31 × −2.55× in bandwidth efficiency for LLMs inference with dynamic computation allocation. Moreover, cross-layer KV reuse reduces up to 25.4% KV storage overhead across varying sequence lengths.