S ² MM: S calable FPGA Acceleration of S ecure M atrix

Homomorphic Encryption (HE) enables secure computation on encrypted data, addressing privacy concerns in cloud computing. However, the high computational cost of HE operations, particularly matrix multiplication (MM), remains a major barrier to its practical deployment. Accelerating Homomorphic Encrypted MM (HE MM) is crucial for applications such as privacy-preserving machine learning.

In this paper, we present S ² MM, the first scalable FPGA-based accelerator designed for HE MM. We show that HE MM is fundamentally bottlenecked by excessive off-chip memory traffic when operating with practical HE parameter sets. To analyze this bottleneck, we develop a cost model that characterizes the on-chip memory requirements of ciphertexts during HE MM, revealing the scalability limits of CPU-based implementations. Guided by the analysis, we propose a novel datapath for Homomorphic Linear Transformation (HLT), the dominant workload in HE MM. The proposed datapath enables fine-grained on-chip data reuse, substantially reducing both off-chip memory traffic and on-chip memory demand. To further improve scalability, S ² MM supports tiled HE MM, removing the matrix size limitation imposed by the HE encoding capacity and enabling efficient computation on large matrices. In addition, the architecture is highly configurable, adapting to diverse matrix sizes and HE parameter sets while maximizing throughput within FPGA resource constraints. We also introduce a systematic configuration methodology to determine optimal design parameters for efficient deployment across varying workloads and platforms. We implement S ² MM on an Alveo U280 and conduct comprehensive evaluations across a wide range of matrix sizes, shapes, and HE parameters. Experimental results demonstrate that S ² MM achieves average speedups of 219× and 37.4× over state-of-the-art CPU and GPU implementations, respectively. We further evaluate S ² MM in an application-level encrypted CNN inference pipeline, achieving a 6.08× end-to-end speedup by accelerating the HE MM-dominated fully connected layers.