Janus: Realizing Practical Operator Parallelism for Latency-Sensitive DNN Inference on GPUs

With the growing deployment of Deep Neural Networks (DNNs) in latency-critical services, optimizing inference efficiency on GPUs has become crucial. While exploiting operator parallelism offers a promising avenue to accelerate inference and improve hardware utilization, existing approaches often overlook two critical factors: hardware resource constraints and latency heterogeneity across operators. This oversight creates a significant discrepancy between the intended schedule and actual runtime behavior, severely degrading inference performance and GPU utilization. To address this, we propose Janus, a novel resource- and latency-aware operator scheduling framework to enable efficient and practical operator parallelism for DNN inference on GPUs. Janus introduces an effective stream allocation mechanism that fully incorporates operator resource constraints and latency heterogeneity. Specifically, it classifies operators into distinct priority levels and leverages the CUDA stream priority mechanism to map them to corresponding streams. By doing so, Janus achieves highly efficient operator parallelism by ensuring that the high-efficiency schedule is faithfully executed at runtime, while simultaneously enabling the hardware scheduler to adaptively scavenge transiently idle resources. We implement a prototype of Janus in PyTorch and conduct comprehensive evaluations using eight representative DNN models on both NVIDIA RTX A5000 and H800 GPUs. Experimental results show that with only a minimal one-time profiling overhead of mere seconds, Janus achieves average inference speedups of 2.13 × and 3.41 × over PyTorch on the RTX A5000 and H800 GPUs, respectively. Compared to Opara, a state-of-the-art operator-level scheduling framework, Janus further delivers average speedups of 1.20 × (RTX A5000) and 1.11 × (H800). Moreover, Janus improves GPU utilization. On the RTX A5000, it achieves 1.16 × the SM occupancy and 1.10 × the SM active rate of Opara on average, which exhibits the highest resource utilization among all baselines.