A Systems-Level Performance Study of Post-Quantum TLS with ML-KEM and ML-DSA in Modern OpenSSL

Large-scale quantum computers threaten the classical public-key foundations of Transport Layer Security (TLS), motivating deployment of standardized post-quantum cryptography (PQC). This study evaluates post-quantum TLS 1.3 as a systems-level deployment problem in OpenSSL 3.4.1 with the Open Quantum Safe provider and in OpenSSL 3.5.0 with native PQC support. We combine cryptographic microbenchmarks, TLS 1.3 connection timing, end-to-end HTTP time to first byte (TTFB), controlled network impairment, packetization analysis, and concurrency tests on two x86-64 systems, and release the measurement archive with the analysis scripts. For hybrid X25519MLKEM768, the median TTFB overhead relative to classical TLS was 6.5–7.5% with OpenSSL 3.4.1 and 1.2–1.3% with OpenSSL 3.5.0. These cross-release comparisons describe complete software configurations. In authentication-matched runs, replacing RSA-PSS with ML-DSA-65 changed median TTFB by −3.0 to +0.1 ms. The full-PQC configuration enlarged the server handshake flight from 1.3 kB to 10.1 kB, mainly through ML-DSA-65 authentication, and increased server-to-client payload segments from 4–5 to 11–13. At a round-trip time of 50 ms, 1% packet loss increased p99 TTFB 6–8-fold in all configurations. Most of this tail consisted of requests delayed by about 1 s, whose frequency matched the probability of losing a TCP connection-establishment segment and was similar for all configurations. Under concurrent load, OpenSSL 3.5.0 completed 46–58% more requests per second.