Practical Benchmarking Configurations for Reproducible Execution-Time Measurements of CI/CD-Style Workloads

Early detection of performance regressions during software development enables precise attribution of performance changes to specific code revisions and significantly reduces debugging effort and development costs. Regression detection requires stable measurements, since run-to-run noise can obscure important performance changes. Achieving stable measurements is particularly challenging for short-running benchmarks commonly used in CI/CD pipelines. We propose two benchmarking configurations that guarantee measurement stability. The first is a general-purpose configuration, aimed at highly stable measurements, enabling regression detection and reliable performance comparisons across single-threaded benchmarks and different versions of the same multi-threaded benchmark. The second targets multi-threaded benchmarks requiring cross-benchmark performance comparisons. These configurations control multiple factors, grouped into three classes: hardware optimizations, system settings, and runtime options. We evaluate these configurations, together with additional variants that isolate each individual factor and quantify their impact, for a total of \(29\) configurations. We evaluate across five environments, including idle and busy UMA/NUMA servers and a personal laptop, using benchmarks implemented in managed and unmanaged languages. Experimental results show that the proposed configurations are stable: the general-purpose configuration achieves an average relative median absolute deviation (RMAD) below 0.55% across all workloads and environments, while the multi-threaded configuration achieves an average RMAD below 2%.