On the Performance Evaluation of Anytime Algorithms in Noisy Optimization

A standardized, elaborate and well-argued performance metric for benchmarking anytime algorithms on noisy optimization problems is missing in the current benchmarking frameworks. This research aims at developing such a metric as an artifact by following the Design Science Research methodology. Existing noise handling practices and benchmarking guidelines are reviewed. Benchmarking experiments, including a test suite containing 13 artificial test functions, were conducted with two state-of-the-art algorithms to assess the performance metric. Based on the experimental results and the existing literature, this study concludes that, rather than relying on a single quantitative value, algorithm performance should be assessed through a comprehensive performance report comprising the numerical loss value, the number of function evaluations, wall-clock runtime, and a performance profile illustrating the trajectory of loss values over time. This approach aims to maximize the reliability, expressiveness, and informational value of performance assessments.