Combined Metric: Coefficient of Determination Meets Quality of Uncertainty

Coefficient of determination is widely used to select machine learning hyperparameters that yield the highest quality of predictions. However, coefficient of determination does not consider uncertainty estimates in predictions, which are also crucial for practical applications. In this work, we develop a combined metric that incorporates both quality of predictions and quality of uncertainty. We validate the metric on a controlled synthetic regression benchmark. We demonstrate that optimization of the combined metric improves the overall estimates of the output variable and its uncertainty compared to optimization of either ingredient metric within this benchmark. The combined metric enables simultaneous assessment of predictions and their uncertainties for regression model selection; extension to broader model classes and tasks is left for future work.