Q performance 2: Toward Unbiased Selection of Machine-Learning Regression Models

Abstract

Selecting a single best machine-learning regression model from a set of competing models can be challenging. While models selected based on cross-validation performance do not guarantee good predictions on external data, models selected solely on external validation performance do not ascertain precise predictions for other external sets. Therefore, we propose three quantitative metrics to guide modelers in selecting the best model using the modeling set information only. Three quantitative data sets of varying sizes and complexities were considered. Each data set was randomly split into a modeling set and an independent test set. The modeling set was further split thrice to generate training and validation sets. Various machine-learning models were developed and validated against the validation sets. Our proposed metrics were computed for each model using only training and validation set performances. The novel metric values guided the selection of the best models, which also demonstrated expected performance on the independent test set. Furthermore, successful applications of our framework on two additional benchmark data sets demonstrated its wider generalizability.