Statistical Views on F beta and other Performance Metrics for Data Mining

We give analytic formulas of asymptotic confidence intervals (CIs) for \(F_{\beta}\) metric with \(\beta\) not necessarily \(1\) . They are useful both for fast computation and also for sample size planning, in order to have tight enough CIs cover the true parameter enough often. We also study joint confidence intervals to be able to make multiple valid statements simultaneously and after looking at the CI results, with a high enough probability asymptotically for large data. The approach can be generalized in several ways and can be used for other performance metrics in data mining, including, e.g., classification accuracy, precision, recall, F metrics, and Jaccard index. We also consider a variance correction method for improving finite sample performance, the Bonferroni bound for faster computation, some variants of bootstraps, and extension to multiclass problems.