Grade Inflation in Generative Models

Generative models hold great potential but only if one can trust the evaluation of the data they generate. It is common to visualize and evaluate quality on two-dimensional visualizations of high-dimensional data. We show that many commonly used quality scores for comparing two-dimensional distributions of synthetic vs. ground-truth data give better results than they should, a phenomenon we call the “grade inflation problem”. We show that the correlation score, Jaccard score, earth-mover’s score, Kullback–Leibler (relative-entropy) score, Jensen–Shannon Score, and Cauchy–Schwarz score all suffer grade inflation to various extents and in various contexts. We suggest that any score that values all datapoints equally, as these do, will also exhibit grade inflation; we refer to such scores as “equipoint” scores. We introduce the concept of “equidensity” scores and present the Eden score as what is, to our knowledge, the first example of such a score. We find that Eden avoids grade inflation and show the highest consistency with human perception of goodness-of-fit of KDEs compared to the equipoint scores above. We propose that any reasonable equidensity score will avoid grade inflation. We identify a connection between equidensity scores and Rényi entropy of negative order. We conclude that equidensity scores are likely to outperform equipoint scores for generative models and for comparing low-dimensional distributions more generally.