Addressing Power of Equivalence Tests for Statistical Fairness Assessments of Machine Learning Models in Healthcare

Background/Objectives: Machine learning and artificial intelligence (AI) have substantial potential to change the practice of oncology through individualized patient care; however, it is important to ensure that any benefits are realized equitably. There are several examples of AI tools exhibiting bias towards traditionally under-resourced or marginalized groups. Proper validation that these models perform well across populations requires rigorous statistical design and interpretation. One component that is currently lacking in fairness assessments is a clear, prespecified definition of the evidence required to conclude that an AI tool is fair. Methods: To address this issue, we embed the notion of fairness assessment of an AI model within the statistical equivalence testing framework. This approach provides an a priori definition of fairness and an explicit decision rule for concluding, with a high degree of confidence, that model performance falls within a prespecified margin. The framework also supplies power and sample size calculations to support the design phase of a fairness assessment, which we develop for the case of modeling a binary outcome and a binary classification. As a demonstration, we use a “four-fifths” rule of thumb to set equivalence margins but emphasize the necessity of clinical and ethical context for choosing the margins in practice. Results: Using two published oncology models, we show that a realistic sample size can be adequate for some performance metrics and inadequate for others. We also address power and sample size considerations for asymmetric association measures and impossibility results for fairness. Conclusions: Assessments of AI performance fairness should be designed with intention such that target metrics and equivalence margins are specified in advance with ample sample size to conclude fairness.