Concordance Indices for Risk Scores With Policy Evaluations
Nicholas HartmanABSTRACT
Objective
To demonstrate the differences between concordance index (C‐index) methodologies and clarify the appropriate usage for risk score evaluations in health services applications.
Study Setting and Design
We performed a methodological comparison of C‐index metrics and illustrated the consequences of these differences through a study of liver failure patients.
Data Sources and Analytic Sample
We analyzed secondary adult liver transplant registry data from the Organ Procurement and Transplantation Network (OPTN), including all waitlist registrations from 2002 to 2022.
Principal Findings
The recommended concordance metric based on Gerds' weighting was higher for the original model for end‐stage liver disease (MELD) than Harrell's C‐Index, Uno's C‐Index, and naïve binary outcome metrics (0.864 [95% confidence interval (CI): 0.840, 0.888] versus 0.854 [95% CI: 0.844, 0.864], 0.832 [95% CI: 0.819, 0.844], and 0.727 [95% CI: 0.715, 0.740]), and it did not increase after the latest MELD formula update (0.874 [95% CI: 0.859, 0.889] to 0.869 [95% CI: 0.853, 0.885]).
Conclusions
The concordance indices that are often used in health services applications have important deficiencies under policy‐related dependent censoring, and researchers must apply appropriate weighting schemes to avoid bias. The findings uncover new interpretations of past evaluation results that have shaped national liver transplant policies.