Fair assessment of fair recommender systems: an experimental framework on model and data characteristics

Recommender systems deliver personalized content to users, but often introduce biases that affect fairness. Despite growing interest, lack of a standardized set of fairness metrics and dataset-specific biases make fair assessment among recommender systems very challenging, as well as the reported findings unreliable. This study proposes an evaluation framework to address these challenges by analyzing 900 experimental runs across two widely used datasets (MovieLens-1M, BookCrossing), three state-of-the-art recommenders, five fairness metrics, and two sensitive attributes. Through regression analyses, statistical tests and effect size analysis, we show that data characteristics and sensitive attribute distributions significantly influence fairness outcomes, while the significance and magnitude of these effects vary depending on the chosen fairness metric. We also uncover significant interaction effects between model and data properties, offering deeper insights into fair model dynamics. Notably, some fairness metrics conflict with accuracy, highlighting trade-offs in fairness-aware optimization. Our framework offers a systematic approach to evaluating fairness in recommender systems, supporting more transparent and comparable research.