A Study of Normative Diversity Metrics in News Recommendations

Recent work on beyond-accuracy metrics for news recommendation has proposed RADio, a framework that operationalizes the concept of normative diversity from the social sciences into rank-aware evaluation metrics. RADio measures normative diversity as the divergence between ranked distributions over normative attributes of news articles. Vrijenhoek et al. caution that RADio’s conclusions may depend strongly on the dataset, cultural context, and the representation pipeline used to define such normative attributes. In this reproducibility study, we empirically study how robust RADio is to such contextual and representation choices.

We evaluate the robustness of RADio along three axes. First, we extend RADio’s original evaluation from the English-language MIND dataset to the Danish EBNeRD and Norwegian Adressa news datasets, to test whether its metric behavior generalizes across editorial and cultural contexts. Second, we systematically vary key components of the attribute representation pipeline to examine how specific implementation choices influence RADio’s metric values. Finally, we analyze how RADio relates to recommendation accuracy by applying simple post-hoc re-ranking and quantifying the trade-offs between accuracy and normative diversity.

Our results show that RADio outcomes are sensitive to both editorial context and representation design. Across datasets and representation choices, we observe that RADio can rank models’ normative diversity differently, and in some settings, the metric either saturates or collapses, limiting its ability to discriminate between models. We analyse why this happens, and give directions on designing the attribute representation pipeline. Furthermore, across all settings we consider, we can substantially increase RADio scores via re-ranking while incurring only modest losses in standard recommendation accuracy.

These findings provide empirical support and nuance for RADio’s originally hypothesized contextual dependence, and underscore the need for context-aware evaluation frameworks when operationalizing normative diversity in news recommendations. In particular, the strong dependence on the attribute representation pipeline suggests that normative evaluation should be designed in close collaboration with editorial stakeholders. At the same time, our experiments indicate that, given appropriate representation choices, simple re-ranking can make normative-diversity-aware news recommendations practically viable. We share our code to facilitate reproducibility of our findings: https://github.com/davidvos/normative-diversity-repro /