Comparing Fair Ranking Metrics
Amifa Raj, Connor Wood, Ananda Montoly and Michael D. Ekstrand. 2020. “Comparing Fair Ranking Metrics”. To appear in Proceedings of the 3rd FAccTRec Workshop on Responsible Recommendation at 14th ACM Conference on Recommender Systems (RecSys 20).
Abstract
Ranking is a fundamental aspect of recommender systems. However, ranked outputs can be susceptible to various biases; some of these may cause disadvantages to members of protected groups. Several metrics have been proposed to quantify the (un)fairness of rankings, but there has not been to date any direct comparison of these metrics. This complicates deciding what fairness metrics are applicable for specific scenarios, and assessing the extent to which metrics agree or disagree. In this paper, we describe several fair ranking metrics in a common notation, enabling direct comparison of their approaches and assumptions, and empirically compare them on the same experimental setup and data set. Our work provides a direct comparative analysis identifying similarities and differences of fair ranking metrics selected for our work.