The Horror: Evaluating Information Retrieval Systems for Kids
Ashlee Milton and Maria Soledad Pera. 2020. “The Horror: Evaluating Information Retrieval Systems for Kids”. In Proceedings of the 4th International and Interdisciplinary Perspectives on Children & Recommender and Information Retrieval Systems (KidRec 2020), co-located with ACM IDC 2020.
Abstract
Evaluation of information retrieval systems (IRS) is a prominent topic among information retrieval researchers–mainly directed at a general population. Children require unique IRS and by extension different ways to evaluate these systems, but as a large population that use IRS have largely been ignored on the evaluation front. In this position paper, we explore many perspectives that must be considered when evaluating IRS; we specially discuss problems faced by researchers who work with children IRS, including lack of evaluation frameworks, limitations of data, and lack of user judgment understanding.