Measuring the Impact of Recommender Systems – A Position Paper on Item Consumption in User Studies
Loepp, B., & Ziegler, J. (2019). Proceedings of the 1st Workshop on Impact of Recommender Systems (ImpactRS ’19).
Abstract
While participants of recommender systems user studies usually cannot experience recommended items, it is common practice that researchers ask them to fill in questionnaires regarding the quality of systems and recommendations. While this has been shown to work well under certain circumstances, it sometimes seems not possible to assess user experience without enabling users to consume items, raising the question of whether the impact of recommender systems has always been measured adequately in past user studies. In this position paper, we aim at exploring this question by means of a literature review and at identifying aspects that need to be further investigated in terms of their influence on assessments in users studies, for instance, the difference between consumption of products or only of related information as well as the effect of domain, domain knowledge and other possibly confounding factors.
