Impact of Item Consumption on Assessment of Recommendations in User Studies

Loepp, B., Donkers, T., Kleemann, T., & Ziegler, J. (2018). Proceedings of the 12th ACM Conference on Recommender Systems (RecSys ’18), 49–53.

Best short paper award

Abstract

In user studies of recommender systems, participants typically cannot consume the recommended items. Still, they are asked to assess recommendation quality and other aspects related to user experience by means of questionnaires. Without having listened to recommended songs or watched suggested movies, however, this might be an error-prone task, possibly limiting validity of results obtained in these studies. In this paper, we investigate the effect of actually consuming the recommended items. We present two user studies conducted in different domains showing that in some cases, differences in the assessment of recommendations and in questionnaire results occur. Apparently, it is not always possible to adequately measure user experience without allowing users to consume items. On the other hand, depending on domain and provided information, participants sometimes seem to approximate the actual value of recommendations reasonably well.

Additional information

Resources

Related publications

Impact of Consuming Suggested Items on the Assessment of Recommendations in User Studies on Recommender Systems

More »