Impact of Item Consumption on Assessment of Recommendations in User Studies

Loepp, B., Donkers, T., Kleemann, T., & Ziegler, J. (to appear). RecSys ’18: Proceedings of the 12th ACM Conference on Recommender Systems. New York, NY, USA: ACM.

Abstract

Participants of recommender systems user studies typically cannot consume recommended items. Still, they are asked to assess recommendation quality and other aspects related to user experience by means of questionnaires. Without having, for instance, listened to recommended songs or watched suggested movies, this however might be an error-prone task, possibly limiting validity of results obtained in these studies. In this paper, we investigate the impact of consumption of recommended items. We present two user studies conducted in different domains showing that in some cases, differences in the assessment of recommendations and in questionnaire results occur. Apparently, it is not always possible to adequately measure user experience without allowing to consume items before filling in a questionnaire. On the other hand, depending on domain and provided information, participants sometimes seem to approximate the value of recommendations reasonably well.

Resources

Related publications

Explaining Recommendations by Means of User Reviews

Sequential User-based Recurrent Neural Network Recommendations

VIEW MORE »