Impact of Item Consumption on Assessment of Recommendations in User Studies

Loepp, B., Donkers, T., Kleemann, T., & Ziegler, J. (2018). In Proceedings of the 12th ACM Conference on Recommender Systems (pp. 49–53). New York, NY: ACM.

Best short paper award

Abstract

In user studies of recommender systems, participants typically cannot consume the recommended items. Still, they are asked to assess recommendation quality and other aspects related to user experience by means of questionnaires. Without having listened to recommended songs or watched suggested movies, however, this might be an error-prone task, possibly limiting validity of results obtained in these studies. In this paper, we investigate the effect of actually consuming the recommended items. We present two user studies conducted in different domains showing that in some cases, differences in the assessment of recommendations and in questionnaire results occur. Apparently, it is not always possible to adequately measure user experience without allowing users to consume items. On the other hand, depending on domain and provided information, participants sometimes seem to approximate the actual value of recommendations reasonably well.

Resources

Related publications

Explaining Recommendations by Means of User Reviews

Sequential User-based Recurrent Neural Network Recommendations

VIEW MORE »