We solicit position and research papers (4 pages excluding references, ACM SIGCONF template) that address challenges in the multi-method evaluation of recommender systems and other personalized systems. This includes
- “lessons learned” from the successful application of multi-method evaluations,
- “post mortem” analyses describing specific evaluation strategies that failed to uncover decisive elements,
- “overview papers” analyzing patterns of challenges or obstacles to multi-method evaluation, and
- “solution papers” presenting solutions towards identified challenges.
We support and encourage submissions addressing diverse evaluation scenarios from various RecSys application domains (e.g., real-time music-recommendation for jogging, emotion-aware microblog discovery, trust in social recommendations and cultural differences, healthy-food recommendation for children education, lifestyle recommendation based on natural language interaction).
Possible questions addressed may include (but are not limited to):
- How can we select evaluation methods that allow to identify blind spots in user experience? What may be criteria to compare and evaluate the suitability of methods for given evaluation objectives and how can we develop those?
- How can we integrate and combine the results of multiple methods to get a comprehensive picture of user experience?
- What are the challenges and limitations of single- or multi-method evaluation of RecSys? How can we overcome such hurdles?
- What are viable user-centric multi-method study designs (guidelines) for evaluating RecSys? What are the lessons learned from successful or unsuccessful user-centric multi-method study designs?
Submissions will undergo single-blind peer-review by two reviewers and a meta-review by a program committee member and will be selected based on quality, novelty, clarity, and relevance; aiming to foster discussions. Accepted papers will be invited to present their work during the workshop and will be included in the UMAP ‘18 adjunct proceedings by ACM. For each of the accepted submissions, at least one author must attend the workshop and register for at least one day of the UMAP 2018 conference.
Authors are invited to submit their papers via the EasyChair submission system.