Towards the end of the year, due to work tasks, I surveyed several papers on Recommender Systems (hereinafter referred to as RS). When I was writing my master’s thesis, Deep Learning (hereinafter referred to as DL) was not yet popular. In the blink of an eye, DL has become a mainstream discipline, and recent recommendation algorithms are all based on DL research. I have also applied it in industry projects and achieved some progress, but I am quite curious about the general effectiveness of applying DL to RS.
While I still remember the content of my survey, I would like to share some insights based on the ACM RecSys 2019 Best Paper, “Are We Really Making Much Progress? A Worrying Analysis of Recent Neural Recommendation Approaches”:
Summary after reading:
- DL has been quite successful in the fields of Computer Vision (CV) and Natural Language Processing (NLP), but less so in other areas than expected.
- From 2015 to 2018, there were 18 DL papers related to top-n recommendations in the top conferences RecSys, WWW, KDD, and SIGIR. Of these, 7 experiments were reproducible, accounting for approximately 39%; meaning 61% of the experiments could not be reproduced.
- The methods typically used as baselines include non-personalized TopPopular, Collaborative Filtering (CF) methods such as ItemKNN, UserKNN, P3alpha, and RP3beta, the hybrid ItemKNN CF + CB, and the Machine Learning (ML) method SLIM.
- The paper authors reproduced these 7 experiments to verify whether the reported progress was real. They found that DL methods did not perform better than the aforementioned classic heuristic methods. Among the 7 papers, only 1 outperformed the aforementioned baseline methods, and even then, only in certain cases.
- The paper authors believe the reasons for this result are that these 7 papers selected poor baseline methods and parameters, used different numbers of epochs for the selected test data, and in some cases, different experimental procedures led to evaluation errors.
Personal insights:
- To verify the progress over the past few years, the paper authors used public datasets as one of the reproducible conditions, selecting 7 papers out of 18 to reproduce their experiments. However, current public datasets have characteristics such as insufficient data volume, less sparse matrices, and fewer features. Such datasets are not the strong suit of DL.
- Most public datasets originate from academia, and their size and feature richness are far lower than those of industry systems. Therefore, the conclusions of the paper authors, as well as the performance of DL on commercial datasets, still require further verification.
References:
https://dl.acm.org/authorize?N684126
https://arxiv.org/pdf/1907.06902.pdf
