Results

If you want to know more detailed explanation, please refer to our paper

1. Overall test RMSE with SD:

Model Dataset ML-1m ML-10m AIV PMF 0.8971 (0.0020) 0.8311 (0.0010) 1.4118 (0.0105) CTR 0.8969 (0.0027) 0.8275 (0.0004) 1.5496 (0.0104) CDL 0.8879 (0.0015) 0.8186 (0.0005) 1.3594 (0.0139) ConvMF 0.8531 (0.0018) 0.7958 (0.0006) 1.1337 (0.0043) ConvMF+ 0.8549 (0.0018) 0.7930 (0.0006) 1.1279 (0.0073) Improve 3.92% 2.79% 16.60%

Above table shows the overall rating prediction errors of five methods on each test set. Note that each dataset is randomly split into a training set (80%), a validation set (10%), and a test set (10%). "Improve" indicates the relative improvements of "ConvMF" over the the best competitor. Compared to three models, ConvMF and ConvMF+ achieve significant improvements on all the datasets.

2. Test RMSE over various sparseness of training data on ML-1m dataset:

This plot shows improvements of ConvMF on three competitors over various spaseness datasets. ConvMF significantly outperforms three competitors over all range over sparseness, and we can see that when the data density increases, the improvements increase. It indicates that CNN of ConvMF is well integrated into PMF for recommendation task to exploit rating information.

3. Impact of Pre-trained Word Embedding Model:

Two plots introduce impacts of pre-trained word embedding model for ConvMF. Left plot shows relative improvements of ConvMF+ over ConvMF on three datasets with various λ v . As data is more extremely skewed (i.e. Amazon Instant Video), an impact of pre-trained word embedding model increases. Note that a high value of λ v leads that ConvMF and ConvMF+ try to exploit description documents of items more than ratings. Right plot shows the effects of the dimension size of word embedding model on Amazon Instant Video dataset. The test error of ConvMF+ is decreased as the dimension size of the pre-trained word embedding model gets higher, because the information contained in the model gets richer.

4. Parameter Analysis:

MovieLens-1m

MovieLens-10m

Amazon Instant Video

Three figures shows impacts of λ u and λ v on three datasets. Specifically, the best performing values of (λ u , λ v ) of ConvMF are (100, 10), (10, 100), and (1, 100) on MovieLens-1m, MovieLens-10m and Amazon Instant Video, respectively. A high value of λ u implies that item latnet model tend to be projeted to the latent space of user latent model (same applies to λ v ). Thus, these best performing values demonstrate that ConvMF well alleviates sparseness of each dataset by balancing the importance of ratings and description documents. Note that a sparse dataset requires high value of λ v .

5. Qualitative Analysis:

Phrase captured by W c 11 max(c11) Phrase captured by W c 86 max(c86) people trust the man 0.0704 betray his trust finally 0.1009 Test phrases for W c 11 max(c test 11) Test phrases for W c 86 max(c test 86) people believe the man 0.0391 betray his believe finally 0.0682 people faith the man 0.0374 betray his faith finally 0.0693 people tomas the man 0.0054 betray his tomas finally 0.0480