Learning Collaborative Filtering and Its Application to People to ...

Learning Collaborative Filtering and Its Application to People to People Recommendation in Social Networks Xiongcai Cai, Michael Bain, Alfred Krzywicki, Wayne Wobcke, Yang Sok Kim, Paul Compton and Ashesh Mahidadia School of Computer Science and Engineering, University of New South Wales, Sydney NSW 2052, Australia {xcai,mike,alfredk,wobcke,yskim,compton,ashesh}@cse.unsw.edu.au Abstract—Predicting people who other people may like has recently become an important task in many online social networks. Traditional collaborative filtering (CF) approaches are popular in recommender systems to effectively predict user preferences for items. One major problem in CF is computing similarity between users or items. Traditional CF methods often use heuristic methods to combine the ratings given to an item by similar users, which may not reflect the characteristics of the active user and can give unsatisfactory performance. In contrast to heuristic approaches we have developed CollabNet, a novel algorithm that uses gradient descent to learn the relative contributions of similar users or items to the ranking of recommendations produced by a recommender system, using weights to represent the contributions of similar users for each active user. We have applied CollabNet to the challenging problem of people to people recommendation in social networks, where people have a dual role as both “users” and “items”, e.g., both initiating and receiving communications, to recommend other users to a given user, based on user similarity in terms of both taste (whom they like) and attractiveness (who likes them). Evaluation of CollabNet recommendations on datasets from a commercial online social network shows improved performance over standard CF. Keywords-Data Mining; Machine Learning; Recommender Systems; Collaborative Filtering

I. I NTRODUCTION Traditional recommender systems attempt to discover user preferences over items by modelling the relation between users and items. The aim is to recommend items that match the taste (likes or dislikes) of users in order to assist the active user, i.e., the user who will receive recommendations, to select items from an overwhelming set of choices. Such systems have many uses in e-commerce, subscription based services and other online applications, where provision of personalised suggestions is required. By applying recommendation techniques, it is possible to greatly increase the likelihood of the successful purchase of products or services, since services or products are personalised and presented to the active user using information obtained from the purchasing behaviour of like-minded users. In online applications with a very large number of choices where customer taste is important, personalised recommendation of items or people becomes essential. Approaches to recommender systems can be categorised

as content-based or collaborative filtering (CF) methods. In content-based methods [1]–[4], the user will be recommended items similar to those the user preferred in the past. Content-based methods analyse the descriptions of items and provided user ratings to infer a model for recommending additional items of interest. CF methods recommend items based on aggregated user preferences of those items, independent of the availability of item descriptions. CF algorithms fall into two categories: memory-based approaches and model-based approaches. Memory-based approaches [5]–[8] use heuristics to make rating predictions based on the entire collection of items previously rated by users. The unknown rating value rc,s of the active user c for an item s is typically computed as an aggregate of the ratings of users similar to c for the same item s. This aggregate can be an average or, commonly, a weighted sum, where the weight is a distance that measures the similarity sim(c1 , c2 ) between users c1 and c2 . By using similarity as a weight, more similar users make a greater contribution to a predicted rating. Typically the similarity between two users is based on their ratings of items that both users have rated. Approaches to compute similarity include correlation [6], [8], cosine-based [5], [9], default voting, inverse user frequency, case amplification and weightedmajority prediction [5], [10]. Those approaches usually use heuristics to model the weights. However, a major problem with heuristically-based weighted sum approaches is that they do not take into account the fact that the similarity of users to one active user may be more reliable than their similarity to another. Taking this seriously means the contribution of similar users to the final rating used in recommendation should be adjusted for different active users. In this paper we address this problem by learning how to weight the contribution of similar users for each active user. Model-based CF [5], [11]–[16] uses the collection of ratings to learn a model, which is then used to make rating predictions. Although model-based methods have reported higher accuracy of recommendation than memory-based approaches, there are also some limitations. Firstly, these methods are computationally expensive since they usually require all users and items to be used in creating models.

Secondly, they attempt to predict the rating of a user rather than correctly rank the items. This is not sufficient, however, since any method that correctly predicts all the ratings will correctly rank all the items, but two methods that predict the ratings equally well may perform differently in predicting the rankings [11]. Instead of using ratings, we directly learn a ranking using a pairwise learning to rank algorithm. Memory-based approaches can be more computationally efficient compared to model-based approaches due to the reduced number of correlated users to be considered when making recommendations. However, they may not model well the underlying contribution since they are based on heuristics to compute weight values using pre-defined similarity measures. In this paper, we introduce CollabNet, a novel recommendation method combining the advantages of memory-based and model-based approaches with learning to rank techniques. The method learns the weight value of similarly-minded users directly from the training data to build a regression model to predict the preferences of the active user. More specifically, the method uses gradient descent methods to learn the underlying social collaboration behaviours relating to ranking recommendations. Thus it has the ability to adjust the model learned to reflect the characteristics of the data. Furthermore, model construction is based on optimising to correctly rank items rather than predict their rating, which makes the algorithm more effective. The main contributions of this paper include: (i) a novel method of learning CF by combining memory-based and model-based approaches based on learning to rank techniques; and (ii) the application of the proposed method to the recommendation of potential friends or partners in social networks. The paper is organised as follows. Section II describes our CF framework. Section III develops a machine learning approach for learning CF. Section IV investigates our learning CF approach applied to people to people recommendation in social networks. Experimental evaluation is in Section V and conclusions are in Section VI. II. S TANDARD C OLLABORATIVE F ILTERING Traditional CF can operate in two directions: user-based or item-based. User-based approaches look for users who share the same rating patterns with the active user (the user whom the prediction is for) and then use the ratings from like-minded users to calculate a prediction for the active user. Item-based methods such as that of Amazon.com [7] use an item-item matrix determining the similarity of items in terms of pairs of items purchased together, which is then used to infer the preference of the active user. The most important step in both approaches is determining similarity. Two items are similar if both are selected together by a set of users. Alternatively, two users are similar if they both select the same set of items (i.e., they have similar taste). The underlying assumption of CF approaches is that those who agreed in the past tend to agree again

in the future. User-based approaches assume that two users will like the same items if they are similar in taste, therefore will recommend to the active user an item selected by a similar user: i ⇒ u : ∃s, (s ↔ u ∧ s → i), where i ⇒ u denotes recommending i to u, s ↔ u denotes that s is similar to u and s → i represents that s selected i. Itembased approaches assume items can be related by the fact that they are frequently selected together by users, and will recommend an item which is similar to items that the active user selected: i ⇒ u : ∃s, (s ↔ i ∧ u → s). III. L EARNING C OLLABORATIVE F ILTERING A. Modelling similar user contributions In the following sections, we discuss CF approaches in the context of people to people recommendation, unless specified otherwise. This means that instead of items we are recommending other users to the active user. Although this places extra requirements on the CF framework (described in Section IV), the learning method is sufficiently general to apply to the more standard item recommendation. To see the application to item recommendation, replace in the text occurrences of similar user with similar item, and recommended user with recommended item. In CF, the rating is calculated based on the rating from similar users, i.e., the value of an unknown rating rc,s for user c and item s is usually computed as an aggregate of the ratings of similar users for the same item s in a userbased CF. In the simplest case, the aggregate can be a simple average. However, the most common aggregation approach is to use a weighted sum based on similarity defined as a distance measure. Since various memory-based algorithms differ in how they define the weights, weight determination becomes a core problem in memory-based CF. Existing work uses simple heuristics to calculate similarities which will be then used as the weights. For example, the correlation approach [6], [8] induces a global model of similarities between users, rather than separate models for classes of ratings. In this paper, we propose to learn a model for the weights rather than use heuristic similarity. More specifically, the performance of CF relies on how reliable the rating of the similar users is. We define the weight as the reliability of similar users. Thus, the reliability is the contribution of a similar user towards the true rating of the active user in the data rather than the similarity calculated from the rating. We use similarity as a criterion to select a set of users (items in item-based CF), whose rating will be considered in calculating the rating. Furthermore, how those selected users (items in item-based CF) contribute to the rating is learned from the dataset. So the same pair of similar users can have different contribution for different active users, i.e., the contribution to recommendation of each similar user is personalised for each active user.

Definition For an active user ua , the predicted rating is oa,r ≡ f (!x) =

!

(1)

(ωk xk )

k∈S

where ωk is the weight of the kth similar user, xk the rating of that similar user and S is the set of similar users. B. CollabNet: Learning the Contributions 1) Pairwise Training: To learn the contribution, we employ a pairwise approach, i.e., we take pairs of recommended users as instances in learning, and formalise the problem of learning as that of classification. Specifically, in learning, we collect recommended user pairs from the recommendation lists, and for each pair we assign a label representing the order of preference of the two recommended users (items). We then define a loss function and tune the weight values to minimise the loss function. Without loss of generality, we generate pairs containing as the first element a successful recommended user or item, i.e., a recommended user (item) that has a successful interaction with (selection by) the active user, and the second an unsuccessful recommended one. We always assign value 1 to the label to indicate that the first element in the pair is preferred to the second one. Alternatively, to generate a training pair with the first element an unsuccessful recommended user or item and second a successful one, the label assigned will be 0 instead. 2) Finding Interaction Pairs: The first step towards learning is to find recommended users with ground truth, i.e., with interactions with the active user. Specifically, we find a set of users by CF who have replied positively or negatively to a communication from the active user. An example is shown in Table I, where ua is the active user and u1 to u5 are users who have had interactions with ua . Table II I TEM -BASED CF S IMILARITY

Table I U SER I NTERACTIONS FOR ua

ua

u1 +

u2 +

u3 +

u4 -

u5 -

u1 u2 u3 u4 u5

u1 1 1 1 1

u2 1 1 1 1

u3 1 1 1 1

u6 1 1 1

u7 1 1

The training instances with respect to ua can then be formed by pairing a positive user with a negative user in the user interaction table. In the case of Table I, we have pairs (u1 , u4 ), (u1 , u5 ), (u2 , u4 ), (u2 , u5 ), (u3 , u4 ) and (u3 , u5 ), all with label value 1, as training instances. 3) Establishing the Similarity Graph: Since the aim of learning is to find the proper weight values ω in (1), we need to find the similar users to each of the users in the pair of a training instance. For the example in Table I, a similarity graph can be represented by Table II for item-based CF

(where an item is a user in people to people recommendation in social networks). In Table II, u1,2,3 are users who have had positive interactions and u4,5 negative interactions with ua according to Table I. Similar users u1,2,3 are also users in training pairs for ua while users u6,7 are other similar users not in a training pair for ua . Similar users are collected based on the similarity definition in Section II. In Table II, a “1” in a cell indicates the users represented by the row and the column of that cell are similar, while a “-” indicates not similar. We have a set of similar users {u2 , u3 , u6 } for recommended user u1 , for which we need to learn the weight values. 4) Learning the Contributions: Our approach is inspired by that of [17]. Given a set of pairs of samples xi , xj in Rd with a reference indicating that sample xi is to be ranked higher than sample xj , we aim to learn a model that predicts the rating that the active user would assign to another user by adjustment of the weight ω in (1). As shown in [17], this is a general formulation where the pairs of ranks need not be complete, i.e., they need not specify a complete ranking of the training data, or even be consistent. As in (1), f : Rd → R indicates that the rating of an active user to a test user is specified by the real values that f takes. If the active user prefers user ui to uj denoted by xi # xj , then there exists f (xi ) > f (xj ). In order to learn the weight ω, a cost function must be defined. Following the approach in [17], we define the probability that the active user prefers ui to uj as Pij (xi #xj ) and the probability of the preference in ground truth as P¯ij (xi # xj ), which can then be modelled by a logistic eoij function: Pij ≡ 1+e oij , where oij = f (xi ) − f (xj ). Once we have these two probabilities, the cost function is defined as the difference between those two probabilities, represented by cross entropy: Cij ≡ C(oij ) = −P¯ij logPij − (1 − P¯ij )log(1 − Pij )

(2)

Having the cost function defined, the rest is to implement the learning framework. A two-layer neural net trained by back-propagation was used in [17]. However, in CF the large number of users makes the dimensionality d too large for this to be practical. We use stochastic gradient descent to find the optimal weight values for the rating prediction. Thus, ∂C we differentiate (2) with respect to the weights: ∂ωij = ∂Cij ∂oij ¯ ∂oij . ∂ω = (Pij − Pij ). Therefore, the weight values are changed in a direction that will reduce the error: ∂oij ∂Cij = −η(Pij − P¯ij ). (3) ∂ω ∂ω where η is the positive learning rate that adjusts the relative size of the change in weights. The learning algorithm iteratively changes the weight values according to (3) to reduce the error for each training pair as in Algorithm 1. ∆ω = −η

Algorithm 1 CollabNet: Learning Collaborative Filtering Initialise n, ω, θ, η, r ← 0; repeat r ← r + 1; m ← 0; ∆ωk ← 0; repeat m←" m + 1; " oi ← k∈S (ωk xik ); oj ← k∈S (ωk xjk ); eoij e ¯ oij ← oi − oj ; Pij ← 1+e oij ; Pij ← 1+e ; ∂o ∆ωk ← ∆ωk + η(−(Pij − P¯ij ). ∂ωij ); until m = n ωk ← ωk + ∆ωk ; until )∇C < θ) return ω;

IV. C OLLAB N ET FOR S OCIAL N ETWORKS As described in Section II, the assumptions for both itembased and user-based CF are only valid for recommending items to users where the selection is determined only by the user, not the item. In social networks, “items” as the recipients of actions are also users who are actively participating in social interactions. When they receive a communication from another user, such as an introduction, they can make different responses, either positive or negative. Therefore, traditional CF is not applicable to people recommendation since it only considers the taste of one side (users) and neglects the other (items) [18]. This requires the recommender framework to be extended in order to be able to handle the bilateral nature of such interactions in people recommendation. It is important to stress that simply treating users as items and applying standard CF does not work as well as the approach we now outline (see results in Section V). In people recommendation, users have taste that determines their favourites when they actively make decisions to select other users. At the same time, users are also passively involved in interactions by being selected by other users, which reflects, in some sense, their attractiveness within the social network. In this regard, both the aspects of users’ taste and attractiveness need to be modelled. We define the similar attractiveness and taste of users as follows: Two users are a similar in attractiveness (ui ↔ ui ) if they are both selected by a set of users in common. Two users are similar in taste t (ui ↔ uj ) if they both selected a set of users in common. Following the SocialCollab recommender [18], for each recommendation candidate ur in the dataset, it first finds a set of users ustr having similar taste, and another set of users usar having similar attractiveness, to the candidate ur . Then it recommends ur to the active user ua if at least one similar user in ustr selects ua and at least one similar user in usar is selected by ua . By applying CollabNet to SocialCollab, we obtain a general learnable framework for people to people recommenda-

tion. For CollabNet, the similarity table in Section III-B must be extended. Since similarity can be based on either taste or attractiveness, we create two similarity graphs corresponding to ustr and usar , for each user in the pair of an instance, taking into account their sent communications and positive replies for taste, and received communications and positive replies for attractiveness. This is illustrated in Tables III and IV, where there are similar users {u2 , u3 , u6 , u8 } to a recommended user u1 for which we need to learn the weights (u2 , u3 , u6 from Table III and u2 , u3 , u8 from Table IV). Table III ATTRACTIVENESS S IMILARITY

u1 u2 u3 u4 u5

u1 1 1 1 1

u2 1 1 1 1

u3 1 1 1 1

u6 1 1 1

u7 1 1

Table IV TASTE S IMILARITY

u1 u2 u3 u4 u5

u1 1 1 1 -

u2 1 1 1 -

u3 1 1 1

u7 1

u8 1 1 -

Once the users similar to each recommended user are determined, the contribution to the final rank of each recommended user can then be learned using Algorithm 1. Then, the recommendation can be ranked by calculating a score contributed by the similar users for each recommended user. V. E XPERIMENTAL E VALUATION In these experiments we aim to evaluate the proposed approach on people recommendation in real world datasets. Since this work is motivated by a specific application to a commercial online social networking site, so far we have evaluated our approaches on this source of data only. However, this is a demanding real-world data set. We compare our learning algorithm CollabNet to several recommendation strategies that do not use weight learning. The training and test data were pre-processed in Oracle 10 and the algorithms were implemented in Matlab. A. Experiment Setup The datasets were collected from a commercial social network site containing interactions between users. Specifically, the data contains records each of which represents a contact (communication) by a tuple containing the identity of the contact’s sender, the identity of the contact’s receiver and an indicator showing whether the interaction was successful (with a positive response from the receiver to the sender) or unsuccessful (with a negative response). The experiments were conducted on a training set covering a one week period and a test set on a subsequent week, both in March, 2009. Both training and test sets contain all users with at least one contact in the respective periods. The datasets used for this research are summarised in Table V. We learn one model for each active user. For learning, an instance contains a pair of users recommended by CF that has the first user preferred to the second user by the

Table V D ATASET D ESCRIPTION Training Set Test Set

#Interactions 188255 199083

#Positive 54754 56677

#Negative DSR 133501 0.29 142406 0.28

#Ua 3746 2865

achieved by CollabNet, where it significantly outperforms all other methods both of SR and Recall. SR for Top 10

Recall for Top 10 0.04

CollabNet SocialCollab

0.7

CF+ CF Default

CollabNet SocialCollab CF+ CF

0.035 0.03

0.6 Recall

active user. To generate these pairs, we take a target user from a positive interaction initiated by the active user as the preferred user and another target user from a negative interaction as the second user in the pair. Then the rating for the preferred user is an indicator with value 1 and the second user value 0. We compare CollabNet with a number of methods including the standard user-based CF [8] and the SocialCollab [18] recommendation methods for people using the evaluation metrics defined in Table VII.

SR

0.025 0.5

0.01 0.005

0.3 2

4

Figure 1.

6 Top N

8

RI for Each Active User 6

SR Top 100 CollabNet 0.44 SocialCollab 0.35 (0.09) CF+ 0.26 (0.18) CF 0.25 (0.19) Default 0.28 (0.16)

Top 10 0.54 0.35 (0.19) 0.30 (0.24) 0.28 (0.26) 0.28 (0.26)

Top 100 1.57 1.25 (0.32) 0.93 (0.64) 0.89 (0.68) 1.00 (0.57)

SRI Top 10 1.93 1.25 (0.68) 1.07 (0.86) 1.00 (0.93) 1.00 (0.93)

4

6 Top N

8

10

CRI for Active Users

7

600 Positive RI Flipped Negative RI

CollabNet 500

5

400 CRI

RI

Table VI R ESULTS OF R ECOMMENDATION - THE NUMBERS IN BRACKETS SHOW THE IMPROVEMENT OF C OLLAB N ET OVER THE METHOD .

2

Comparison of SR (left) and Recall(right) for Top 10.

3

As shown in Table VI, SocialCollab achieves about 0.35 SR on average for the top 100 recommendations for each active user. This means an SRI of about 1.25. However, CollabNet boosts the SR to about 0.44 for the top 100 recommendations on average with the highest SR of 0.54 for the top 10 recommendations. We also compare CollabNet to the standard CF and its extended version CF+ modelling user selection by either initiating an interaction or giving a positive response to a contact initiated by another user [18]. The details of the comparison results of those algorithms on Top 100 and Top 1000 are shown in Table VIII, which shows that the proposed algorithm CollabNet outperforms SocialCollab, CF+ and user-based CF for recommendation.

0

10

4

B. Results of Recommendation

0.02 0.015

0.4

2

← AURIC − Positive: 802.70

1

← AURIC − Flipped Negative: 279.70

0

0 200 400 600 800 1000 1200 Index of Active User (sorted descending on RI, negative first)

300 200 100 0

0

500

1000 1500 2000 2500 Number of Active Users

3000

Figure 2. Left: Ranking Improvement (RI). The negative RI is flipped with respect to the origin to compare with the positive RI, showing the overall ranking improvement of CollabNet. Right: Cumulative Ranking Improvement (CRI). The increasing CRI curve indicates the stable increase in RI with the increase in the number of active users.

C. Results of Ranking We also show results for ranking in Figure 2, where a point in the positive RI curve denotes an improved ranking of the recommended users for an active user by learning, while that in the negative RI curve denotes a worse ranking. In Figure 2, the positive RI curve covering a large range of active users indicates that there are many more active users receiving a better ranking by learning. The area under the RI curve (AURIC) for positive RI is about 3 times that for negative RI, which demonstrates that the increase of RI is much higher than the decrease of RI for the same number of active users. This can also be seen in Figure 2 where cumulative ranking improvement (CRI) is shown, which indicates the stable increase in RI. VI. C ONCLUDING R EMARKS

We also compared CollabNet to the recommendation performance of the other algorithms on the top 10 of the ranked recommendation list as shown in Figure 1. We can see that although without learning SocialCollab performs better than default (DSR) on the top 10 recommendations, CollabNet performs much better than SocialCollab on the top 10 recommendations on both SR and Recall. When we compare CollabNet to CF+ and standard CF on the top 10 recommendations, it shows that standard CF performs at just about the default level and CF+ performs slightly better than standard CF, however, the most significant improvement is

We have proposed an approach for people recommendation by collaborative filtering and machine learning. Our experimental results show that the novel CollabNet recommender performs well in people to people recommendation on data from a commercial online social networking site. The proposed learning algorithm is able to rank the recommendations in order to further improve the success of predicted user interactions. The proposed algorithm CollabNet outperforms all other methods including standard CF as measured on both Precision (SR) and Recall. For future work, we will investigate the the scalability of CollabNet, the

Table VII E VALUATION M ETRICS Name

Description The proportion of the true predicted successful interactions (PSI) to all predicted successful interactions: n SR = ntpsi , where ntpsi is the number of true PSI and npsi the number of PSI.

Success Rate (SR) or Precision

psi

Default Success Rate (DSR) Success Rate Improvement (SRI) Recall Ranking Improvement (RI)

tsi The proportion of successful interactions (SI) to all interactions in the dataset: DSR = n nall where ntsi is the number of true SI and nall the number of all interactions. SR The ratio of success rate to the default success rate: SRI = DSR . ntpsi The proportion of the true PSI to all true SI: Recall = n . tsi The product of the difference between the predicted rank and ground truth rank#and the response indicator: " 1 if #ua , uj $ ∈ SI RI i = riij , riij = (ngj − npj ) × sj , ngj = gj /ni , npj = pj /ni , sj = j −1 otherwise where gj is the reference rank index for recommended user j, pj the predicted rank index, ni the number of recommended users for the active user i. Positive RI values indicate improved ranking.

Table VIII C OMPARISON OF SR FOR T OP 1000 R ECOMMENDATIONS Top CollabNet SocialCollab CF+ CF

10 0.54 0.35 0.30 0.28

20 0.50 0.34 0.28 0.27

30 0.49 0.34 0.27 0.26

40 0.49 0.35 0.27 0.26

50 0.47 0.34 0.28 0.26

60 0.47 0.35 0.28 0.26

“cold start” problem of providing recommendations for new users, and a general probabilistic framework for ranking in the context of the CollabNet algorithm, as well as compare CollabNet with approaches to link prediction. ACKNOWLEDGMENT This project is funded by the Smart Services Cooperative Research Centre. R EFERENCES [1] Y. Zhang, J. Callan, and T. Minka, “Novelty and Redundancy Detection in Adaptive Filtering,” in Proceedings of the 25th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval, 2002, pp. 81–88. [2] D. Billsus and M. J. Pazzani, “User Modeling for Adaptive News Access,” User Modeling and User-Adapted Interaction, vol. 10, pp. 147–180, 2000. [3] R. J. Mooney and L. Roy, “Content-Based Book Recommending Using Learning for Text Categorization,” in Proceedings of the 5th ACM Conference on Digital Libraries, 2000, pp. 195–204. [4] M. Pazzani and D. Billsus, “Learning and Revising User Profiles: The Identification of Interesting Web Sites,” Machine Learning, vol. 27, pp. 313–331, 1997. [5] J. S. Breese, D. Heckerman, and C. Kadie, “Empirical Analysis of Predictive Algorithms for Collaborative Filtering,” in Proceedings of the 14th Conference on Uncertainty in Artificial Intelligence, 1998, pp. 43–52. [6] J. Konstan, B. Miller, D. Maltz, J. Herlocker, L. Gordon, and J. Riedl, “GroupLens: Applying Collaborative Filtering to Usenet News,” Communications of the ACM, vol. 40, no. 3, pp. 77–87, 1997. [7] G. Linden, B. Smith, and J. York, “Amazon.com Recommendations: Item-to-Item Collaborative Filtering,” IEEE Transactions on Internet Computing, vol. 7, no. 1, pp. 76–80, 2003. [8] U. Shardanand and P. Maes, “Social Information Filtering: Algorithms for Automating “Word of Mouth”,” in Proceedings of the ACM Conference on Human Factors in Computing Systems (CHI’95), 1995, pp. 210–217.

70 0.45 0.35 0.27 0.26

80 0.45 0.35 0.27 0.26

90 0.44 0.35 0.26 0.25

100 0.44 0.35 0.26 0.25

200 0.40 0.36 0.27 0.26

300 0.39 0.36 0.27 0.26

400 0.38 0.37 0.28 0.27

500 - 1000 0.37 0.37 0.28 0.27

[9] B. Sarwar, G. Karypis, J. Konstan, and J. Reidl, “ItemBased Collaborative Filtering Recommendation Algorithms,” in Proceedings of the 10th International Conference on World Wide Web, 2001, pp. 285–295. [10] J. Delgado and N. Ishii, “Memory-Based Weighted-Majority Prediction for Recommender Systems,” in Proceedings of the ACM SIGIR’99 Workshop on Recommender Systems: Algorithms and Evaluation, 1999. [11] D. Billsus and M. J. Pazzani, “Learning Collaborative Information Filters,” in Proceedings of the Fifteenth International Conference on Machine Learning, 1998, pp. 46–54. [12] L. Getoor and M. Sahami, “Using Probabilistic Relational Models for Collaborative Filtering,” in Working Notes of the KDD-99 Workshop on Web Usage Analysis and User Profiling, 1999. [13] T. Hofmann, “Collaborative Filtering via Gaussian Probabilistic Latent Semantic Analysis,” in Proceedings of the 26th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval, 2003, pp. 259– 266. [14] D. Pavlov and D. M. Pennock, “A Maximum Entropy Approach to Collaborative Filtering in Dynamic, Sparse, HighDimensional Domains,” in Neural Information Processing Systems, 2002, pp. 1441–1448. [15] L. Ungar and D. Foster, “Clustering Methods for Collaborative Filtering,” in Proceedings of the AAAI-98 Workshop on Recommender Systems, 1998, pp. 112–125. [16] B. Cao, J.-T. Sun, J. Wu, Q. Yang, and Z. Chen, “Learning Bidirectional Similarity for Collaborative Filtering,” in Machine Learning and Knowledge Discovery in Databases, W. Daelemans, B. Goethals, and K. Morik, Eds. Berlin: Springer-Verlag, 2008, pp. 178–194. [17] C. Burges, T. Shaked, E. Renshaw, A. Lazier, M. Deeds, N. Hamilton, and G. Hullender, “Learning to Rank Using Gradient Descent,” in Proceedings of the 22nd International Conference on Machine Learning, 2005, pp. 89–96. [18] X. Cai, M. Bain, A. Krzywicki, W. Wobcke, Y. S. Kim, P. Compton, and A. Mahidadia, “Collaborative Filtering for People to People Recommendation in Social Networks,” in AI 2010: Advances in Artifical Intelligence, J. Li, Ed. Berlin: Springer-Verlag, 2010.