Case-Based User Profiling for Content Personalisation - CiteSeerX

Case-Based User Profiling for Content Personalisation Keith Bradley , Rachael Rafter & Barry Smyth Smart Media Institute Department of Computer Science University College Dublin Belfield, Dublin 4, Ireland {Keith.Bradley, Rachael.Rafter, Barry.Smyth}@ucd.ie

Abstract. As it stands the Internet’s “one size fits all” approach to information retrieval presents the average user with a serious information overload problem. Adaptive hypermedia systems can provide a solution to this problem by learning about the implicit and explicit preferences of individual users and using this information to personalise information retrieval processes. We describe and evaluate a two-stage personalised information retrieval system that combines a server-side similarity-based retrieval component with a client-side case-based personalisation component. We argue that this combination has a number of benefits in terms of personalisation accuracy, computational cost, flexibility, security and privacy.

1

Introduction

The Internet currently consists of approximately 800 million pages of information across more than 8 million web sites, and there are over 200 million regular users from all walks of life with varying degrees of computer literacy and a wide range of interests and preferences. Retrieving the right information for a particular user is one of the most difficult information retrieval tasks imaginable. It is made more difficult because a typical Internet user is unlikely to provide a well-specified search query to a search engine. Many search queries contain only two or three search terms for example, and the inevitable result is that the user is swamped with a deluge of irrelevant retrieval results. Recent work in the areas of adaptive hypermedia, content personalisation and user profiling(e.g. [1,2,5,6]) has presented a number of potential solutions to this problem by automatically learning about the likes and dislikes of individual users in order to personalise information retrieval for each user. For example, by constructing a detailed profile on a user it may be possible to supplement that user’s search queries with additional relevant search terms, or it may be possible to filter the retrieval results to weed our irrelevant hits. In this paper we describe part of the work carried out in the CASPER (Case-Based Profiling for Electronic Recruitment) project [3,10,11]. Very briefly, the aim of CASPER is to build a more intelligent search engine for use in specialised Internet information retrieval applications – specifically, our application test-bed is the

JobFinder web site (www.jobfinder.ie), which specialises in the provision of job listings. Several approaches to this problem were examined in the CASPER project with a view to discovering synergies between the methods. Rafter et al., in this volume, discuss a different but complimentary approach that uses Automatic Collaborative Filtering to also provide a job recommendation facility for CASPER. The focus of that work is on providing a query-less content-free filtering service, whereas in this paper we focus on a content-based search service. Here we describe and evaluate a two-stage personalised job case retrieval system (CASPER PCR) that combines a server-side similarity-based retrieval component with a client-side case-based personalisation component. We argue that this combination benefits from superior cost, flexibility, and privacy characteristics.

2

Related Work

Recently a number of successful adaptive hypermedia systems have appeared under a number of headings including “personalised systems”, “recommender systems”, and “information butlers”. Many share a common goal: to learn about the implicit preferences of individual users and to use this information to serve these users better. PIMA is a client-side, content-based recommender system that monitors a user's actions in an application environment such as MS Word and suggests relevant resources from the Web to aid the user in their current task [4]. As the user works the PIMA client builds a summary of the current page using standard content-based information retrieval techniques. This information then forms the basis of queries that are submitted to various web-based resources such as search engines. PIMA is essentially a reactive system and does not maintain persistent profiles of the user. In other words profile information is lost at the end of a session. Both Personal Web Watcher (PWW) [9] and Letizia [8] are content-based systems that recommend web-page hyperlinks by comparing them with a history of previous pages visited by the user. Letizia uses the unused time when the user is actually reading a web page to search the adjoining hyperlinks. The user’s profile is composed of keywords extracted from the various pages visited. PWW uses an offline period to generate a bag of words style profile for each of the pages visited during the previous browsing session. Hyperlinks on new pages can then be compared to this profile and graded accordingly. GroupLens [7] is a server-side recommendation engine for Usenet news. A user’s profile is created by recording the user’s explicit ratings of various articles. Automatic collaborative filtering is then used to statistically compare one user’s likes and dislikes with another user and to recommend articles from other similar users profiles. PTV is a personalised TV listings service, providing users with personalised TV guides based on their learned viewing preferences [12,13]. It employs a hybrid recommendation strategy using both content-based and collaborative methods. Its server uses explicit user gradings on various programmes, channels, and viewing times to build a profile. CASPER’s PCR component shares many of the same motivations and objectives as the above systems. Like GroupLens and PTV, CASPER maintains long-term user

profile and like PIMA, PWW and Letizia it implements a client-side personalisation policy. Like PIMA, PWW, Letizia and PTV, our system uses a content-based approach to personalisation, while these systems use unstructured keyword-type profiles, CASPER PCR uses a structured case-base profile representation.

3

Case-Based User Profiling for Content Personalisation

JobFinder is an award-winning Irish recruitment site that provides conventional search engine functionality over a dynamic recruitment database. Its reliance on exact-match retrieval functionality introduces many problems for the average jobhunter. For example, a JobFinder user might specify that they are looking for a software development job with a salary of £25k. However, the reality is that they may actually be looking for a permanent C++ or Java software development job, with a salary in the region of £25k (preferably greater), in the Dublin or Wicklow area. In other words, the supplied query is usually only a very small part of the equation. Furthermore, as it stands, two users with the same queries will receive the same search results even though they may both have very different personal preferences – a second user, looking for a “software development job in the region of £25k”, may actually be looking for a Visual Basic job in Cork, and not want to see information on Dublin, C++ or Java jobs. This makes JobFinder an excellent test-bed for a more sophisticated approach to information retrieval, and the aim of the CASPER project is to assess the potential impact of artificial intelligence and content personalisation techniques in a JobFinder setting. Specifically, in the remainder of this paper we will describe and evaluate a two-step personalised retrieval engine designed to improve the precision of JobFinder’s searches. CASPER SERVER

CASPER PCR CLIENT

Sample Case Query

User Sets Feature importance Results Form

Case Base

Update Profile

Retrieved Cases

Search Form

Sim Search

User Feedback

Description :

Original Text

Type

:

Perm

Salary

:

2000030000

Experience :

2

Location

:

Dublin

Skills

:

Java, www

Example of Retrieved Case

Recommendations Personalise

Fig. 1. Overview of the two-stage retrieval process in CASPER PCR

3.1

CASPER Personalised Retrieval Architecture

Fig. 1 outlines the architecture of CASPER’s personalised case retrieval engine. When a user enters a new search query, a server-side similarity-based search engine is used to select a set of similar job cases (see Section 3.2). However, unlike a traditional search engine, CASPER does not return these results to the user directly. Instead, they are processed on the user’s machine by the personalisation client. Personalisation is essentially a post-processing retrieval task where the result-set is compared to a user profile in order to filter-out irrelevant jobs (see Section 3.3). Each user profile is simply a set of graded jobs that the user has viewed (and graded) in the past as part of the normal JobFinder functionality. The final result of a retrieval session is a set of jobs that are not only relevant to the user’s search query but that are also appropriate given the preference encoded within the user’s profile. 3.2

Stage 1: Similarity-Based Retrieval

The first stage of the CASPER PRC system is equivalent to a traditional searchengine in the sense that it takes a user’s query and compares it to job cases in the job case-base to identify the best matches. However, unlike a traditional search engine, this process is based on a similarity-based retrieval technique rather than on an exactmatch technique. This means that relevant jobs can be identified even though they share no features with the original user query. Each job case is made up of a fixed set of features such as job type, salary, key skills, minimum experience etc. And during retrieval the user’s search query or target query is compared on a feature by feature basis to each of the job cases. The similarity metric shown in (1) is used to compute the similarity between each job case and the target query, based on the similarity of corresponding features; this metric is a fairly standard weighted-sum metric. (1)

n

Sim(t , j ) =

∑

similarity (t i , j i ) * wi 

i =1

The key to the success of this similarity-based approach is the ability to accurately estimate the similarity between individual features of a job case and target query. Each case will contain different types of features; for example, the minimum experience feature is a numeric feature while key skills contains symbolic values. We have defined a range of different feature similarity metrics to handle each of the different case features. For example, numeric features can be handled using a simple distance function; although some numeric features (for example salary) may use a more sophisticated asymmetric distance function to account for certain types of biases (for example most users are happy to accept jobs with higher salaries than they asked for, but the reverse is not always true). Symbolic features are more difficult to handle. In CASPER we have defined a domain ontology in which symbolic features are represented as concept trees, and

symbolic feature similarity is based on subsumption relationships and on the distances between nodes in these trees; see Fig. 2(a). For example, CASPER PCR performs two types of similarity-based matching using concept trees. The first type is called subsumption matching (see Fig. 2(b)) and allows for different granularities of abstraction to be used in a search query. For example in Fig. 2(b) the user has submitted a query for OOP or Object Oriented Programming based jobs. Any job containing a concept that is a descendant of OOP in the tree is taken to be an exact match. This is logical since all descendants of OOP in the tree are object oriented programming skills.

(a)

(b)

Fig. 2. (a): A concept tree: this tree shows the domain knowledge available for job skills (b): Subsumption Matching: descendants of a node are taken as exact matches for that node

The second type of concept similarity method is based on concept proximity as shown in Fig. 3(a) and Fig. 3(b). In other words, the closer two concepts are in the tree the more similar they are. As an example in Fig. 3(a) the distance between C++ and Java is 2. These concepts are therefore taken to be quite similar. Distance is measured in terms of the number of inter nodal edges or links between the two concepts. By contrast the similarity between Java and Pascal is judged to be quite low as shown in Fig. 3(b) since there are four edges between them.

(a)

(b)

Fig. 3. (a): Proximity Matching: the skills of C++ and Java were taken to be conceptually quite similar (b): Proximity Matching: in this case Pascal and Java were deemed less similar

It is quite possible for concepts to recur within a given concept tree, that is, for a concept to have multiple parents. The final similarity score is then taken to be the best similarity score from all of the permutations of candidate / target available.

3.3

Stage 2: Result Personalisation

As it stands CASPER’s similarity-based retrieval engine allows job cases to be retrieved on the basis of a fluid feature-based similarity assessment procedure borrowed from the case-based reasoning community. However, as it stands this retrieval approach is still ignorant of the implicit need of individual users, and therefore retrieval results are not personalised for individual users. This motivates the need for the second stage of retrieval, which is capable of personalising the retrieval results from the client-side. Before describing how this personalisation process works, let us first talk briefly about CASPER’s user profiling functionality. As users interact with the JobFinder system they are presented with various jobs and a number of useful options. For example, users can apply for jobs online, they can submit their resume to the relevant contact point, or they can e-mail themselves a copy of the job for their files. As part of the CASPER project we have explored various techniques for monitoring this type of user activity and for converting it into useful user profiles [10,11]. The upshot of this is that user profiles are available for each JobFinder user and these profiles contain a record of which jobs a user has liked and which jobs they have not liked. The client-side personalisation technique views the personalisation task as a classification problem. In other words, personalising the retrieval results amounts to classifying each individual retrieved job as either relevant or not relevant. To do this we propose a nearest-neighbour type classification algorithm that uses the graded job cases in a target user’s profile as training data. Each profile case serves as a training example with the job grade (liked or disliked / good or bad) serving as the example classification. In order to classify a candidate retrieved job we compare it to each profile job case, using the similarity metric defined earlier, to locate the k nearest profile jobs; the value of k is usually a low odd number and needs to be set experimentally for the classification domain in question. A judgement about the job’s relevance can then be based on the classifications of these nearest neighbours. For example, one simple strategy is to take the majority classification of the nearest neighbours; according to this strategy, the candidate job shown in Fig. 4 would be classified as ‘good’ or relevant (with k=5).

B

G G

G B B

? B

G

G B

B Bad

G

G Good Jobs ? Candidate Job

Fig. 4. Client-side personalisation is viewed as a classification problem and implemented with a k nearest-neighbour classification algorithm.

  Re l( j , P ) = 1 if  Sim( ji , p ).classification( p ) > 0 , − 1 otherwise . ∑  p ε knn( j ,P )   

(2)

Instead of the simple majority classification strategy, CASPER PCR uses a weighted-similarity metric whereby the similarity of a profile case to the retrieval candidate determines how much influence that case has in the overall classification, as shown in (2). The relevance of a candidate job j to user profile P is the sum of the product of the similarity values of the k nearestneighbours to j by their class value (+1 indicates that the user liked the job and –1 indicates that they disliked it).

4

Experimental Evaluation

Of course ultimately the important question to be asked about CASPER’s approach is does it work? Does it produce retrieval results that are more relevant to a given user than JobFinder’s conventional retrieval engine? Does it produce personalised results that reflect a user’s implicit preferences? In this section we describe some preliminary experiments, carried out on real users, to evaluate the effectiveness of the personalisation side of CASPER PCR.

4.1

Set-Up

Eleven subjects were selected for this experiment, all of them postgraduate students and staff members in the Department of Computer Science at University College Dublin. In addition a subset of 58 jobs (from a database of 3800 jobs) were chosen as the basis for user profiles for the subjects. These jobs were selected because they provided a good cross-section of employment characteristics. Each subject was asked to grade each of the 58 jobs as relevant or not relevant to provide the raw data for the 10 user profiles.

4.2

Experiment 1

The aim of this experiment is to evaluate the impact of k (the number of nearest neighbours used in the classification process) on the classification/personalisation accuracy (this is equivalent to the precision of the filtered result set). We used a leaveone-out method to compute the overall personalisation accuracy for each profile for different values of k (1 – 15). That is, for each profile and value of k, we classified each profile job using the remaining 57 profile jobs, and compared the predicted classification (relevant or not relevant) to the actual known classification as provided by the user during the initial grading process. The personalisation accuracy for a given user is the percentage of correct classifications over the set of 58 jobs. For each

Mean Classification Accuracy

value of k, this produces 10 personalisation accuracy values, which are averaged to produce a mean accuracy over all profiles for this value of k. The results are presented in Fig. 5 as a graph of mean classification accuracy versus k. The results show that as k increases so too does the classification accuracy, up to a point. The maximum accuracy tends to be obtained at around the k=11 or k=13 mark. This behaviour is to be expected as nearest-neighbour algorithms are known to be adversely affected by training noise for low values of k. Our user profiles are known to contain a lot of noise in the sense that users are often inconsistent in their grading of jobs, and hence we would expect improved performance from higher values of k. All of the jobs presented to the users were obtained from a query submitted to the first stage retrieval engine. We therefore chose to use the number of jobs liked by the user from this initial query as our baseline. Since we are looking at the average accuracy for each value of k across all users, we averaged this base measure for all users to give the baseline shown in Fig. 5. This baseline therefore corresponds to the average accuracy of the first stage retrieval system. In general the results are very positive, showing a personalisation accuracy of over 70% for reasonable values of k and relatively small profiles. In fact, it is particularly positive given the high degree of noise in the profile data. The personalisation system consistently outperforms the first stage retrieval when taken on its own by a significant margin.

0.7 0.65 0.6

PCR

0.55

Base

0.5 0.45 0.4 1

3

5

7

9

11

13

15

k

Fig 5. Mean personalisation/classification accuracy across all users for varying values of k.

4.3

Experiment 2

The aim of this next experiment is to examine the personalisation accuracy for different individual users at a given value of k. In this case we chose k=13, since this

value returned the highest overall accuracy. The personalisation accuracy for k=13 was calculated using the same method described above and the results are shown in Fig. 6 for each of the 10 users. In general the results are positive although there is a high degree of variation between individual users. Overall, 8 out of the 10 users benefit from a personalisation accuracy in excess of 60%. However, for one user (6) the reported accuracy was only 57%; this users were found to have a very noisy profile with many inconsistencies apparent in the way that the original jobs were graded. Again we used the number of jobs liked by the user out of the original query as a baseline. Clearly in almost all cases the personalisation engine achieved better results and indeed achieved significant improvements in over half of those tested. 1 Classification Accuracy

0.9 0.8 0.7 0.6 PCR BASE

0.5 0.4 0.3 0.2 0.1 0 1

2

3

4

5

6

7

8

9

10

User

Fig 6. Mean personalisation/classification accuracy for all users at k = 13.

5

Discussion

CASPER’s PCR approach to client-side personalisation brings with it a number of potentially significant advantages. First and foremost is the fact that the computational load is transferred from the server-side to the client side, and this brings with it the potential for greatly improved overall response times. However, possibly the most important benefit has to do with the privacy and security issues. CASPER PCR does not require server-side access to user profiles, and unlike collaborative personalisation techniques [7,12,13], there is no need to access the profiles of other users. This means that an individual user can benefit from personalisation safe in the knowledge that her profile has never left her client

machine; this is especially important in any personalised applications that may involve the collection of sensitive personal data, for example, medical data or financial data. This advantage is not available with many other personalisation techniques and is becoming more and more critical in the context of promised privacy legislation for e-commerce and Internet systems. Finally, although we have described the CASPER PCR approach in the context of the JobFinder domain, there is of course no reason why the approach could not be applied to a range of other information retrieval domain. In fact, we would suggest that any existing search engine technology could benefit from CASPER’s client-side personalisation strategy, assuming that suitable user profiles are available or can be collected.

6

Conclusions

We have motivated the need for personalised information retrieval systems and described one particular implementation in the form of CASPER PCR. This technique works as a two stage process: stage one performing a similarity-based server-side retrieval, and stage two performing a content-based filtering of the retrieval results with respect to a store user profile. Our evaluation demonstrates the technique to be effective. We have argued that this combination of server-side search and client-side personalisation offers a number of important advantages over many existing approaches, not the least of which are the advantages pertaining to user privacy and security issues. The benefit, from a users point of view, of hiding valuable and private profile data from a remote server cannot be understated, and in the future may even a critical feature for the universal adoption of personalisation technology. We believe that this positions our work well as a realistic, practical and safe personalisation technology for the future.

7

References

1 Balabanovic M, Shoham Y.: FAB: Content-Based Collaborative Recommender. Communications of the ACM 1997; 40:3:66-72. 2 Billsus D & Pazzani M.: Learning Collaborative Information Filters. In: Proceedings of the International Conference on Machine Learning. Morgan Kaufmann, Madison, Wisc. 1998 3 Bradley, K., Rafter, R., and Smyth, B.: Personalised Case Retrieval. In: Proceedings of the 10th Irish Conference on Artificial Intelligence and Cognitive Science, Cork, Ireland. (1999) 4 Budzik, J., Hammond, K. J., Marlow, C., and Scheinkman, A: Anticipating Information Needs: Everyday Applications as Interfaces to Internet Information Sources. In Proceedings of the 1998 World Conference on the WWW, Internet, and Intranet. AACE Press, (1998) 5 Goldberg D, Nichols D, Oki B M, Terry D.: Using Collaborative Filtering to Weave an Information Tapestry. Communications of the ACM 1992; 35: 12: 61-70 6 Kay J.: Vive la Difference! Individualised Interaction with Users. In: Proceedings IJCAI ’95, Montréal, Canada, (1995), 978-984

7 Konstan J.A, Miller B.N., Maltz D, Herlocker J.L., Gordon L.R., Riedl J: Group Lens: Applying Collaborative Filtering to Usenet News. Communications of the ACM, 40(3) (1997) 77-87 8 Liebermann, H.:Letizia: An agent that assists web browsing. In: Proceedings 14th International Conference Artificial intelligence (IJCAI) (1995) 9 Mladenic, D.: Personal WebWatcher: Implementation and Design. Technical Report IJSDP-7472, Department of Intelligent Systems, J.Stefan Institute, Slovenia (1996) 10 Rafter R., Bradley K., Smyth B.: Passive Profiling and Collaborative Recommendation. In: Proceedings of the 10th Irish Conference on Artificial Intelligence and Cognitive Science, Cork, Ireland, September.(1999) 11 Rafter R., Bradley K., Smyth B.: Personalised Retrieval For Online Recruitment Services. In: Proceedings of the 22nd Annual Colloquium on IR Research, Cambridge, UK, (2000) 12 Smyth B & Cotter P.: Surfing the Digital Wave: Generating Personalised Television Guides using Collaborative, Case-based Recommendation. In: Proceedings of the 3rd International Conference on Case-based Reasoning, Munich, Germany, (1999) 13 Smyth B & Cotter P.: Sky’s the Limit: A Personalised TV Listings Service for the Digital Age. In: Proceedings of the 19th SGES International Conference on Knowledge-Based and Applied Artificial Intelligence (ES99). Cambridge, UK, (1999)