A Case-Based Approach to User Modeling

A Case-Based Approach to User Modeling Alessandro Micarelli, Filippo Sciarrone, Leonardo Ambrosini, and Vincenzo Cirillo Dipartimento di Informatica e Automazione Universita di Roma Tre Via della Vasca Navale, 79 I-00146 Roma, Italia [email protected]

Abstract. In this paper we present a case-based approach we have used

for the realization of HUMOS, a hybrid user modeling shell system that has been designed for building and maintaining long term models of individual Internet users. A distinguished feature of HUMOS is the hybrid approach to user modeling, where a sub-symbolic module is integrated into a case-based reasoner. A case study in information ltering systems on the World Wide Web is also presented. The described system, named WIFS, is based on HUMOS and is capable of selecting HTML/text documents, collected from the Web, according to the interests and characteristics of the user. Presently the system acts as an intelligent interface for the Web search engines. The experimental results we have obtained are encouraging and support the choice of the case-based approach to user modeling for adaptive systems.

1 Introduction An interactive system, in order to adapt its behavior to the needs of the user, must be capable of dynamically building a representation of the user's interests and characteristics (see, for example, [11]). This representation is commonly called the User Model. User Modeling is an important area of Arti cial Intelligence research, and involves the explicit acquisition and representation of information about the user. In recent years researchers have been challenged with the de nition and realization of General User Modeling components. These shells are designed to provide developers of interactive systems with domain-independent tools for building a User Modeling component. In particular they provide: (1) a representation language for the user's knowledge, beliefs and goals; (2) inference capabilities; (3) mechanisms for detecting inconsistencies and revising knowledge and beliefs. This paper presents HUMOS (Hybrid User Modeling System), a User Modeling shell system, developed in Java and speci cally conceived for Internet applications. HUMOS has been designed for building and maintaining long term models of individual users. These are models concerning a single user which are accessed, possibly, for more than one session. One distinguishing feature of this system is its hybrid architecture: a combination of a Case-Based Reasoner with a

sub-symbolic module (here, an arti cial neural network). The use of an arti cial neural network for CBR systems can also be found, for example, in [19] [13]. As a rst application of the HUMOS engine, we present a system for Information Filtering [3] [9] [17] [18] of HTML/Text documents collected from the WWW, where the selection of the documents relevant for a particular user is performed on the basis of a model which represents the user's interests and characteristics. The system has been used as an intelligent interface for the search engine AltaVista. The paper is structured as follows. In Section 2 the hybrid component of the User Modeling system is presented. In Section 3 we present a case study in the eld of Adaptive Information Filtering on the WWW. Then we describe the results of the evaluation of the integrated system. In a concluding section we give some nal remarks.

2 The Hybrid Component in HUMOS A User Model in HUMOS is a description of the user in terms of a set of multivalued attributes. The user modeling process entails identifying the current user; retrieving the relevant User Model (or performing a preliminary interview to create one if none exists); updating the model on the basis of how users interact with the system; and, lastly, furnishing data to answer explicit requests about the user made by the host system. HUMOS, like other User Modeling Shells (see, for example, [4] [12]) extends its beliefs about the user by relying on a set of stereotypes. A stereotype is the formal description of a prototypical user of a given kind [23]. Stereotyping is a way of default reasoning about the user: by classifying the user we exploit a lot of default information we have on that class of users. This information may be revised later on by the system (inference activity, consistency checking) when it gets more accurate knowledge about user's interests. Activation of stereotypes is generally carried out on the basis of triggering conditions [23]: for each stereotype there are several conditions that, if asserted within the model, are sucient to activate the stereotype, i.e. to classify the user. Usually the triggering conditions may be logical OR or AND of simpler conditions. Given the goal of the present paper, we will limit ourselves to indicating how HUMOS oers an original case-based approach to user stereotyping (see [1] [2] for more details concerning other aspects of HUMOS architecture). A possible case-based approach to the selection of the most suited stereotype, on the basis of the user behaviour, is presented in Fig. 1. At the heart of the component is a Case Library with a basic stock of cases of typical users (as de ned by experts in the domain concerning the documents to be ltered). Each case is represented by a frame with three slots: \user behavior" (a pattern constituted by the actual values of the attributes for a particular user, initially gathered by the system by means of a preliminary interview and dynamically updated on the basis of feedback given by the user), \active stereotype" (which is in eect a pointer to a Library of stereotypes), and a procedural attachment, activated when the old case is indexed, which triggers the knowledge

Case Library Stereotypes Library

User Description

Indexing Module

. . .

. . .

Neural Network User Description

Rules Activation

Active Stereotype

Fig. 1. Case-based Approach to User Modeling. base of adaptation rules in order to adapt the stereotype selected to the content of the user model). When the system is presented with a pattern of attributes relative to the particular user, the indexing module tries to nd the old case that closely matches (according to a speci c metric) the new case. The selected old case contains all the relevant information useful for classifying the user, i.e., the most suited stereotype and the daemon that activates the adaptation rules, starting from the selected stereotype and the actual pattern representing the user behaviour. One problem posed by this approach is in the determination of a metric to be used in the indexing module. In fact all user modeling systems face a common problem: they must classify users even though data is usually incomplete and often con icting. Our proposed solution (similar to a framework which has already been successfully experimented in the domain of adaptive hypermedia, see [14] [15]) consists of using a function-replacing hybrid [7] [8], where an arti cial neural network implements (i.e., is functionally equivalent to) the Indexing and Case Library features (the box in bold line in Fig. 1) typically present in traditional implementations of case-based systems. One advantage of this choice is that the distributed representation and reasoning of the neural network allows the system to deal with incomplete and inconsistent data and also allows the system to \gracefully degrade" [24]. Since this kind of classi cation problem is in general not linearly separable, a Multi-Layer-Perceptron [24] with three distinct layers has been used (see Fig. 2). The rst layer, the input layer, is composed of the neurons relative to the n attribute-value pairs (that are coded into numeric values) present in all the stereotypes; input values are provided by the weight that each pair has in the current pattern. The output layer is composed of as many neurons as the number of the stereotypes. The output values are computed by the network according to a given input; this corresponds to the computation of a rank-ordered list of stereotypes present in the library (Fig. 4). As for the hidden layer, there are no theoretical guidelines for determining the number of hidden nodes. We have selected the optimal number of hidden neurons in the context of the training procedure, where the back-propagation algorithm [24] has been used. During the training phase, we have used the Simulated Annealing algorithm [10] for avoiding local minima as well as escaping

Stereotypes Library

p1

Stereotype 1

p2

Stereotype 2

p3 Stereotype 3



......

...... pm

......

Stereotype m



Input Layer

Hidden Layer

Output Layer

Fig. 2. The Arti cial Neural Network. from them when necessary. The number of hidden units has been set to 20 (see Fig. 3) after the network had been trained in 5 dierent con gurations (see [14] [15] for more details concerning the authoring phase of the network). In our initial experimental comparison between a traditional implementation of the indexing process using triggers and our hybrid equivalent the performance of the hybrid proved better. Two main parameters were considered: a) root mean square error (RMSE) to express the \distance" between the values of the rank ordered list computed by the system and the expected ones; b) accuracy, de ned as percentage of the stereotypes, from among those most expected by the human experts for each testing pattern, that actually appeared (at least in the second position) in the list provided by the system. The RMSE using the hybrid architecture reached 0.019 and the accuracy reached 95%, whereas with the traditional triggers, activated by selecting a commutation switch, the RMSE was 0.214 and the accuracy 80% (see Fig. 4).

3 A Case Study: Adaptive Information Filtering on WWW HUMOS has already been integrated within other host systems: in Ocram-CBR, a case-based shell for educational applications [21], and in two Ocram-CBR

Fig. 3. RMSE of the networks during the test.

Triggers Hybrid Accuracy

RMSE 0.214

0.2

100

95 80

0.019 0

0 a)

b)

Fig. 4. Triggers vs Hybrid approach.

WIFS Current query

External Retriever

Current doc.

Semantic Networks & Terms Data Base

Stop List

WWW

AltaVista

HUMOS Stereotypes Module

Filtering Unit

Query Unit

Doc. H. Unit

User Model

Feed. Unit

Model Management Unit

Truth Maintainance Unit

Model Unit

Modeling Rules KB

User Interface

USER

Fig.5. The architecture of the integrated system. applications, a business letter-writing tutor [22] and a user-adapted training system to train employees about business engineering [20]. But the most eective use of HUMOS has been up to now in the eld of Information Filtering on the Web (see [2] for further details on the intelligent interface to a search engine we are going to explain in this Section). Fig. 5 shows the integrated architecture of the complete system, composed by two modules: HUMOS (the User Modeling component) and WIFS (the Information Filtering component). The ltering process is made up of the following steps (see [2] for more details): The Logon phase: identi cation of the current user. The initialization phase, performed by the Model Handling Unit: retrieval of the corresponding user model from HUMOS. Whenever the user is not known, the Unit conducts a preliminary interview in order to collect a rst set of information. In particular it asks the user her/his likes and dislikes, assigning each an importance weight (positive for interesting attributes, otherwise negative). This information is then sent to HUMOS to identify a suitable stereotype and create an appropriate model. { The editing/customising phase, performed by the Model Handling Unit: user perusal and editing of the Model created for her/him. The User Model and the modi cations performed by the user are sent to HUMOS in order to

{ {

{

{

{

{

carry out the modeling process through the following activities: activation of stereotypes, ring rules and inconsistency checking. The querying phase, performed by the Query Unit: display of a window (see Fig. 6) where the user can input her/his query and establish the searching and ltering modalities. AltaVista syntax lets the user write both boolean queries (boolean AND/OR combinations of keywords) and structured queries (which allow the user to constrain matches to certain attributes, such as document type, title, host, etc.). The parsing phase, performed by the External Retriever: parsing, analysis and extraction of the structured representations of the HTML/Text documents retrieved by AltaVista. The structural elements are: title, abstract, author/s, URL, size, relevant keywords (with the help of the Stop-List, i.e., a list of irrelevant words like conjunctions, articles, etc., that should be ignored by the system) and their frequency in the text. The ltering phase, performed by the Filtering Unit: activation of the ltering algorithm MAF (Matching Algorithm for Filtering) we have proposed, in order to assign a score to each representation calculated as the similarity between the document, the user model and the query. It is interesting to note how MAF computes this similarity: besides evaluating the conventional vector product (between corresponding vectors of document, pro le and query), MAF properly exploits the occurrence of semantic links and terms (see below) found in the document. Supporting these structures provides an accurate ltering process. All documents are ordered by descending score and shown to the user by the Document Handling Unit. An important feature of MAF is the ability to identify topics composed of multiple keywords: in fact the system can distinguish between topics like \user modeling" and single keywords like \user" and \modeling". The feedback phase, performed by the Feedback Unit: input of a relevance value, assigned by the user, for each document viewed. The value tells the system how satis ed the user is with it. This feedback, the representation of the document and the query are then all used to modify the user model by inserting newly found topics and updating the weight of topics already in the model. Thus the model evolves according to user behavior. The UFM also checks the weight of each component, and will delete any component whose weight is below a certain value; all attributes that are not \refreshed" by the user will be set aside. The updated user model is then sent to HUMOS in order to carry out the modeling process activities.

The insertion of new components into the model is performed by the Feedback Unit according to our proposed algorithm SAF (Semantic net/DB-based Algorithm for Feedback). If the system nds an unknown keyword k in a document, it rst uses the Terms Data Base (TDB) to nd the meaning of the keyword. The TDB is structured to ease this task and evolves dynamically. If k is already in the TDB, then the model is updated by inserting k 's components, already known by the system: this dynamically broadens the semantic aspects of the model and its inferential capabilities as well. If k does not have a value in

Fig. 6. The interface of the system. the TDB, a Semantic Network is called up, the structure of which has a central node representing a potential topic of user interests and a set of satellite nodes representing keywords which co-occur in the same document. In this case the unknown keyword k is inserted as a \co-keyword" in the model and, by using the wheighted semantic links present in the Semantic Network, k is connected to the model components found in the document. This enables the system to distinguish between dierent meanings of a word by the context in which it occurs, hence dynamically widening the semantic potential of the user model and permitting a more accurate ltering.

4 Empirical Evaluation Generally speaking, the aim of an empirical evaluation is a careful observation in informative conditions, probing results, forming interpretations of data, and testing these interpretations prospectively [5]. Following the above general guidelines, we have performed the evaluation activity of our system through real-time access to the World Wide Web. During the tests, four (randomly chosen) users have searched various kinds of information concerning their area of interest on the Web. Each user started with the preliminary interview managed by the system. After that HUMOS carried out the modeling process activities to create the initial model. Next, each user input

15 queries, personally analyzed all ltered documents and gave a relevance feedback (value between -10 and +10) to all of them. After each ltering process we have obtained the relevance ordering of the collection and calculated the normalized precision and recall [25] for both relevant and irrelevant documents. In order to evaluate and compare the performance of our information ltering process with standard information retrieval techniques, we have disabled HUMOS and, as a consequence, all WIFS algorithms (obtaining AltaVista behaviour). In this second case the precision and the recall have been evaluated working on the same collections gathered during the previous tests. The classical statistical parameters on the sample (the 60 queries) have been calculated (average, standard deviation, etc.) and the dierence between WIFS and AltaVista samples have been evaluated by using a non-parametric (or distribution-free ) statistics (the Wilcoxon Signed-Rank test [26]; the reader is referred to [6] [16] for more detail concerning distribution-free statistics). Finally, the analysis of the statistical results have been performed to draw the statistical conclusion and the research conclusion. The null hypothesis H0 in our experiment is the following: \There is no dierence in performance between WIFS and AltaVista, measured in terms of precision and recall (i.e., the statistical populations concerning precision and recall are the same)". The alternative hypothesis H1 is: \The dierences observed between the distributions of the precision and the recall are not due to chance but are due to the dierence between the populations to which they belong". The results of the Wilcoxon test are shown in Table 1. In the rst row, the numerical values for the Wilcoxon T+ parameter, for precision and recall are shown. They are very high values in the context of a Wilcoxon test. In the second row, the calculated probability for both precision and recall, is very low (about 10,4), less than the signi cance level  = 0:01 we have chosen. Therefore, we can conclude that, in both cases, the null hypothesis H0 can be rejected and the alternative hypothesis H1 can be accepted. This means that the dierences observed from the sample sets are not due to chance, but to dierent underlying distributions to which they belong. On the basis of H1, the WIFS precision and recall parameters have distribution shifted towards a higher mean with respect to the corresponding AltaVista parameter. In Table 2 some useful statistical parameters, calculated directly on the distribution, are shown. For instance, the mean of the precision of WIFS (Pw ) is 0.71 while the precision of AltaVista (Pa) is 0.52. The added value given by our User Modeling system is apparent. We can also see that the means and the medians are quite similar: this can be interpreted as a possible symmetry in the distributions. These parameters are useful to perform inferences on the con dence intervals [6] of the means. As a further computation, we have determined these con dence intervals, shown in Table 3. For brevity's sake, we do not describe the details of these computations. Suce to say that, starting from the computed con dence intervals, Pw is higher than Pa from 4% up to 35%, while Rw is higher than Ra from 14% up to 37%. The above statistical results support our choice of using a case-based approach to user modeling for adaptive systems.

Table 1. Wilcoxon Test Results Precision Recall T+ 1621.5 1755.5 Pr (T+ > c) 10,4 10,4 H0 rejected rejected H1 accepted accepted

Table 2. Statistical Parameters

Pw Pa Pw , Pa Rw Ra Rw , Ra

Mean 0.71 0.52 St. Dev. 0.23 0.25 Variance 0.05 0.06 Mediane 0.75 0.50

0.19 0.87 0.61 -0.03 0.13 0.22 -0.01 0.02 0.03 0.25 0.90 0.61

0.26 -0.09 -0.03 0.29

Table 3. Con dence Intervals of the Mean 99% inf. 99% sup. 95% inf. 95% sup. Pw 0.64 0.79 0.65 0.77 Pa 0.44 0.60 0.46 0.58 Rw 0.82 0.91 0.83 0.90 Ra 0.54 0.68 0.55 0.66

5 Conclusions In this paper we have described a Case-based approach to the construction of a general User Modeling shell system, capable of building a representation of user needs and characteristics (User Model). Our shell system is based on a hybrid arcitecture, where an arti cial neural network is integrated into a case-based reasoner. One advantage of this architecture is the inherent fault tolerance to noice in data representing user behavior, which allows the system to \gracefully degrade". A case study in Information Filtering has also been presented and evaluated. The experiments have shown that, thanks to the case-based User Modeling component, our Information Filtering system improves the capabilities of AltaVista by more than 30%. In our work, we have turned to statistics to analyze the system behaviour, and demonstrated that the system performance \is not due to chance". A more extensive test of the system has been planned as a future work. We have also planned to develop and evaluate further features with the goal of improving the performance of both modeling and ltering processes. As for the modeling process, we are improving the modeling capabilities of the users by using a dynamic updating process of the user model. As far as the ltering process is concerned, we are integrating the query modality with the sur ng modality to obtain a system able to autonomously retrieve and lter documents.

References 1. Ambrosini, L., Cirillo, V. and Micarelli, A.: A Hybrid Architecture for UserAdapted Information Filtering on the World Wide Web. Proc. of The 6th International Conference on User Modeling UM-97, Chia Laguna, Sardinia, Italy, June 1997. 2. Ambrosini, L., Cirillo, V. and Micarelli, A.: A User-Adapted Interface for a Search Engine on the World Wide Web. Proc. of WebNet 97 - World Conference of the WWW, Internet and Intranet, Montreal, Canada, November 1997. 3. Belkin, N.J., and Croft, W.B.: Information Filtering and Information Retrieval: Two sides of the Same Coin?. Communications of the ACM, 35(12), 1992, pp. 29-38. 4. Brainik, G. and Tasso, C.: A shell for developing non-monotonic user modeling systems. International Journal of Human-Computer Studies, 40, 1994, pp. 31-62. 5. Cohen, P.R.: Empirical Methods for Arti cial Intelligence. MIT Press, Cambridge, Massachusetts, 1995. 6. Devore, J.L.: Probability and Statistics for Engineering and the Sciences. Brooks/Cole Publishing Company, Monterey, California, fourth edition, 1995. 7. Fu, L.M.: Neural Networks in Computer Intelligence. McGraw-Hill, 1994. 8. Goonatilake, S. and Khebbal, S. (eds.): Intelligent Hybrid Systems. John Wiley & Sons, 1995. 9. Hoerer, M., Knaus, B. and Winiwarter, W.: An Evolutionary Approach to Cognitive Information Filtering. Proc. of The 18th International ACM SIGIR Conference on Research and Development in Information Retrieval, Seattle, WA, USA, July 1995, pp. 1-15.

10. Kirkpatrick, S., Gelatt, C.D. and Vecchi, M.P.: Optimization by simulated annealing. Science, 220, 1983, pp. 671-680. 11. Kobsa, A. and Wahlster, W. (eds.): User Models in Dialog Systems. SpringerVerlag, 1989. 12. Kobsa, A. and Pohl, W.: The User Modeling Shell System BGP-MS. User Modeling and User-Adapted Interaction, 4(2), 1995, pp. 59-106. 13. Malek, M.: A Connectionist Indexing Approach for CBR Systems. In: M. Veloso and A. Aamodt (eds.) Case-Based Reasoning - Research and Development, Proc. of ICCBR-95, Lecture Notes in Arti cial Intelligence, 1010, Springer-Verlag, 1995, pp. 520-527. 14. Micarelli, A. and Sciarrone, F.: A Case-Based Toolbox for Guided Hypermedia Navigation. Proc. of The 5th International Conference on User Modeling UM-96, Kailua-Kona, Hawaii, USA, January 1996, pp. 129-136. 15. Micarelli, A. and Sciarrone, F.: A Case-Based System for Adaptive Hypermedia Navigation. In: I. Smith and B. Faltings (eds.) Advances in Case-Based Reasoning, Proc. of EWCBR-96, Lecture Notes in Arti cial Intelligence, 1168, SpringerVerlag, Berlin, 1996, pp. 266-279. 16. Minium, E., King, B.M. and Bear, G.: Statistical Reasoning in Psycology and Education. John Wiley and Sons, third edition, 1993. 17. Moukas, A.: Amalthaea: Information Discovery and Filtering Using a Multiagent Evolving Ecosystem. Applied Arti cial Intelligence, 11, 1997, pp. 437-457. 18. Mukhopadhyay, S., Mostafa, J., Palakal, M., Lam, W., Xue, L. and Hudli, A.: An Adaptive Multi-level Information Filtering System. Proc. of The 5th International Conference on User Modeling UM-96, Kailua-Kona, Hawaii, USA, January 1996, pp. 21-28. 19. Naughton, S.: The Application of Neural Network Paradigms to the Development of a Generic Case Retrieval Engine. Final Year Project Report, Department of Computer Science, Trinity College Dublin, Ireland, 1994. 20. Papagni, M., Cirillo, V. and Micarelli, A.: A Hybrid Architecture for a UserAdapted Training System. Proc. of The 5th German Workshop on Case-Based Reasoning, Bad Honnef, Germany, March 1997. 21. Papagni, M., Cirillo, V. and Micarelli, A.: Ocram-CBR: A Shell for Case-Based Educational Systems. In: D.B. Leake and E. Plaza (eds.) Case-Based Reasoning - Research and Development, Proc. of ICCBR-97, Lecture Notes in Arti cial Intelligence, 1266, Springer-Verlag, Berlin, 1997, pp. 104-113. 22. Papagni, M., Cirillo, V., Micarelli, A. and Boylan, P.: Teaching Through CaseBased Reasoning: An ITS Engine Applied to Business Communication. Proc. of The 8th World Conference on Arti cial Intelligence in Education AI-ED 97, Kobe, Japan, August 1997, pp. 111-118. 23. Rich, E.: User are Individuals: Individualizing User Models. International Journal of Man-Machine Studies, 18, 1983, pp. 199-214. 24. Rumelhart, D.E. and McClelland, J.L. (eds.): Parallel Distributed Processing. MIT Press, Cambridge, Massachusetts, 1986. 25. Salton, G. and McGill, M.J.: Introduction to Modern Information Retrieval. McGraw-Hill Publishing Company, NewYork, NY, 1983. 26. Wilcoxon, F.: Probability Tables for Individual Comparisons by Ranking Methods. Biometrics, 3, 1947, pp. 119-122.