Multi-Attribute Utility Theory and Adaptive

Kabassi, K. & Virvou, M. (2006). Multi-Attribute Utility Theory and Adaptive Techniques for Intelligent Web-Based Educational Software. Instructional Science, 34(2), pp. 313-158.

Multi-Attribute Utility Theory and Adaptive Techniques for Intelligent Web-Based Educational Software Katerina Kabassi & Maria Virvou

Abstract This paper describes how the Multi-Attribute Utility Theory can be combined with adaptive techniques to improve individualised teaching in an Intelligent Learning Environment (ILE). The ILE is called Web F-SMILE, it operates over the Web and is meant to help novice users learn basic skills of computer use. Tutoring is dynamically adapted to the individual learner based on the learner modelling component of the system and the Multi-Attribute Utility Theory (MAUT) that is employed to process the information about the user. As a result, MAUT provides a way for the system to select on the fly the best possible advice to be presented to users. Advice is dynamically formed based on adaptive presentation techniques, where adaptation is performed at the content level and adaptive navigation support, which is performed at the link level of the hyperspace of the tutoring system. The adaptivity of learning depends on factors such as the learner's habits, prior knowledge and skills, which are used as criteria for the application of MAUT in the educational software. In this way, a novel combination of MAUT with adaptive techniques is used for intelligent web-based tutoring. 1. Introduction The enormous growth of software technology has affected almost every profession and many aspects of today's everyday life. Therefore, software technology skills have become a vital qualification for many people. However, the possibility of attending a formal training course or educational class is not feasible for everyone. An excellent solution for this problem is provided by the Internet and the WWW. Indeed, learning through the Web can take place anywhere, at any time, through any computer and without necessarily the presence of a human tutor. The advantages that WWW may offer to education have become a great influence on computer assisted learning which is now turning into Web-based learning. Although learning through the Web addresses a wide variety of users with diverse background knowledge, still the majority of Web-based educational applications are rather static and represent a generic approach to tutoring that does not take into account the individual needs of each student. This problem may be addressed by educational software technologies such as Intelligent Tutoring Systems (ITSs) and Intelligent Learning Environments (ILEs) that have been particularly effective at personalising tutoring. Indeed, one aim of work in ITSs and ILEs is to provide each student with a learning experience similar to the ideal one-to-one tutoring. In order to achieve this goal, ITSs and ILEs try to present the teaching material in a flexible way based on Artificial Intelligence techniques. Intelligence and adaptivity in ITSs and ILEs are achieved by the incorporation of a learner modelling component. Learner modelling involves the construction of a qualitative representation that accounts for student behaviour in terms of existing background knowledge about a domain and about students learning the domain (Sison & Simura, 1998). Such a representation, called a learner model, can assist an ITS, an ILE, or an intelligent collaborative learner in adapting to specific aspects of student behaviour (McCalla, 1992).

1

Kabassi, K. & Virvou, M. (2006). Multi-Attribute Utility Theory and Adaptive Techniques for Intelligent Web-Based Educational Software. Instructional Science, 34(2), pp. 313-158. As a consequence of the indisputable virtues that the technology of ITSs and ILEs may offer to Web-based education, recently a lot of research energy has been put in this area (e.g. De Bra et al. 2002). However, in the case of Web-based ITSs and ILEs the problem of navigation by students has also to be addressed. The contents of the teaching material should be presented to students in a way that depends on their own level of knowledge, so that a wider range of users of multiple backgrounds may be accommodated. Thus, a page which is appropriate for novice users should not be presented to advanced users who may find it boring and trivial. A solution to this problem may be given by the use of adaptive hypermedia methods and techniques. These techniques may be used to inform readers that certain links lead to material that the readers are not ready for, or even to compensate for missing knowledge by adding explanations to the pages a reader visits (De Bra 2002). Indeed, educational systems such as InterBook (Brusilovsky et al., 1998) and AHA! (De Bra et al. 2002) use adaptive hypermedia to guide users through the teaching material. Without such help users can "get lost" even in reasonably small hyperspaces, or may use inefficient browsing strategies (Hammond, 1989). Adaptive hypermedia techniques have been evaluated and the results offer strong evidence that their use in an Adaptive Hypermedia System can improve human-computer interaction as they help users navigate through the teaching material and does not present them irrelevant information (e.g. Murray et al. 2000, Brusilovsky & Maybury 2002). In view of the above, we have developed a web-based ILE that incorporates a user modelling component and uses both adaptive presentation and navigation support to tailor the information presented to a user. The web-based ILE is called Web F-SMILE (File-Store Manipulation Intelligent Learning Environment) and aims at teaching users how to use the Windows operating system. The fact that Web F-SMILE is Web-based provides an opportunity to a wide range of users, to be trained on basic computer skills. At the same time the ILE that makes use of adaptive techniques can ensure high interactivity, adaptivity and individualisation. The system uses the information about each user that is stored in the Learner Modelling Server, in order to adapt the tutoring process to the specific user. The main body of the paper is organised as follows: Section 2 presents and discusses related work. In Section 3 we give an overall description of Web F-SMILE together with a short example of its operation. This example is used in all the subsequent sections in order to show how the reasoning of Web F-SMILE is generated. In Section 4 and 5 we describe the learner modelling component of Web F-SMILE and the criteria that are used for this purpose. Section 6 gives the exact calculations that are needed based on MAUT theory for the system’s tutoring agent to produce advice tailored to a particular user. Section 7 presents how MAUT has been used in accordance with the results of an empirical study for the calculation of the weights of the criteria that are taken into account for the generation of advice. Finally, in Section 8 we discuss the conclusions drawn from this work. 2. Related Work The effectiveness of an ILE depends heavily on the system’s ability to make good decisions about when and what the learner should learn. The key to effective assistance in learning with an ITS or ILE is learner modelling. Learner modelling is an important task but has been recognised as complex and difficult and often requires sophisticated reasoning techniques. In the literature of ITSs and ILEs, several AI methods and approaches have been proposed in order to serve the learner modelling procedure. Special emphasis has been given to numerical techniques for decision making. Indeed, Jameson (1996) in a review of the techniques for the uncertainty management presents among others the systems that have used the Bayesian Networks and the Dempster-Shafer theory of evidence.

2

Kabassi, K. & Virvou, M. (2006). Multi-Attribute Utility Theory and Adaptive Techniques for Intelligent Web-Based Educational Software. Instructional Science, 34(2), pp. 313-158. Bayesian Networks can be applied to user modelling in many different ways. More specifically, they have been used to assess a user’s knowledge (Desmarais et al. 1995), help him/her during problem solving (Conati & VanLehn 1996; Conati et al. 2002), recognise the plans of users (Pynadath & Wellman 1995, Huber et al. 1994) or predict users’ responses (Jameson et al. 1995, Van Mulken 1996, Horvitz & Barry 1995). A main disadvantage of the use of Bayesian networks in a tutoring system is that the variables involved in the decision should be initialised. Such a problem can be addressed by the Dempster-Shafer theory of evidence (see for example (Bauer 1995)). However, the main problem with the use of the Dempster-Shafer theory is that the decision making process is more complex than it is when a Bayesian Network is used. On the other hand, Bayesian inference has often been viewed as antithetical to human reasoning (Jameson 1996). In an effort to simulate the reasoning of a human advisor that watches the user that interacts with a learning environment we exploited the utility of Multi-Criteria Analysis. Furthermore, we address the problem of initialisation of the criteria values by combining a multi-criteria theory with stereotypes. In a first attempt, we adapted Simple Additive Weighting (SAW) (Fishburn, 1967; Hwang &Yoon, 1981) in a learning environment called Web-IT (Kabassi & Virvou 2004). In that system SAW was used for modelling adult learners of different age groups. However, the SAW method relies on the results of an empirical study where experts are asked to rank the criteria they use in order of importance. However, usually in tutoring systems experts are not able to specify the importance of each criterion they use to form advice. In this respect, MAUT seems to provide a better solution since it requires an empirical study where experts are asked to specify equivalent situations rather than give a ranking for the criteria they use. This procedure seems more natural to human expert-tutors, thus the results of this kind of empirical study are more reliable. As a result in Web F-SMILE, we have applied MAUT for the reasoning of a tutoring agent. However, the criteria that were used in Web-IT did not seem appropriate for Web F-SMILE. Therefore, we conducted an empirical study in order to identify the criteria that are appropriate for modelling all learners irrespective of their age. MAUT is an evaluation scheme that has been used by many systems for evaluating the users’ interests and preferences. Therefore, MAUT has been extensively used in application areas such as ecommerce. More specifically, within the CAWICOMS project (Ardissono et al. 2001, Schütz & Schäfer 1997), MAUT has been ascribed as an evaluation process in order to determine the interests of a customer-user and support him/her in configuring the desired product. A quite different approach is used by Matsuo & Ito (2002) and Kudenko et al. (2003) in product recommendation systems. There, MAUT is used to quantify not just one user’s but several users’ preferences while planning a joint purchase. The values of the criteria in such systems for the evaluated object are always pre-fixed and do not usually change over time. In an e-learning system, on the other hand, the information about the learner changes as the learning process proceeds and the user learns more about the domain being taught. A more dynamic approach of the application of MAUT in personalised e-commerce is described in TravelPlanner (Chin & Porage 2001). This system uses MAUT in combination with stereotypes to evaluate the available travelling opportunities and proposes the user the one that fits best his/her needs and preferences. However, TravelPlanner is based on stereotypical knowledge and not on an individual user model. The main drawback of such an approach is that the users are not easily grouped in stereotypes and may change their characteristics over time. In Web F-SMILE we use stereotypes only for the initialisation of user models when there is not yet sufficient information collected for the individual users. Then, in contrast with TravelPlaner, Web F-SMILE uses information from individual learner models. Furthermore, an important difference of all the approaches for e-commerce that were described above, with the one adopted in Web F-SMILE is that the above mentioned systems try to simulate the 3

Kabassi, K. & Virvou, M. (2006). Multi-Attribute Utility Theory and Adaptive Techniques for Intelligent Web-Based Educational Software. Instructional Science, 34(2), pp. 313-158. reasoning of the users interacting with them. In Web-SMILE, on the other hand, MAUT is used to simulate the reasoning of human tutors who watch the learner while interacting with a system and make decisions about what the user should learn further in order to achieve his/her goals. This difference derives from the difference in the application area. In Web-based learning, a human tutor can usually evaluate better the learner’s needs than the learner himself/herself. This is so, because the learner is not always capable of deciding what s/he needs in order to achieve his/her goals and plans. On the other hand, in cases such as travel planning the user knows better what fits his/her needs and preferences. 3. Web F-SMILE Web F-SMILE (Web File-Store Manipulation Intelligent Learning Environment) is an intelligent learning environment for novice users of a GUI (Graphical User Interface). Web F-SMILE allows users to manipulate their file-store in a similar way as the Windows 98/NT Explorer (Microsoft Corporation, 1998). Web F-SMILE is meant to help users during their navigation and manipulation of the file-store and provide adaptive tutoring in case this is considered necessary. In general, the system tries to act as a human expert (tutor) that helps users while interacting with the computer at work, at school or at home and constantly helps them by presenting them the pieces of knowledge that they are not aware of and that they need in order to accomplish their goals. The domain that has been selected for teaching concerns the main concepts of the Windows operating system. More specifically, the students learn everything about folders and files, file extensions, main commands in graphical user interfaces and command language interfaces as well as more complex concepts such as the way files and folders are physically stored in the hard disk, the way they are organised by the operating system etc. Furthermore, the users are familiarised with sending and receiving electronic mail as well as organising their electronic mailbox. Every time a user issues an action, Web F-SMILE reasons about it in terms of the system’s expectations about the user’s recognised goals. In case this action contradicts the system’s expectations, it tries to identify the user’s misconception and provide adaptive tutoring. Web FSMILE uses adaptive hypermedia techniques to protect learners from information overflow and to help them understand new pieces of knowledge that are being taught. The two main adaptive hypermedia techniques that exist are: (i) adaptive presentation, where adaptation is performed at the content level and (ii) adaptive navigation support, which is performed at the link level (Brusilovsky, 1996). In Web F-SMILE, both these techniques are used and the adaptivity of learning depends on factors such as the learner's habits, prior knowledge and skills. In particular, adaptive presentation techniques are used to present examples of use of an unknown command in the context of the learner’s own file-store. Therefore, the system generates examples dynamically so that it may use the names of the particular learner’s existing files and folders. More specifically, Web F-SMILE selects the names of files and/or folders that are frequently used by the learner to make sure that the learner is very familiar with them. Moreover, Web F-SMILE uses different font types and icons to provide adaptive navigation support. For this reason, it uses adaptive link annotation techniques to present other parts of knowledge that are believed to be of interest to the learner for the particular case. The idea of adaptive annotation is to augment links with some form of comments, which can tell the user more about the current state of the nodes behind the annotated links. In Web F-SMILE, whenever a link appears on a page, the font type and the icon that appears in front of the link are annotated so as to reflect the state of the topic or exercise behind the link, with

4

Kabassi, K. & Virvou, M. (2006). Multi-Attribute Utility Theory and Adaptive Techniques for Intelligent Web-Based Educational Software. Instructional Science, 34(2), pp. 313-158. respect to the student’s current knowledge state. The meaning of different font types and icons are presented in the table below. Examples of links are illustrated in Figure 1. Icon

Fonts Bold – 12 points Regular – 12 points

Italics – 12 points

Regular – 10 points

Meaning This kind of link leads to theory topics that the user should study in order to achieve his/her goals and complete his/her plans. This kind of link leads to theory topics that the user has already visited but has not learnt. Therefore, the system proposes him/her to revise them. The user has studied the particular theory topic and has executed successfully the commands that are related to it. Therefore, the system informs the user that it is not necessary to revise this theory topic. The user does not know basic theory topics that are prerequisite for studying the particular theory topic. Therefore, the system warn the user that s/he is probably not ready for studying this particular theory topic but does not forbid him/her to enter this theory topic.

A simple example of the system’s operation taken from a real interaction of a user with Web FSMILE is presented in figure 2. The user of our example is novice in the usage of computers and file manipulation programs. A task of this particular user is presented in this example. The user's initial file-store state is shown in figure 1.

Figure 1: The Learner’s Initial File-Store State

5

Kabassi, K. & Virvou, M. (2006). Multi-Attribute Utility Theory and Adaptive Techniques for Intelligent Web-Based Educational Software. Instructional Science, 34(2), pp. 313-158.

Figure 2: A Sample Screen for the Tutoring of ‘Moving Objects’ The user wants to move a part of the contents of a floppy disk that was given to him. In order to achieve his goal, the user executes the command cut(A:\proposals\), which is the first step for moving objects in the file-store. After completing this action, he selects the folder ‘C:\My Documents\’ and tries to move the objects that he has cut. However, he issues the command ‘copy(C:\My Documents\)’, rather than ‘paste(C:\My Documents\)’ which was the appropriate command for his plan. Obviously, the user mistakenly believed that copy was the right command in order to complete his plan of moving the folder ‘proposals’ from floppy disk A:\ to the folder ‘My Documents’ on hard disk C:\. Web F-SMILE finds the particular action as ‘Unintended’ and tries to diagnose the user’s misconception. The system after applying the formulae that have been obtained from the application of MAUT concludes that the user does not know how to move files or folders in the file-store. Therefore, the user is presented with a lesson that involves the parts of the theory ‘Moving Objects in the file-store’ and ‘Moving Folders’ and he is advised to read more details on the topics ‘Moving Files’ and ‘Copying Objects’ (Figure 1). Furthermore, the user is provided with an example about moving a folder, which is adapted to the user’s own file-store. More specifically, the user is shown how to move the folder ‘proposals’ from floppy disk A:\ to the folder C:\My Documents\. More information about the system’s reasoning and how it draws inferences about the user are given in the next sections. After the user has completed the particular task, he executes some more commands (e.g he creates a folder called ‘Temporary’ in C:\My Documents\ and fills it with some content by copying objects

6

Kabassi, K. & Virvou, M. (2006). Multi-Attribute Utility Theory and Adaptive Techniques for Intelligent Web-Based Educational Software. Instructional Science, 34(2), pp. 313-158. successfully, etc). Then the user tries to move the file ‘plans’ from A:\ to the folder C:\My Documents\. However, instead of selecting the object and issuing the command cut, he first issues the command copy and then the command delete. However, he issued the command delete without having selected any objects. Taking into account this fact and the information that is stored in the user model, Web F-SMILE concludes that the user does not know how to delete objects and that he should revise how to move files or folders in the file-store. Furthermore, the information provided by the user model indicates that the user has now learned how to copy objects and, thus, the particular sections are marked as read and comprehended. The user is presented with a lesson that involves the parts of the theory ‘Delete Objects’ and he is advised to revise the topics ‘Moving Files’, ‘Moving Folders’ and ‘Moving Objects’ (Figure 3). Furthermore, as one will notice the example presented to the user in the second case is different from the one presented in the first sample screen, despite the fact that the theory topic is the same. This is because the file store of the user has changed and the examples are always adapted to the user’s own file store state.

Figure 3: A Sample Screen for the Tutoring of ‘Moving Objects’ after the user has tried to move some objects 4. Learner Modelling in Web F-SMILE Every time the learner interacts with Web F-SMILE, the system collects new information about the user and updates the learner model of the particular learner. In case the system cannot locate a learner model for a particular student, it tries to initialise the learner model using stereotypes. User

7

Kabassi, K. & Virvou, M. (2006). Multi-Attribute Utility Theory and Adaptive Techniques for Intelligent Web-Based Educational Software. Instructional Science, 34(2), pp. 313-158. stereotypes are employed in order to provide default assumptions about users until systems have acquired sufficient information about each individual user. Indeed as Rich (1989; 1999) points out, a stereotype represents information that enables the system to make a large number of plausible inferences on the basis of a substantially smaller number of observations; these inferences must, however, be treated as defaults, which can be overridden by specific observations. Stereotypes constitute a powerful mechanism for building user models (Kay 2000) and are widely used in ITSs and ILEs (for example (Huang et al. 1991), (Murphy and Mc Tear 1997)). In Web F-SMILE, users are classified into one of three major classes according to their level of expertise, namely, novice, intermediate and advanced. Each one of these classes represents an increasing mastery in the use of the particular file-store manipulation system. Such a classification is considered important because it enables the system to have a first view of the usual errors and misconceptions of a user, belonging to a certain class. For example, novice users are usually prone to mistakes due to erroneous command selection or erroneous command execution whereas expert users usually make mistakes due to carelessness. Therefore, another classification that was considered important was dividing users into two groups, careless and careful. In Web F-SMILE, all default assumptions in stereotypes give information about the values of the criteria that are used in order to evaluate each theory topic. The stereotypes that are related to the user’s level of expertise give information about the values of the following criteria: • frequency of an error (f), • percentage of the correct executions of a command in the total executions of the particular command (e) • and the number of times that the user has visited each part of the theory (v). For example, a novice stereotype gives the value 0.7 to the criterion f for errors due to lack of knowledge whereas the same criterion for the same kind of error take just the value 0.4 in the case that the intermediate stereotype has been activated. Accordingly, the percentage of the correct executions of a command in the total executions of that command takes its highest values in the case of the expert stereotype. The stereotypes that are related to the user’s carelessness give information only about the criterion degree of carelessness (c). A definition of these criteria is provided in the next section. According to the stereotypes that have been activated, the criteria take different values and as a result the system makes different decisions that are adapted to each user. For example, the criterion c takes the value 0.9 for the careless stereotype and the value 0.3 for the careful stereotype. Although the application of stereotypes can prove rather effective due to the similar behaviour that users of the same class may have, every one is an individual that differs from all the others in many aspects. Therefore, stereotypes in Web F-SMILE have only been used for initialising the user model, until there is more information about each individual user. More specifically, in case the user has not interacted with the system for a sufficient period of time the user model does not have adequate information about the particular learner. Therefore, the learner model’s information is acquired by the stereotypes that have been activated for the particular user. However, the system is also constantly collecting information about a particular user’s behaviour and errors and informs the individual user model of the user. The information that is stored in the individual user model is used for calculating the values of the criteria that take part in the decision making process which is based on MAUT. The information in the individual user model is updated using evidence about the cognitive state of a user that is collected during his/her interactions with the system. As all tests were carried out in computer labs, an interaction is defined as the time between the user logs on and off the system and is not connected to the number of commands issued by a user. Therefore, a user may have one or many 8

Kabassi, K. & Virvou, M. (2006). Multi-Attribute Utility Theory and Adaptive Techniques for Intelligent Web-Based Educational Software. Instructional Science, 34(2), pp. 313-158. different interactions in a day. After 20 interactions of the user with the system, information is acquired 50% by the stereotypes and 50% by the individual user model. The stereotypes are deactivated after the user has completed 40 interactions with the system. 5. Definition of the criteria Since the goal of Web F-SMILE is to simulate the reasoning of a human decision maker that behaves as a trainer, we conducted an empirical study (Virvou & Kabassi 2003) which aimed at investigating the reasoning of human decision makers in terms of the decision problems of Web FSMILE. The empirical study should involve a satisfactory number of human experts, who represent the decision makers. Therefore, 10 human experts were asked about the criteria that they take into account while making decisions about what the students should learn. All the human experts that were selected possessed a first and/or higher degree in Computer Science and had teaching experience related to the use of file manipulation programs. All human experts were shown some interactions of real users with a standard Explorer and were asked to state the kind of advice they would give to the user. As soon as this process was completed the human experts were interviewed about the criteria that they take into account when providing individualised advice. From the criteria that appeared in this phase of the experiment, only those proposed by the majority of the human experts were selected. As a result of this process, the attributes that most human tutors take into account in order to provide personalised help and tutoring were defined as follows: Frequency of an error (f). The value of this criterion shows how often a user makes a particular type of error in the total errors that s/he makes. This is calculated from the information stored in the learner model. In the beginning, this value is acquired by the stereotype that the user has been classified into (novice, intermediate or advanced), as the system does not have adequate information about the user yet. In particular, the system searches the stereotype of the learner in order to acquire a value for the frequency of errors due to lack of knowledge for each category of learners. However, if the system has managed to collect information about the particular learner, the value of the criterion is acquired by the information stored in the individual learner model. Percentage of the correct executions of a command in the total executions of the particular command (e). A command is considered to be correct if it is compatible to the user interface’s formalities as well as to the user’s goals and plans. For example, if the percentage of the correct executions (e) of a command is 0.96 then the user is expected to have a good knowledge of the command. The value of this criterion is acquired by the stereotype of novice, intermediate or advanced learners, until the system manages to collect more information about the particular learner. As soon as the information in the individual learner model is adequate, the value of the criterion is estimated based on the facts that have been collected for the particular user and is stored in the individual learner model. Degree of carelessness (c). If a user is very careless and makes many errors then there is a possibility that these errors are not due to lack of knowledge but due to carelessness. In such case, it would not be wise for the system to advise the user to read any part of the theory since the user already has the required knowledge but has made a mistake due to carelessness. The value of this criterion is dynamically calculated by dividing the number of errors that the particular user makes due to carelessness with the total number of user’s errors. In case, however, the individual user model does not have adequate information about this fact, this degree is acquired by the careless or careful stereotype. 9

Kabassi, K. & Virvou, M. (2006). Multi-Attribute Utility Theory and Adaptive Techniques for Intelligent Web-Based Educational Software. Instructional Science, 34(2), pp. 313-158. The number of times that the user has visited each part of the theory (v). If a user has visited a part of theory many times, it is likely that s/he knows the commands connected with that part. The value of this criterion is also acquired by the individual learner model or the novice, intermediate and advance stereotype accordingly, depending on the sufficiency of the information stored in the individual learner model. Degree of relevance (r). This criterion shows how relevant the action that the user intended to issue is with respect to the action explained in a particular piece of the theory. Its value is calculated dynamically based on the similarity of the actions and does not require information from the learner model. The relevance of two commands is static and is pre – calculated. Its value is based on the relative position of the command in the hierarchy of users’ actions that is stored in the domain representation component of the system. Two commands that are neighbouring in the hierarchy of users’ actions have a high degree of relevance; for example the commands cut and copy. In addition to their relative distance, the effects of the commands have also been taken into account. For example, cut and copy commands have a similar effect; they place one or more objects into the clipboard, so they have a great relevance. Moreover, it has been observed that novice users tend to entangle two commands when these are neighbouring on the screen. So the relevance of two commands also depends on their relative distance on the screen. Finally, the relevance of two objects is dynamically calculated as the items of the file store are constantly changing. The value of relevance between two objects depends on their relative position in the file store, as this is displayed in the screen. Moreover, the similarity of their names is also taken into account. For example, files document1.doc and document2.doc have a high degree of relevance. As soon as the final set of criteria was selected they were presented to all human experts, in order to review them. Most of the human experts agreed that the particular set was satisfactory in modelling their reasoning process. Furthermore, the experts were asked if these criteria were only used when they provided advice in a domain related to Computer Science. According to most human experts, the only criterion that seemed to be domain dependent was the one that corresponded to the number of correct executions of a command (e). If this procedure was to be applied in a different domain, such as Mathematics, this criterion should represent the exercises that were related to a theory topic and the user had successfully completed. The rest of the criteria were domain independent and, therefore, the results of the above mentioned empirical study could be used with a small adjustment of criterion e in a different domain, as well. For the user of the example in Section 3, the individual learner model had adequate information for the calculation of the degrees described above. In view of this information, 26% of the user’s errors were connected with copying objects ( f = 0.26 ). This percentage was even greater for moving objects ( f = 0.39 ). Furthermore, the user had made some additional errors that concerned these commands. He had made 3 carelessness errors while copying objects and 6 while moving objects. Therefore, the degree of carelessness was 0.08 and 0.16 for copying and moving objects, respectively. Some additional information about the degree of knowledge of each theory topic can be acquired from the times that the particular user had visited each part of the theory and the percentage of correct executions of each command. The user of the example had visited both theory topics of copying and moving objects just once ( v = 1 ). Furthermore, only 29% of his trials to copy an object ( e = 0.29 ) succeeded whereas the corresponding percentage of success for moving objects was 21% ( e = 0.21 ).

10

Kabassi, K. & Virvou, M. (2006). Multi-Attribute Utility Theory and Adaptive Techniques for Intelligent Web-Based Educational Software. Instructional Science, 34(2), pp. 313-158. The degree of relevance for the topic of ‘Moving Objects’ and ‘Moving Folders’ is 0.90 whereas the respective degree for the theory topics of ‘Moving Files’ and ‘Copying Objects’ is 0.70. The exact way that these criteria are combined through MAUT is explained in the next section. 6. MAUT for Tutoring

It is among the goals of Web F-SMILE to provide to the user the ‘right’ pieces of information in the ‘right’ way. For this purpose, Web F-SMILE should be able to make decisions of what the user’s intentions were and which parts of knowledge s/he needs in order to achieve his/her goals and plans. More specifically, the decision problem in Web F-SMILE consists of two parts. The first part is to find an action that the user must have really intended instead of the problematic one issued. The second part is to find the theory topic that is most appropriate to be presented to a user so that s/he may acquire the piece of knowledge needed for him/her to achieve his/her initial goals. This problem is addressed in the following way: First, the system generates an action other than the one issued by the user, which was problematic. Then, the system relates the generated action with a part of the theory that the user should read so that s/he acquires the pieces of knowledge that s/he needed for the use of this intended action. Thus the decision problem in Web F-SMILE has been considered to consist of the calculation of two degrees. First, what we call, the degree of certainty dcertainty shows how certain the system is that the user actually intended the action that has been generated by the system as more appropriate for the user’s goals than the action issued by the user. Second, what we call, the degree of knowledge dknowledge represents the level of knowledge of a user with respect to each part of the theory that could be relevant to the particular task that the user had undertaken when a problem occurred. The calculation of each of these two degrees is based on the criteria that were presented in the previous section. More specifically, the criteria that are taken into account for the calculation of the degree of certainty dcertainty are the degree of relevance of the generated action with a particular piece of the theory (r), the frequency of the errors of a user connected with the particular command (f) and the degree of carelessness of the user (c). The criteria that are taken into account for the calculation of the degree of knowledge dknowledge are the number of correct executions of a command (e) by the user and the times the user has visited each part of the theory (v). In order to select what to present to the user, the system calculates the degree of certainty dcertainty and the degree of knowledge dknowledge for every action that the system generates in order to capture the user’s real intention and then calculates the degree of a tutoring need on a topic. The higher the degree of certainty is for a particular action the more likely it is that this action was intended by the particular user and that this user was mistaken in the execution of the action issued. In contrast, the lower the degree of knowledge, the more likely that the particular user needs additional tutoring on the particular topic. In view of the above, the value of the degree of a tutoring need (dtutoring) is given by the formula dtutoring=dcertainty/dknowledge. The reason for dividing these two degrees is that the higher the degree of certainty the higher the need for tutoring. However, the degree of a tutoring need and the degree of knowledge are inversely proportional, which means that the higher the degree of knowledge, the less the need for tutoring on the particular topic. Each of the degrees of certainty dcertainty and knowledge dknowledge are estimated as a weighted sum. Thus, the degree of certainty (dcertainty) is calculated as follows:

11

Kabassi, K. & Virvou, M. (2006). Multi-Attribute Utility Theory and Adaptive Techniques for Intelligent Web-Based Educational Software. Instructional Science, 34(2), pp. 313-158.

d certaint y =k 1r + k 2 f + k 3 c

(1)

Additionally, the degree of knowledge (dknowledge) is calculated as follows:

d knowledge = k 4 v + k5 e

(2)

The main goal of Web F-SMILE is to try to optimise the degree of tutoring need for every topic of the theory. In order to achieve this, Web F-SMILE tries to optimise the degree of certainty for every alternative action generated, which means maximising the function (1). Additionally, Web F-SMILE tries to minimise the degree of knowledge for each topic of the theory, as dknowledge is inversely proportional to the degree that the system tried to optimise. Thus it tries to minimise function (2). In the reviewing process of the criteria that was conducted in the empirical study, the human experts showed how the criteria could be used in a different domain. The formulae used for the application of MAUT do not need any adjustments, as the criteria have already been adapted. However, the experiment that is described in the next section and is essential for the estimation of the criteria weights should be adapted to each domain. 7. Empirical Study for the Estimation of Weights

According to MAUT, to estimate the values of the weights k j ’s in formulae (1) and (2), one must try to obtain (n-1) pairs of indifferent suggestions that are associated with the evaluated user’s action, where n is the number of criteria. A pair of indifferent suggestions refers to situations where the DM has found two suggestions that are equally important for him/her and cannot prefer one for another. Taking into account the theory of MAUT that can be translated in mathematics as U(s1)=U(s2), where s1 and s2 are the suggestions made by the DM. Therefore, we will compare the degree of certainty for actions that are generated by the DM and the degree of knowledge for topics related to these actions. What is essential for the estimation of the weights is to find some pairs of indifferent actions that the DM generates and some pairs of indifferent relevant topics of the theory. Two actions s1 and s 2 are indifferent when dc( s1 ) = dc( s 2 ) . Similarly two topics of the theory t1 and t 2 are indifferent when dk (t1 ) = dk (t 2 ) . For the calculation of the degree of certainty dcertainty and the degree of knowledge dknowledge in Web F-SMILE the criteria that are taken into account are three and two, respectively. Therefore, two pairs of indifferent actions and one pair of indifferent topics should be found in order to calculate the weights of the criteria that Web F-SMILE takes into account when calculating the values of the degree of certainty and the degree of knowledge, respectively. For this purpose, during an empirical study, 15 novice learners were asked to interact with a standard file manipulation program as they would normally do and their actions were video recorded. The protocols collected were given to the 10 human experts that had initially selected the criteria that the system uses in order to analyse them. It is important that the human experts that participate in the experiment for the definition of the criteria are the same as the experts that analyse the protocols in order to make valid conclusions. The protocols collected consisted of 1342 actions. From these actions the human experts found 192 as possibly not intended. Each user had executed almost 90 actions. Such actions were essential so that the human expert could collect enough information about the user and draw some safe 12

Kabassi, K. & Virvou, M. (2006). Multi-Attribute Utility Theory and Adaptive Techniques for Intelligent Web-Based Educational Software. Instructional Science, 34(2), pp. 313-158. conclusions. Furthermore, the user should issue some actions before the individual user model of the system could acquire enough information about the user and, thus, calculate the values of the criteria. The analysis of the comments of human experts revealed that, indeed, in some cases the human experts thought that two actions were equally possible to have been intended by the user. Additionally, the empirical study revealed that sometimes two topics of the theory were equally needed by the user in order to achieve his/her goals. From these cases, we isolated the cases where all human experts were indifferent between two actions or two theory topics. Finally, from the cases where the human experts’ were unanimous, we selected 2 pairs of indifferent actions for using them in the calculation of the weights in the formula of the degree of certainty dcertainty and 1 other pair of indifferent topics of the theory for using them in the formula of the degree of knowledge dknowledge. If this procedure was to be applied in a different domain then the empirical study described above should be repeated in order to select new examples of indifferent actions or theory topics. As soon as the new examples have been selected then the values of the criteria are estimated and the calculations described below are repeated. From a user’s protocol we selected the example presented below where the expert users found two alternative actions that corresponded to two different topics. The particular example was chosen due to the fact that the alternative actions proposed by the human experts corresponded to different theory topics, which the experts thought that were equally important to propose to the learner in order to take additional tutoring. The user’s initial file-store state is presented in figure 3.

Figure 3: The Learner’s Initial File-Store State After having inserted a new floppy disk in his computer, the user explores the folder C:\My Documents\, selects the file C:\My Documents\File1.txt and then executes the command paste. However, the user had not previously executed a copy or a cut command. Therefore, the particular action was found by the human experts as unintended and they suggested that the user probably wanted to issue one of the two following actions: s1 s2

Copy(C:\My Documents\file1.doc) Cut(C:\My Documents\file1.doc)

If the user’s intention was the first alternative action, the experts suggested that he would need more tutoring on the theory topic of ‘Copying Objects’ ( t1 ); otherwise the experts suggested that the users would need additional tutoring on the theory topic of ‘Moving Objects’ ( t 2 ). However, the human experts could not decide which one was more appropriate for the particular user. Therefore, from the particular user protocol we selected one pair of indifferent actions and one pair of indifferent topics of the theory.

13

Kabassi, K. & Virvou, M. (2006). Multi-Attribute Utility Theory and Adaptive Techniques for Intelligent Web-Based Educational Software. Instructional Science, 34(2), pp. 313-158. The relevance of both actions suggested to the action selected by the user was 0.80 ( r = 0.80 for both s1 and s 2 ), as the commands copy and cut are equally similar to command paste. The value of this criterion is not connected to the information stored in the user model and depends only on the information stored in the domain representation component, as both actions refer to the same object (file1.doc). The value 0.80 has been determined as the commands cut and copy are neighbouring in the menus of the systems as well as in the hierarchy of users’ actions. According to the information of the learner model the frequency of an error connected with the copy command was 0.40 ( f s1 = 0.40 )

whereas the frequency of error connected with the cut command was 0.32 ( f s2 = 0.32 ). Furthermore, the user had made some carelessness errors connected with these commands. More specifically, the degree of carelessness in the execution of the copy command was 0.04 ( c s1 = 0.04 ) whereas the corresponding degree for the cut command was 0.16 ( c s2 = 0.16 ). Moreover, the user had visited the theory topic of ‘Copying Objects’ ( v t1 = 1 ) just once and he had taken tutoring on ‘Moving Objects’ ( v t 2 = 2 ) twice. Finally, the percentage of correct executions of the command copy was 60% ( e t1 = 0.60 ) whereas the corresponding percentage for the command cut was 11% ( e t 2 = 0.11 ). From another user’s protocol we selected the example presented below where the expert users found two alternative actions that were equally possible to have been intended by the user. The user’s initial file-store state is presented in figure 4.

Figure 4: The Learner’s Initial File-Store State The user of the example issued the command copy after having explored A:\. However, the action copy(A:\) cannot have any result. If one wants to copy the contents of the floppy disk, s/he should select all the objects before copying them. The human experts thought that the particular action was unintended and, therefore, produced two alternative actions:

s3 s4

Copy(Α:\May.txt) Cut(Α:\May.txt)

The two alternative actions have been created since the floppy disk contained the file ‘May.txt’ only. The relevance of the alternatives with the action issued by the user was 1 and 0.9, respectively. Moreover, the frequency of errors connected with the copy command was 0.13 ( f s3 = 0.13 ) and the frequency of erroneous executions of the cut command was 0.32 ( f s4 = 0.32 ). Furthermore, the user had made some carelessness errors connected with these commands. More specifically, the degree of carelessness in the execution of the copy command was 0.05 ( c s3 = 0.05 ) whereas the corresponding 14

Kabassi, K. & Virvou, M. (2006). Multi-Attribute Utility Theory and Adaptive Techniques for Intelligent Web-Based Educational Software. Instructional Science, 34(2), pp. 313-158. degree for the cut command was 0.03 ( c s4 = 0.03 ). The values of all the criteria for the selected actions ( s1 , s 2 , s3 , s 4 ) and the theory topics ( t1 ,t 2 ) that were selected are presented in Table 1 and Table 2, respectively.

r s1 s2 s3

c

f

0.80 0.80 1

0.40 0.32 0.13

0.04 0.16 0.05

0.90 0.32 0.03 s4 Table 1: The values of the criteria for the actions selected by the empirical study

v

e

1 0.60 t1 2 0.11 t2 Table 2: The values of the criteria for the topics of the theory selected by the empirical study . The values of the criteria, as these have been identified in the empirical study, are assigned to the equations of the indifferent pairs using the formula 3 and formula 4. This task results in the generation of 2 corresponding linear equations for each formula. The linear equations together with the fact that the sum of the k j ’s is 1, allows their value to be determined. 7.1 Degree of certainty

As already mentioned, the empirical study revealed 2 pairs of actions, ( s1 , s 2 ) and ( s3 , s 4 ) , which were generated by the human trainers in response to a user's error that were equally possible to have been intended by the user. This means that for these actions we have the following equations: dc( s1 ) = dc( s 2 ) and dc( s3 ) = dc( s 4 ) . The degree of certainty dcertainty is calculated using the formula (1), which means that dc = k1r + k 2 f + k3c . As soon as the pairs are collected, the values of the criteria (r , f , c) are defined taking into account the information of the domain representation and the learner model of the user. More specifically, the values of the criteria of Table 1 are used for generating the linear equations: s1 I s 2 ⇔ dc( s1 ) = dc( s 2 ) ⇔ 0.80k1 + 0.40k 2 + 0.04k 3 = 0.80k1 + 0.32k 2 + 0.16k 3 s3 I s 4 ⇔ dc ( s1 ) = dc ( s 2 ) ⇔ 1k1 + 0.13k 2 + 0.05k 3 = 0.90k1 + 0.32k 2 + 0.03k 3 These equations together with the equation

3

∑k j =1

j

= 1 , which is more simplified to k1 + k 2 + k 3 = 1 ,

form a system of 3 equations in 3 unknowns, which is easy to be solved. After this process, the values

15

Kabassi, K. & Virvou, M. (2006). Multi-Attribute Utility Theory and Adaptive Techniques for Intelligent Web-Based Educational Software. Instructional Science, 34(2), pp. 313-158. of the weights of the criteria were found to be k1 = 0.515 , k 2 = 0.291 and k 3 = 0.194 . Therefore, the final formula for the calculation of the degree of certainty was found to the following:

dc = 0.515r + 0.291 f + 0.194c

(3)

7.2 Degree of knowledge The empirical study revealed 1 pair of topics of the theory (t1 , t 2 ) which was connected with the user’s intentions and were equally known and needed by the learners. For these actions we have dk (t1 ) = dk (t 2 ) . The degree of knowledge dknowledge is calculated using the formula (2), which means that dk = k 4 v + k 5 e . As soon as the pairs are collected, the values of the criteria (v, e) are defined taking into account the information of the domain representation and the learner model of the user. More specifically, the values of the criteria of Table 2 are used for generating the linear equations: t1 I t 2 ⇔ dk (t1 ) = dk (t 2 ) ⇔ 1k 4 + 0.60k 5 = 2k 4 + 0.11k 5 This equation together with the equation

5

∑k j =4

j

= 1 forms a system of 2 equations in 2 unknowns,

which is easy to be solved. After this process, the values of the weights of the certainty parameters were found to be k 4 = 0.329 and k 5 = 0.671 . Therefore, the final formula for the calculation of the degree of certainty is:

dk = 0.329v + 0.671e

(4)

Applying the values of the attributes in functions 5 and 6, the system can estimate the value of the degree of a tutoring need (dtutoring=dcertainty/dknowledge). For example, for the user of the sample session presented in Section 3, the degree of a tutoring need for the theory topic ‘Moving Objects’ and ‘Moving Folders’ is 1.30 (dtutoring=1.30) whereas the same degree for the theory topics ‘Moving Files’ and ‘Copying Objects’ are 1.08 and 0.87, respectively. Therefore, the system decided that the theory topics that were most appropriate for the particular user was ‘Moving Objects’, ‘Moving Folders’ and ‘Moving Files’. These theory topics had the highest value of the degree of a tutoring need. Additionally, the system decided to advise the user to revise the topic of ‘Copying Objects’ as it also had high degree of a tutoring need. The degree of tutoring need dtutoring was close to null for all the other theory topics and, therefore, they are not presented to the user.

8. Discussion In this paper, we have described how a multi-criteria decision making theory has been applied in Web F-SMILE, an intelligent learning environment over the Web that trains users in basic Information Technology skills. In particular, it helps users learn how to work with the Windows operating system focusing on concepts related to the management of the file store and the electronic mailbox. The system constantly watches the user and in case it suspects that s/he is involved in a problematic situation, it provides adaptive tutoring. Therefore, we use two adaptation techniques in

16

Kabassi, K. & Virvou, M. (2006). Multi-Attribute Utility Theory and Adaptive Techniques for Intelligent Web-Based Educational Software. Instructional Science, 34(2), pp. 313-158. order to adapt tutoring to the characteristics, needs and level of knowledge of each individual user. In particular, adaptive presentation techniques are used to present examples of use of an unknown command in the context of the learner’s own file-store or electronic mailbox. Moreover, Web FSMILE uses adaptive link annotation techniques to present other parts of knowledge that are believed to be of interest to the learner for the particular case. In this process the most important phase of the system is the decision making of what to present to the particular user based on his/her level of knowledge and individual characteristics. Although research in student modelling focuses on modelling the reasoning of users, who are also learners, multi-criteria decision aid has not been explored as much as it could. Therefore, in our approach we show how a theory from Multi-Criteria Analysis may be used to model the decisionmaking competence of human experts and enrich our system with the ability of teachers to guide students’ learning in accordance with their learning characteristics and level of knowledge. The theory that we have used is called Multi-Attribute Utility Theory (MAUT) and we have applied it in order to rank the multiple pieces of knowledge that are required for a user to achieve his/her goals. Similarly with the whole area of Multi-Criteria Analysis, MAUT has been very popular in domains such as e-commerce. However, the advantages of such a decision making theory have not been explored for the purposes of an e-learning system. The application of MAUT to Web F-SMILE in order to estimate the multi-attribute utility of alternatives has shown that applying a decision making theory in an e-learning system can improve the decision process of a system so as to simulate the human decision making of an expert trainer. More specifically, Web F-SMILE uses the information that is stored in the individual user model in order to calculate the values of the criteria and then applies MAUT in order to decide which kind of alternative action and which theory topic it is going to propose to that user as part of the adaptive teaching process. As a result, the table of contents presented to each user may vary considerably even if two users have made the same error. This is due to the fact that the criteria take different values from the students’ individual user model. Furthermore, the theory topic presented to each learner is formed dynamically, as the examples provided each time are adapted to the current file store state of the particular user. MAUT provides a formal way of aggregating different criteria and a way for the calculation of the weights of these criteria so as to reflect the reasoning of human experts of the domain. Yet, the main advantage of MAUT in comparison to other decision making theories is the nature of the experiments that are needed prior to the application of this kind of theories in a learning environment. MAUT is based on an analysis of equivalent situations that the human experts must have identified. In contrast, in other theories human experts are asked to give a ranking of the criteria they use. However, the experts usually find it more natural to identify equivalent situations than give a ranking of the criteria they use. Thus, the results of the empirical study that is required for the application of MAUT are more reliable. However, for the results to be reliable, the equivalent situations should be identified by some experts. If the experts reviewed are only a few then the situations that have been selected may not be considered equivalent by human experts in general but only by a small minority. Additionally, if a satisfactory number of experts state that the situations selected are equivalent then even if more experts were selected to participate the experiment the results would not have changed. Therefore, if 8 human experts agree in the selection of the final set of equivalent situations then the results are considered to be accurate. The results of the empirical study were verified in a second experiment where the set of criteria selected were given to some human experts that confirmed their utility. As these human experts stated, this set could be used in any domain simply by adjusting the criterion that corresponds to the 17

Kabassi, K. & Virvou, M. (2006). Multi-Attribute Utility Theory and Adaptive Techniques for Intelligent Web-Based Educational Software. Instructional Science, 34(2), pp. 313-158. number of correct executions of a command. If this procedure was to be applied in a different domain, such as Mathematics or Physics, this criterion should represent the exercises that are related to a theory topic and the user has successfully completed. As the set of criteria remains stable, the formulae used for the application of MAUT do not need any adjustments. However, what may vary considerably are the weights of the criteria. More specifically, every time the theory is to be used in a different domain, an empirical study should be conducted, in which human experts are asked to observe users interacting with a system and advise them. In decision-making and decision-support tasks, a model of the user’s characteristics and goals is required so that appropriate decisions can be made. Therefore, in our research special emphasis has been given on the learner modelling of Web F-SMILE. More specifically, Web F-SMILE uses a combination of stereotypes and techniques that refer to the individual user in order to adapt its interaction with each learner. The combination of these techniques constitutes a novel approach for the improvement of the reasoning of e-learning systems. The user model of the system maintains information about the user’s possible errors and misconceptions as well as a representation of the user’s knowledge of the domain. Therefore, if the domain taught by the system is to be expanded then the user model maintained by the system can also be expanded to keep information for every concept handled. It is among our future plans to evaluate the application of MAUT in Web F-SMILE. The main aim of the application of MAUT in a learning environment is to simulate the reasoning of human experts when providing individualised assistance. Therefore, the evaluation of the system is planned to involve a comparison of the Web F-SMILE responses to the responses of human experts who will observe the same users and reason about their actions in accordance to the method described in (Virvou & Kabassi 2004). Finally, we plan to apply MAUT in another learning environment of a different domain so as to show the generality of the method proposed in this paper.

9. References [1] S.R. Alpert, M.K. Singley, P.G. Fairweather, Deploying Intelligent Tutors on the Web: An Architecture and an Example, International Journal of Artificial Intelligence in Education, 10 (1999) 183-197. [2] L. Ardissono, A. Felfernig, G. Friedrich, D. Jannach, R. Schäfer, M. Zanker, Intelligent Interfaces for Distributed Web-Based Product and Service Configuration, in: M. Zhong et al. (eds.), Web Intelligence 2001: Lecture Notes in Artificial Intelligence, Vol. 2198, Springer Verlag, Berlin, Heidelber, 2001, pp. 184-188. [3] M. Bauer (1995) A Dempster-Shafer approach to modelling agent preferences for plan recognition. User Modeling and User-Adapted Interaction. [4] P. Brusilovsky, Methods and techniques of adaptive hypermedia User Modeling and User Adapted Interaction, 6 (1996) 87-129. [5] P. Brusilovsky, J. Eklund, E. Schwarz, Web-based education for all: A tool for developing adaptive courseware, Computer Networks and ISDN Systems, 30 (1998) 291-300. [6] D. Chin, A. Porage Acquiring User Preferences for Product Customization, in: M. Bauer, P. Gmytrasiewicz, J. Vassileva (Eds.) Proceedings of the 8th International Conference on User Modeling 2001, UM 2001, LNAI 2109, 2001, 95 –104. [7] A. Collins, R. Michalski, The Logic of Plausible Reasoning: A core Theory, Cognitive Science, 13 (1989) 1-49 [8] A. Collins, P. Neville, K. Bielaczyc, The role of different media in designing learning environments, International Journal of Artificial Intelligence in Education, 11 (2000) 144-162. 18

Kabassi, K. & Virvou, M. (2006). Multi-Attribute Utility Theory and Adaptive Techniques for Intelligent Web-Based Educational Software. Instructional Science, 34(2), pp. 313-158. [9] C. Conati, A. Gertner and K. VanLehn, Using Bayesian Networks to Manage Uncertainty in Student Modeling. User Modeling and User-Adapted Interaction, 12(4) (2002) 371-417. [10] C. Conati & K. VanLehn (1996) POLA: A student modelling framework for Probabilistic OnLine Assessement of problem solving performance. In S. Carberry & I. Zukerman (Eds.) Proceedings of the Fifth International Conference on User Modeling, pp. 75-82. [11] P. De Bra, Adaptive Educational Hypermedia on the Web, Communications of the ACM, 45, 2002, 60-61. [12] P. De Bra, A. Aerts, D. Smits, N. Stash (2002): AHA! the next generation. In: Hypertext'02 Proceedings of the Thirteenth ACM Conference on Hypertext and Hypermedia. June 11-15, 2002, College Park, Maryland, USA. p.21-22 [13] P. De Bra, G.J. Houben, H. Wu, AHAM: A Dexter-based Reference Model for Adaptive Hypermedia, in: K. Tochtermann, J. Westbomke, U.K. Wiil, J. Leggett (eds.) Proceedings of the 10th ACM Conference on Hypertext and Hypermedia, New York: ACM Inc, 1999, 147-156. [14] M.C. Desmarais, A. Maluf, A. & J. Liu (1995) User-expertise modelling with empirically derived probabilistic implication networks. User Modeling and User-Adapted Interaction. [15] D. Duncan, P. Brna, L. Morss, A Bayesian Approach to Diagnosing Problems with Prolog Control Flow, Proceedings of the 4th International Conference on User Modeling, 1994. [16] P.C. Fishburn, (1967). Additive Utilities with Incomplete Product Set: Applications to Priorities and Assignments, Operations Research. [17] N. Hammond, Hypermedia and learning: Who guides whom?, in: H. Maurer (ed.) Computer Assisted Learning, Lecture Notes in Computer Science, Vol. 360, Springer-Verlag, Berlin, 1989, 167-181. [18] Horvitz E. & Barry M. (1995) Display of information for time-critical decision making. In P. Besnard & S. Hanks (eds.) Proceedings of the Eleventh Conference on Uncertainty in Artificial Intelligence, pp. 296-314. [19] X. Huang, G. McCalla, J. Greer, E. Neufeld, Revising deductive knowledge and stereotypical knowledge in a student model, User Modeling and User Adapted Interaction, 1 (1991) 87-115. [20] M. J. Huber, E. H. Durfee, & M. P. Wellman (1994) The automated mapping of plans for plan recognition. In R. Lopez de Mantaras & D. Poole (eds.) Proceedings of the Tenth Conference on Uncertainty in Artificial Intelligence, pp. 344-351. [21] C.L. Hwang, K. Yoon, Multiple Attribute Decision Making: Methods and Applications. Lecture Notes in Economics and Mathematical Systems 186 (1981) Berlin/Heidelberg/New York: Springer. [22] A. Jameson (1996) Numerical Uncertainty Management in User and Student Modeling: An Overview of Systems and Issues. User Modelling and User-Adapted Interaction. [23] A. Jameson, R. Schäfer, J. Simons, & T. Weis (1995)Adaptive provision of evaluationoriented information: Tasks and techniques. In C.S. Mellish (ed.) Proceedings of the Fourteenth International Joint Conference on Artificial Intelligence, pp. 1886-1893. [24] K. Kabassi & M. Virvou (2003). Combination of a Cognitive Theory with the Multi-Attribute Utility Theory. In V. Palade, R. J. Howlett, L. Jain (eds.): Knowledge-Based Intelligent Information and Engineering Systems – KES 2003, Lecture Notes in Artificial Intelligence, subseries of Lecture Notes in Computer Science, Vol. 2773, Springer, Berlin, Part I, pp. 944-950. [25] K. Kabassi, M. Virvou, Personalised Adult e-Training on Computer Use based on Multiple Attribute Decision Making. Interacting with Computers, 16(1) (2004) 115-132.

19

Kabassi, K. & Virvou, M. (2006). Multi-Attribute Utility Theory and Adaptive Techniques for Intelligent Web-Based Educational Software. Instructional Science, 34(2), pp. 313-158. [26] J. Kay, Stereotypes, Student Models and Scrutability, in: G. Gautier, C. Frasson, K. VanLehn (eds.), Lecture Notes in Computer Science, Intelligent Tutoring Systems (Proceedings of the 5th International Conference on Intelligent Tutoring Systems), Springer Verlag, Berlin, 2000, 19-30. [27] L. Kolonder, Case-Based Reasoning, Morgan Kaufmann Publisher Inc., San Mateo CA, 1993. [28] D. Kudenko, M. Bauer, D. Dengler, Group Decision Making Through Mediated Discussions, Proceedings of the 9th International Conference on User Modelling, 2003. [29] R. Lahdelma, P. Salminen, J. Hokkanen, Using Multicriteria Methods in Environmental Planning and Management, Environmental Management, 26 (2000) 565-605. [30] D.B. Leake, Case-Based Reasoning: Experiences, Lessons and Future Directions, AAAI Press, Menlo Park, California, 1996. [31] G. Linden, S. Hanks, N. Lesh, Interactive Assessment of User Preference Models: The Automated Travel Assistant, in: Jameson et al. (eds), User-Modeling: Proceedings of the Sixth International Conference (UM'97). New-York, NY: Springer Wien, 1997, 67-78. [32] J. Martin, K. VanLehn, Student assessment using Bayesian nets, International Journal of Human Computer Studies, 42 (1995) 575-591. [33] G. McCalla, The central importance of student modelling to intelligent tutoring, in E. Costa (Ed.) New Directions for Intelligent Tutoring Systems, Berlin: Springer Verlag, 1992. [34] Microsoft Corporation, Microsoft® Windows® 98 Resource Kit. Microsoft Press, 1998. [35] M. Murphy, M. McTear, Learner modelling for intelligent CALL, in: A. Jameson, C. Paris, and C. Tasso (Eds.) Proceedings of the 6th International Conference on User Modeling, Springer Verlag, Berlin, 1997, 301-312. [36] T. Murray, J. Piemonte, S. Khan, T. Shen, C. Condit, Evaluating the Need for Intelligence in an Adaptive Hypermedia System, in: Gauthier, G., Frasson, C. and VanLehn, K. (Eds.) Intelligent Tutoring Systems, Proceedings of the 5th International Conference on Intelligent Tutoring Systems, Lecture Notes in Computer Science, Vol. 1839, Springer-Verlag, Vienna, 2000, 373-382. [37] V.A. Petrushin, K.M. Sinitsa Using probabilistic reasoning techniques for learner modelling. World Conference on AI in Education, 1993, 418-425. [38] C. Peylo, T. Thelen, C. Rollinger & H. Gust (2000). A Web-based intelligent educational system for PROLOG. In Peylo, C. (Ed.): Proceedings of the International Workshop on Adaptive and Intelligent Web-based Educational Systems (held in Conjunction with ITS 2000). [39] D.V. Pynadath & M.P. Wellman (1995) Accounting for context in plan recognition, with application to traffic monitoring. In P. Besnard & S. Hanks (eds.) Proceedings of the Eleventh Conference on Uncertainty in Artificial Intelligence, pp. 472-481. [40] E. Rich, Stereotypes and User Modeling, in: Kobsa, A. & Wahlster, W. (eds.) User Models in Dialog Systems, 1989, 199-214. [41] E. Rich, Users are individuals: individualizing user models, International Journal of HumanComputer Studies, 51 (1999) 323-338. [42] R. Schrank, D. Edelson, A role for AI in education: using technology to reshape education, Journal of Artificial Intelligence in Education, 1(2) (1990) pp. 3-20. [43] W. Schütz, R. Schäfer, Bayesian networks for estimating the user's interests in the context of a configuration task, in: R. Schäfer, M. E. Müller, and S. A. Macskassy (eds.), Proceedings of the UM2001 Workshop on Machine Learning for User Modeling, 2001, 23-36. [44] M.E. Shiri, E. Aїmeur, C. Frasson, Student Modelling by Case-Based Reasoning, in: B.P. Goettl, H.M. Halff, C.L. Redfield, V.J. Shute (eds.): Fourth International Conference on

20

Kabassi, K. & Virvou, M. (2006). Multi-Attribute Utility Theory and Adaptive Techniques for Intelligent Web-Based Educational Software. Instructional Science, 34(2), pp. 313-158. Intelligent Tutoring Systems. Lecture Notes in Computer Science, Vol. 1452. Springer-Verlag, Berlin, 1998, 394-404. [45] R. Sison, M. Shimura, Student Modeling and Machine Learning, International Journal of Artificial Intelligence in Education, 9 (1998) 128-158. [46] D. Sperber, D. Wilson, Relevance: Communication and Cognition. Language and Though Series, Harvard University Press, Cambridge, MA, 1986. [47] S. van Mulken (1996) Reasoning about the user’s decoding of presentations in an intelligent multi-media presentation system. In S. Carberry & I. Zukerman (eds.) Proceedings of the Fifth International Conference on User Modeling, pp. 67-74. [48] P. Vinke, Multicriteria Decision-Aid. Wiley, 1992. [49] M. Virvou & K. Kabassi (2004). Evaluating an Intelligent Graphical User Interface by Comparison with Human Experts. Knowledge-Based Systems, 17(1), pp. 31-37.

21