cost of building an adaptive capability into an application and the technical ..... intelligent support systems, agent-based interaction requires the computer.
Adaptive Systems: from intelligent tutoring to autonomous agents1 David Benyon* and Dianne Murray+ ABSTRACT Computer systems which can automatically alter aspects of their functionality or interface to suit the needs of individuals or groups of users have appeared over the years in a variety of guises. Most recently attention has focused on intelligent interface agents which are seen as specialised, knowledge-based systems acting on behalf of the user in some aspect of the interaction. Similar requirements for automatic adaptation have been noted in intelligent tutoring systems, natural language systems and intelligent interfaces. This paper brings together the research which has emanated from a number of backgrounds and provides a unifying perspective on adaptive systems in general. We present an architecture for adaptive systems and a methodology for their development. The paper also describes software support for producing adaptive systems and offers some experimental evidence to justify both the desirability and feasibility of exploiting an adaptive system approach to human-computer interaction. keywords: adaptive systems, intelligent interfaces, user models, domain models, autonomous agents, adaptive system architecture, development methodology 1.
INTRODUCTION
The concept that computer systems should be capable of adapting themselves to suit the needs either of individuals or of different classes of users is an apparently attractive, if slightly unusual, one. Adaptive computer interfaces have been discussed over the past ten years or so and have had a number of advocates (Edmonds, 1982; Innocent, 1982; Zissos and Witten, 1985) but it is only now that consideration of how to build such systems is gaining prominence (Kobsa and Wahlster, 1989; Browne, Totterdell and Norman, 1990; Hancock and Chignell, 1989, Sullivan and Tyler, 1988; SchneiderHufschmidt, Kuhme and Malinowski, 1993; Gray, Hefley and Murray, 1993). 1
Some parts of this paper have been published in Benyon and Murray, 1993
*
Computing Department, Open University, Milton Keynes, MK7 6AA, UK
+
Social and Computer Sciences research group, Dept. of Sociology, University of Surrey, Guildford, UK page 1
Recent interest in the development of multi-agent systems (Maes, 1991; Laurel, 1990) and intelligent interfaces (Chignell and Hancock, 1988) have also focused attention on this important area. Early applications of the adaptive system concept have been rather disappointing and problems have proved far harder to deal with than was first expected. Will the dream of interface agents go the same way? In this paper we provide a unifying perspective on adaptive systems which facilitates the comparison of different systems and offers guidance on the circumstances under which software designers should consider an adaptive system approach to human-computer interaction design. After briefly considering the arguments for and against adaptive systems in this section, we survey existing applications of the concept (Section 2). This illustrates the range of problems which software developers have sought to solve using an adaptive system approach. Although there are many differences in the design of such systems, we believe that they all share a common architecture. This is presented in Section 3. In the following section we consider a methodology for the development of adaptive systems within the framework provided by the adaptive system architecture. The next section discusses our own experience of developing specific adaptive systems and serves to illustrate in concrete terms the more abstract nature of the discussion of sections 3 and 4. The final section (Section 6) offers our conclusions as to the likely developments of adaptive systems over the next few years. One of the aims of this paper is to provide a reference model for the comparison of adaptive systems. Our intention is to survey the field and to draw together the numerous efforts which have been expended. We believe that much can be learnt from comparing and contrasting the philosophies and experiences emanating from areas as apparently diverse as intelligent tutoring systems and agent-based human-computer interaction. Emerging from this experience is a discipline which promises to impact not just human-computer interaction, but also on the multiple interacting agent systems which we see as being a major development in HCI and computer science over the next few years. 1.1
Why should we have adaptive systems?
Computer systems can be difficult to learn and once learnt, may be easily forgotten. As the proliferation of computer applications and different delivery platforms continues apace, the number of individuals exposed to the vagaries of a range of computer environments likewise continues to grow. Computer applications tend to embody specific characteristics which make the chosen design solution better suited to some users than others. However, many systems could potentially have a range of different features to match the diversity of user populations. Even when practised users learn one system well, the software industry’s predilection for development of new features, bug-fixes and production of new (i.e. slightly different and ‘improved’) versions (Thimbleby, 1990) means page 2
much interface functionality is continually changing. There is also the problem of users changing their perception of and proficiency with different software systems. Discretionary users are those users who have the option not to use a system. They must be persuaded or otherwise enticed into making use of the facilities. Once discretionary users are convinced that a system is worth using, they may alter their working practices and become reliant on the system. As with novice users, they soon become too constrained by the features of the original system. We need to provide such users with effective methods of graduating to more complex use of systems. Some designers do, of course, implement systems to cater for different users, environments and working practices. Some build different systems for different markets. Others allow individualisation by means of ‘customising’ facilities or 'macros'. The drawback with such approaches is that the user must learn functions which are tangential to their main task. Although some routines will only have to be set once, others involve learning specialised commands. Tailoring facilities are typically very general, do not take finegrained, individual differences into account and do not cater for a user’s task needs, either perceived or implicit. Although users may learn and become committed to the use of a piece of software and so are no longer discretionary with respect to the system itself, they are still discretionary with respect to customising the system to better suit their needs. The use of metaphor and analogy as a means of making system functionality accessible to user populations has been claimed to overcome many usage problems. However, this can be a hit-and-miss affair, well-suited for some but not for the population as a whole, dependent as it is upon a closely shared appreciation of the basis of the metaphor. Recent criticism (Kay, 1989) of metaphor in interface design adds weight to this argument. Adaptive systems are systems which can alter aspects of their structure, functionality or interface in order to accommodate the differing needs of individuals or groups of users and the changing needs of users over time (Benyon, Innocent and Murray, 1987). Adaptive systems seek to take over the burden of tailoring systems to individuals and groups. One important question, of course, is whether this goal is feasible technically, operationally or economically. For example, Thimbleby (1990) argues that only complex systems can benefit from an adaptive capability and that, being complex, it is not possible to provide such a capability because the user patterns of usage will be weaker. He asserts that there is simply no enough bandwidth in the user interface to accommodate the required functionality and adaptation. Similar arguments can be made regarding the cost of building an adaptive capability into an application and the technical problems concerned with speed of processing, knowledge representation and so on. Not only has it been asked whether systems can actually incorporate enough suitable knowledge about an individual user in order to make adaptive responses to that person a viable proposition, but the basis upon page 3
which user characteristics can be inferred has been questioned (Kobsa, 1990; Pullinger, 1989). We do not accept that the feasibility argument can be theoretically proven. A very simple adaptive mechanism may be highly effective. The cost associated with the implementation of adaptive systems can be justified if they significantly improve usability and the quality of interaction. Furthermore, the inclusion of an adaptive capability may not be such a large overhead if it arises as a natural consequence of better attention and metrics being applied to interactive system design. One of the objectives of the work reported here has been to develop cost-effective software support for adaptive systems development. The feasibility of adaptive systems remains a usability issue (Benyon, 1993). Suffice it to say at this point that we strongly believe that computer systems and interfaces to computer applications can be made to adapt to suitable and identifiable user characteristics, to the benefit of the user in terms of ‘effectiveness’ of performance, speed and accuracy of execution and personal satisfaction, and that appropriate characteristics can be identified, inferred and tested. We propose that some systems must be inherently adaptive if they are to fulfil their purpose and that many systems which have to deal with variety, of users or of environments, may be effective as adaptive systems. 2.
APPLICATIONS OF THE ADAPTIVE SYSTEM CONCEPT
Benyon and Murray (1993) provide a detailed review of adaptive user interfaces. Other useful reviews are provided by Kok (19xx) and Norcio and Stanley (1988). One of the problems with the area is that similar ideas have emerged from different disciplines. As is often the case, different disciplines employ their own terminology which make comparisons and generalisation difficult. The classification below has been chosen to reflect the different historical backgrounds which contribute to adaptive systems research. Systems which are described as ‘intelligent’ may take many forms and are built for many reasons and to many different ends (Elkerton, 1987; Mason and Edwards, 1988). Claims, however, have in the past been somewhat overstretched and few systems have fully lived up to their grandiose expectations. Future systems may well tackle smaller, more manageable areas and deal with issues which are more tractable. 2.1
Intelligent Interfaces
Intelligent Interfaces emerged as an important theme in the UK Government funded Alvey initiative in the early to mid-eighties where the term ‘Intelligent Front End’ (IFE) was preferred. An IFE was then conceived of as an interface to an existing piece of software which would be accessed by a large number of diverse users. The IFE would provide a usable interface to packages which were too complex or too technically daunting to offer easy interaction to such a variety of users (Bundy, 1983; Innocent, 1982; Benyon, 1984). The Monitor system (Benyon, 1984, Benyon and Murray, 1988), page 4
although implemented in an intelligent tutoring system domain, focused on the structure of adaptive systems in general. The work provided an architecture for adaptive systems which included an explicit representation of individual users and an explicit representation of the domain. This was qualitatively different from other ‘adaptive’ systems which provided adaptation according to various mechanical or statistical means (Furnas, 1985; Greenberg and Witten, 1985) and which included a limited and implicit user model. There was no attempt to maintain a long-term representation of user characteristics. More recently intelligent interfaces have been applied to information retrieval (Chignell and Hancock, 1988; Branjik, Guida and Tasso,1990). 2.2
Natural Language systems
A significant strand in the investigation of adaptive systems has been the extensive attention paid to Natural Language (NL) interfaces, which provide a tough focus for research into aspects of plan, goal and belief recognition. Many systems are simulations or representations of small subsets of an entire dialogue, but they serve to illustrate the difficulties of inferring user’s intentions and future actions from dialogue only. Related research strands such as speech act and discourse theories, decision theory and rhetorical structure theory have been employed, but the problems remain difficult to overcome, even when only considering written and not spoken interaction. Natural language systems adapt by generating text appropriate to the particular query and characteristics of individual users. To do this they have to infer the user’s needs and focus of attention from the (ambiguous) use of natural language. Anaphoric references (the use of words such as 'it', 'that', etc.) and ellipsis (where information is missing from a statement) offer difficult syntactic problems, but inferring the semantics of an utterance and the intention which the user had in making that utterance are even more intractable problems which have generated a wealth of research studies in both AI and Computational Linguistics. However, it is now becoming more favourable to consider that NL systems must also include some form of model of the user or dialogue partner in addition to algorithms and heuristics for inferencing and mechanisms of domain knowledge representation. Wahlster and Kobsa (1989) provide several examples of recent systems and current research projects which are more pertinent to HCI concerns. 2.3
Intelligent Tutoring Systems
The rationale of computer-based teaching systems is that, for given students and topics, a computer system can alleviate the variance of human-based teaching skills and can determine the best manner in which to present individually targeted instruction in a constrained subject domain. In order to minimise the discrepancy between a user’s knowledge state and the representation of an identified expert’s knowledge (a ‘goal state’) the ITS must page 5
be able to distinguish between domain-specific expertise and tutorial strategy. A computer coach assesses the current state of a learner’s knowledge (Dede, 1986 covers this area comprehensively) and provides instruction tailored to that individual’s training needs within an overall context of courses, syllabi and tutorial objectives (Self, 1987). ITS are expert-based systems which must be able to recognise errors and misconceptions, to monitor and intervene when necessary at different levels of explanation, and to generate problems on a given set of instructional guidelines. A ‘student model’ of the user of an ITS system stores information on how much the student ‘knows’ about concepts and relationships which are to be learnt and about the student’s level and achievements. It often contains a history of task performance and some detailed representation of the state of an individual’s knowledge in a specified subject area. Some of this may be held in the form of a user profile and can have other uses in management and score-keeping. Student models are explicit representations of a state of knowledge of a specific domain, possibly with individual personal information to maintain records of performance and achievements. Other models - the Tutor model and the Expert model - have very specialised functions. The Expert model is a representation of the knowledge to be imparted, together with the explanatory facility for remedial intervention. It may also contain an error recognition and evaluation feature, and problem-solving model. The Tutorial model arranges teaching strategy, initiates remedial actions and monitors and assesses performance in conjunction with the evaluative function. Problem-generation from the knowledge-base will offer a sequence of problems, adapting the difficulty level on the basis of previous performance, but will also present new problem classes and review or practise already known items. An interface controller arranges the relevant output and accepts student input. The distinction between a student model and other types of model hinges on the use to which the model is put and its location within the interface management system. John Self (Self, 1985) classified the types of knowledge which determine the overall performance of a teaching program as; knowledge of how to teach (which includes knowledge of students in general), knowledge of what is being taught and knowledge of who is being taught (i.e. knowledge of one student in particular). This corresponds to the Tutor model, the Expert model and the Student model. The integration of the Expert and Student models is crucial to the provision of generative, or adaptive tutoring systems (that is, those in which dialogues and teaching ‘frames’ are not predetermined but, rather, are generated interactively on the basis of an evaluative function operating on the Student model). The field of ITS encompasses many detailed considerations such as the teaching strategy (e.g. intelligent computer coach versus the Socratic tutor, discovery-learning versus guided discovery-learning and task-centred page 6
learning-by-doing). They are adaptive systems in that they aim to tailor the tutoring to the needs of individual learners. Historically they have attended principally to ensuring that a student attains a certain level of competence in a well-specified domain. This is measured by applying theories of learning instantiated in concrete examples and comparing performance to that of an expert. 2.4
Intelligent Support Systems
Another popular application intelligent interface systems is in the provision of context-dependent ‘active’ help (Fischer, Lemke and Schwab, 1986; Finin, 1983; Hansen, Holgaard and Smith, 1988; Bronisz, Grossi and Jean-Marie, 1989; Stehouwer and van Bruggen, 1989). On-line help systems track the user’s context and incorporate assistant strategies and a set of action plans in order to intervene when most appropriate or when the user appears to be having difficulty. Intelligent help systems share some characteristics with ITS since a diagnostic strategy is required to provide the most appropriate help for that user in that particular situation. However, they also have to be able to infer the user's high level goal from the low level data available in the form of command usage. Various strategies and approaches have been suggested. (Fischer et al., 1986; Fischer, Morch, and McCall, 1989; Chin, 1986; 1989; Mason, 1986; Jerrams-Smith, 1985). Intelligent help has further developed into ‘critiquing systems’ (Fischer, 1989), where users must be competent in the subject domain being critiqued, rather than being tutees or learners (Moore and Swartout, 1988; Fischer, 1987; Fischer, 1989). A major problem with these mixed-initiative and co-operative dialogue is concerned with who has the ‘controlling hand’. If an assistant or critic offers a piece of advice that an individual overrides, how can that assistant behave ‘sensibly’ while helping with the remainder of the same task, especially if the task is repeated with slightly different data (as in medical expert systems, for example). If the assistant is an intelligent computer system, how can it avoid giving the same bad advice over and over again and actually learn from the user? The system must have knowledge of how to be a ‘competent assistant’ by taking into account those things which the user knows best. 2.5
Explanation Systems
A similar but slightly different strand has become apparent in recent research - that of being able to provide an explanatory facility for the behaviour of the system (e.g. Paris, 1989); Cohen and Jones, 1989). Expert systems have been criticised for failing to provide adequate and suitably-tailored explanations (Steels, 1987) and to be effective these systems must tailor their explanation to the assumed knowledge of the user (Kass and Finin, 1988; Carroll and McKendree, 1988). An explanation-based system combines the problems of NL generation, help and tutoring and presents extremely stubborn problems to researchers and to system builders. Lehner (Lehner, 1987) suggests that the page 7
interaction between a user and an expert-system is actually a situation in which two problem-solvers co-operatively attempt to solve common decision problems. Of course, each will have different decision processes, make use of different heuristics and operate on divergent, if fundamentally identical, data. 2.6
Co-operative Intelligent Agents
Recent interest in computer supported co-operative work (CSCW), distributed artificial intelligence (DAI) and HCI have taken the adaptive system concept further and led to the development of interface agents. Co-operative systems require models of all the systems and humans participating in the interaction (Seel, 1990). Agents are entities capable of voluntary, rational action carried out in order to achieve goals and holding a representation or ‘belief’ in the state of the world. They come to hold these beliefs through existing data and by deriving new belief from interaction with external sources and as a result of internal reasoning. Issues which become evident here are those concerned with a number of interacting agents. In more complex systems agents may be interacting in a number of ways, modelling other agents and adapting and responding to a variety of needs. Multiple agent systems incorporating surrogates to filter routine tasks and provide personalised ‘theatrical metaphor’ interfaces are cited as the wave of the future (Negroponte, 1989) and simple agent-based interaction is a prerequisite of co-operative dialogue, between human or computer as in the interface agents which provide ‘expertise, skill and labour’ described by Laurel (Laurel, 1990). Such developments can be seen an extension of dialogue assistants (Alty and McKell, 1986; Alty and Mullin, 1987; Coutaz, 1989). Essentially agents are adaptive systems, but systems which are specialised and know about only a very small part of the world. Much of the work presented here is highly relevant to the design of agent-based systems. An important issue for cooperative or support systems is, firstly, how knowledge is actually represented (both task specific details and knowledge about the user or co-operative partner) and, secondly, how the interaction or conversation can be realised. Co-operative support systems can also be thought of as task-orientated dialogue systems, which are actually characterised by the ‘conversational roles’ which each partner is expected to adopt. Active dialogue partners in mixed-initiative dialogues are those which try to identify a user’s intentions in order to exhibit co-operative behaviour. To behave co-operatively, the system must discover the presumable plans underlying the user’s question or statement; represent those plans in its knowledge base; examine them for hidden obstacles and provide information which overcomes those obstacles. In keeping with the needs of ITS, active help systems and more general intelligent support systems, agent-based interaction requires the computer partner to maintain explicit models of a user’s goals and plans and to have sufficient information about prior knowledge and about known page 8
misconceptions or errors in the user’s current state of knowledge. Implementations of intelligent support systems must cope with problems of scale in deciding upon relevant and irrelevant information and with uncertainty in time relationships or object selection. There must also be mechanisms in order to allow the planner to recover when assumptions are unjustified or conditions change. 2.7
Summary
Although the system categories described are quite different and work has taken place from different disciplines, with the researchers employing different techniques and specialised language to explain their concepts, they seem to share an underlying architecture. All are adaptive systems in that they automatically alter aspects of the system to suit the requirements of individual or groups of user - or more generally to suit the needs of other agents in the system. All have to infer characteristics of the other agent from the interaction. Work on active help, ITS, NLS, explanation-based and cooperative agent systems contribute to the confusion because they describe systems from distinct perspectives. We need to consider how adaptive systems are designed and how the adaptive functionality can be incorporated into existing architectures. 3.
AN ARCHITECTURE FOR ADAPTIVE SYSTEMS
In this section we present a description of the components which all adaptive systems must have. Clearly the complexity of the system and the requirements of the application have an impact on the detail contained in each component. The advantage of developing a generalised architecture for adaptive systems is that it enables researchers to talk in the same language, to compare different systems and to develop appropriate representation techniques. In order to provide a perspective for the following discussion, we may represent the overall structure of an adaptive systems as shown in Figure 1. An adaptive system has a model of the system with which it is interacting. Often this other system will be a human and hence we refer to this representation as the user model. An adaptive system also includes some representation of the application which is to have the adaptive capability. This is the domain model. The interaction of user and system is described in the interaction model. Each of the three components of Figure 1 will be considered in detail, relating their contents to the typology of adaptive systems identified in Section 2. Our aim here is to draw together the strands of research identified in Section 2 and to concentrate on an elaboration of the components of an adaptive system. We do not concern ourselves with the run-time issues which are central to user interface management systems (UIMS), nor to the details of various implementations. We seek to develop a conceptual model of adaptive systems. The design-time issues which are the concern of such page 9
devices as user interfaces design environments (UIDEs), user modelling shells and adaptive system development environments are discussed in Section 4. Model of the other system. The 'User Model'
Model of the system itself. The 'Domain Model'.
Model of the user-system interaction. The 'Interaction Model'
Figure 1 overview of an adaptive system 3.1
The User Model
A user model is a representation of the knowledge and preferences which the system ‘believes’ that a user (which may be an individual, a group of people or a non-human agent) possesses. It is separable by the system from the rest of its knowledge and contains explicit assumptions about the user. The user model is used to provide adaptivity either by intervention or by co-operative agreement with a user. Providing adaptive functionality requires that a user model controls an inference engine and often implies that it samples user behaviour. It may infer perceived goals and courses of actions and act upon such decisions, altering features of the interaction to meet the task and personal needs of individuals. User models may be highly pragmatic in that they represent only what is required in order to facilitate the required adaptation. Other user models may be orientated towards a realistic description of the user. Murray (1987a; 1987b) refers to these as ‘Embedded User Models’ (EUMs). In contrast to mental models or designers’ models, which are intangible, user models are representations of user features and decisions which are accessible by the system software. The knowledge represented in the user model may be acquired implicitly from inferences made about the user or it may be explicitly elicited from the user. Explicit acquisition may be achieved through some co-operative behaviour such as asking relevant questions. Alternatively, the user model may be fed with previously stored information, held on whatever medium may be appropriate or transportable. The use of user records, profiles or scores in a personal data store such as a ‘smart’ card is one mechanism to allow fullscale use of adaptive systems with explicitly acquired data (Murray, 1989). This, however, highlights the need for absolute determination and ratification of some of the moralistic problems we discuss below. page 10
Knowledge for the user model can be acquired implicitly by making inferences about users from their interaction, by carrying out some from of test, or from assigning users to generic user categories usually called 'stereotypes'. The notion of user stereotypes derives from the work of Elaine Rich (Rich, 1981; 1983; 1989). Stereotypes represent a structured collection of traits or characteristics, stored as facets, to which is attached a value, and optionally a confidence-level and rationale. Some traits are triggers and have an attached probability rating which can mediate or inhibit firing of a whole stereotype. They can be used to '…provide a way of forming plausible inferences about yet unseen things on the basis of things that have been observed' (Rich, 1983). Stereotypes model users on a variety of dimensions and represent characteristics of users in a hierarchy. At the top of the hierarchy is the ‘any person’ stereotype which defines the characteristics relevant to all users of the system. All stereotypes at lower levels in the hierarchy may inherit the characteristics of this stereotype. Lower level stereotypes will depend on the application, but retain the property of inheriting characteristics from parents. At the bottom of the hierarchy is the individual who may inherit characteristics from a large number of stereotypes. One of the problems with such a representation is to decide just what happens when conflicting characteristics are inherited. Conflict resolution rules must then be included to deal with such situations. Other mechanisms may be employed to infer user characteristics. In terms of our adaptive system architecture, it is the interaction model which is primarily responsible for acquiring implicitly knowledge about users. We deal in detail with these mechanisms in Section 3.3. Rich (1989) describes the space of user models using two dimensions. The first, canonical versus individual, describes whether the model is of one single user or a collection of models of different individuals. A canonical model represents the 'typical' user and is not usually stored explicitly in the system. It is the designer's model identified earlier. Rich also distinguishes long-term models from short-term models. Long-term models are representations of fairly stable user characteristics such as expertise or interest. Short-term models are employed in transitory problem-solving behaviour for the specific task in hand, focusing on specific topics and goals in the immediate interaction. This distinction is also referred to as local (short-term) user models versus global (long-term) user models. Sleeman (1985) provides a more detailed description of short-term individual models with reference to ITS. He describes student modelling techniques as scalar, ad hoc, profile, overlay and process. The profile model encompasses a stereotype model while the others correspond to descriptions of the mechanisms of modelling student knowledge and inferring the state of that student’s learning. Overlay models are particularly pertinent to ITS. Here the expert's knowledge of the domain is represented and the student's knowledge is captured as a page 11
subset of that knowledge - overlayed on the expert's knowledge. In this way, aspects of the expert's knowledge which the student does not possess stand out. The problem with this approach is that the student may believe things about the domain which the expert does not. Perturbation models (Kass and Finin, 1989) can be used to represent beliefs which are outside the expert's view of the domain. 'Buggy' models (Burton 1982) and 'mal-rules' (Sleeman, 1981) may be used to represent common misconceptions which users may have about the domain. Short-term individual, or local user models are of particular concern in natural language (NL) systems. The problems of inferring users intentions or their focus of attention from some natural language statement requires that the system models the relevant part of the preceding discourse. NL systems tend to encode a lot of linguistic knowledge concerning how and where anaphoric and elliptic references occur. Inferring the focus of attention of the user is similarly important in such systems. In these cases long-term user models are not required, but short-term use of language is central. Intelligent interfaces on the other hand have to deal with fundamental cognitive characteristics such as users preferences for particular styles of display, basic cognitive capabilities such as spatial ability and preferred learning styles (van der Veer, 1990; Benyon, 1993b). In these systems longterm, cognitively valid models are vital. Similar cognitive capabilities become central to intelligent support systems and explanation systems (Finin, 1989) where the requirements of the system demand that help or explanations take into account users' existing knowledge and the intention which they have in asking for advice. Autonomous agent systems have the added complication of representing 'second level' beliefs. They must not only infer what they believe the other system believes, but must also base their adaptation on what they believe that the other system believes that they believe. User models may contain one or more of three types of knowledge of the user. Firstly, the user model may hold data about what the system believes the user believes about the domain. It is domain dependent data. Because of the similarity of this data to that held by ITS, we refer to this portion of the user model as the student model. Student model data may be kept at one or more of three levels; the intentional, or task, level, the logical level and the physical level. (The reasons for identifying these levels are explicated in Section 3.2.) The task level describes user goals in the domain. For example, an intelligent interface to an information retrieval domain may need to infer that the user is attempting to discover how many items meet some criteria rather than actually trying to obtain a list of those items. Failure to recognize the intentions underlying some user action will result in a less satisfactory interaction. An intelligent support system may need to infer that a user is trying to display the contents of a directory and not list the content of a file in page 12
response to a query such as 'how do I use ls to display my directory'. A natural language interface should be able to infer that the user has misunderstood a concept and requires further elaboration of that concept rather than simply responding automatically to some syntactic analysis of the dialogue. A second level of description of user knowledge of the domain is the logical level. Here the system records what it believes the user understands about the logical functioning and the logical concepts embodied by the domain. For example, an intelligent interface should be capable of recognising that attempting to execute a database language 'Select' statement in response to a help system prompt is a logical, or semantic error (given that the help system cannot execute database language statements) rather than a syntactic error of attempting to obtain help on the 'Select' statement. Finally the system records the user's (inferred) knowledge at the physical level. For example. an intelligent help system must recognize when a user understands the semantics of a command but has forgotten the syntax if it is to provide appropriate advice. At each of these levels the user model should record the user knowledge and the user's erroneous beliefs. Domain independent data may be considered either as fundamental psychological data or as profile data. Psychological data is concerned with essential cognitive and affective traits of users and is held in the psychological m o d e l component of the user model. There is an ever increasing body of experimental evidence which confirms that users differ in cognitive skills and personality traits which significantly affect the quality of certain interaction styles and user requirements (van der Veer, 1990; Egan, 1988; Jennings and Benyon, 1992). These characteristics of users are particularly resistant to change by the user and hence are particularly important for adaptive systems. If users find it difficult or impossible to change aspects of their make-up, these are exactly the characteristics to which the system should adapt (van der Veer; 1990, Benyon, 1993b). Spatial ability is a characteristic which a[[ears particularly relevant to HCI (Vicente and Williges, 1988; Vicente, Hayes and Williges, 1987; Egan, 1988), particularly where users have to navigate through the conceptual space of file structures or system modes. We know of no system which adapts to users' affective states, but clearly interface agents will have to represent these characteristics if they are to fulfil their promise (Kay, 1990; Laurel, 1990). Data concerning the background, interests and general knowledge of users is held in a user profile component of the user model. This data is not psychological in nature, but may interact with cognitive characteristics in a number of ways. For example, users with poor spatial ability may be able to deal effectively with an interface style if they have a certain level of experience using that style (Jennings and Benyon, 1992). Knowledge of generic applications is stored in the user profile as is much of the stereotype-inherited data such as being a business traveller (Morik, 1989) or feminist (Rich, 1983).
page 13
Student Model
Psychological Model
User Profile
User Model
Figure 2 Main components of a user model The user model thus consists of the three interlinking components shown in Figure 2. The student model component is created directly from the domain model (Section 3.4). Both the psychological and profile components have to be represented explicitly, preferably using some user modelling software (see Section 5). All aspects of the user model will require considerable prototyping, evaluation and refinement before they capture appropriate and salient aspects of users with sufficient accuracy for a given domain. 3.2
The Domain Model
The user model is required in an adaptive system so that it can alter aspects of the system in response to certain inferred or given user characteristics. The domain model is required in order to define the aspects of the application which can be adapted or which are otherwise required for the operation of the adaptive system. Other terms which have been used for this concept include application model, system model, device model and task model. The domain model will serve a number of purposes. Firstly, it forms the basis of all the inferences and predictions which can be made from the user-system interaction. It is important therefore that the model is at an appropriate level of abstraction to allow the required inferences to be made. There may be mechanisms which, as a result of some observed behaviour or stated characteristic, predict that some problem will occur or infer a user’s attempt to achieve some goal. In order to do make these inferences the system must have an appropriate representation of the domain. For example, in UC the fact that a user knows the command r m is used to infer that the user knows ls as well. This is only possible because UC has a domain model which specifies a certain relationship between the r m and ls commands. In TRACK (Carberry, 1989) the domain model includes sequences of actions (plans) which are required to achieve a particular goal. This model is used to infer the user's goal from their observed actions. In addition to inferences, the domain model forms the basis for all the adaptations which the system can make. The system can only change aspects of the application which are described by the domain model. In HAM-ANS (Morik, 1989), for example, the domain model contains a representation of page 14
hotels which includes ‘quietness’ as an attribute. The system exploits this representation in making a recommendation of a hotel to a particular user, by emphasizing that a particular hotel is quiet. This adaptation is only possible because the domain model contains this attribute. In TAILOR (Paris, 1989) the domain model represents two levels of description of components of complex devices such as telephones so that it can provide appropriate explanations for different users. Any system which is capable of evaluating its own actions also requires a domain model. The domain model holds the characteristics of the application which are measurable, so that they can be evaluated for effectiveness against the required criteria. The final use of the domain model is to form the basis of the student model component of the user model. The system needs to record what it believes the user believes about certain aspects of the application. The domain model must describe the system so that it can store data about the user's understanding of the various concepts and functions in the application. The Domain Model consists of one or more abstractions of the system. These abstractions allow the adaptive system to reason about the target application, to facilitate adaptations to other agents, to evaluate its effectiveness and to supply the student model part of the user model with its content. If the system is to be capable of adapting the screen displays then these must be described in the domain model. If it is to adapt the functionality of the system, then the domain model must represent alternative functional capabilities and the relationship between functions. Similarly if we want the system to alter the description of concepts then these too must be modelled. If the system is to infer user goals, then goals and plans must be explicitly modelled. In most adaptive systems the domain model is implicit. The representations are embedded in the system code or are only available after a significant amount of processing. For example, in Grundy (Rich, 1983) the classification of books as suitable for particular types of people can only be obtained from the stereotype user models. There is no explicit representation of the domain. Chin (Chin, 1986) recognised the restrictions of this in the UC system and categorised domain knowledge into stereotypical levels of difficulty. Hence, ‘simple’ concepts in Unix include the commands r m , ls and cat and the concept of a file, ‘mundane’ commands include v i and diff. Other commands are classified as ‘complex’ or ‘esoteric’. The benefits to be gained from having an explicit and well-defined domain model are considerable and have long been recognized in AI (e.g. Buchanan, 19xx). A separate domain model provides improved domain independence which means that refining the domain model is much easier. This is most important as it is unlikely that any adaptive system design will have a perfect representation of the domain at the first attempt. A separate and explicit domain model is more easily used for multiple purposes such as providing explanations of the system’s behaviour. page 15
We see the domain model as a description of the application which contains facts about the domain, i.e. the objects, their attributes and the relationships between objects. The domain model is the designer’s definition of the aspects of the application relevant to the needs of the adaptive system. A central question in constructing a domain model is deciding what level of description should be represented. Since the domain model forms the basis of the student model component of the user model, it is important that the domain model is to a large extent cognitively valid. That is, it should capture a view of the domain appropriate to human information processing. One of the areas to consider a realistic cognitive representation of computer systems is the work on Task Analysis techniques (Wilson, Barnard, Green and Maclean,1988; Diaper, 1988; Payne and Green; 1989, Bösser, 1987). These are formalisms which attempt to represent some aspect of the application. In a critical review of cognitive task analysis, Benyon (1992a) identified three levels of description provided by these techniques. The user has something which s/he wishes to achieve outside the computer system (e.g. produce a letter). This goal is then translated into a number of tasks which have to be performed using a specified device such as a particular word processor (e.g. start the word processor, edit the letter, print it). Tasks are decomposed into a hierarchy of simpler subtasks. These can be further decomposed through several levels until some ‘simple task’ (Green and Payne, 1989) or ‘unit task’ (Card et al., 1983) is reached. These simple tasks are those which can be characterised by having no problem solving or control structure component. Simple tasks may be different for experts and novices. For example, the edit task is decomposed until we get to the level of a simple task which is to move the cursor one character forward. This general model of task analysis is presented in Figure 4 and the focus of various task analysis techniques is shown in Table 1. Nielsen (Nielsen, 1986) provides a similar argument concerning different representations. He focuses on the difference between the "invisible layers" (conceptual features) of the application concerning the goals, tasks and semantics of the system and the "visible layers" which are concerned with perceptual and physical features.
page 16
External Task or Goal
Mapping
(Internal) Tasks Device Independent
Device Dependent
Mapping
(Physical) Actions
Figure 4 Generalised Task Analysis Model
GOMS
Goal level
Internal Task level Physical Action level
Goals
Operators
Methods
Tasks
Actions
TAG ETIT
External task
Internal task
Payne
Problem space
Device space
CLG
Task level
Semantic level
Nielsen
Invisible layers
Syntax and lexical levels Visible layers
Table 1 Emphasis of some task analysis techniques This model reflects the abstraction/presentation dichotomy of the software architectures (Benyon and Murray, 1993, see also IEEE Software 1989), but also includes the goal or external task level of description. Goals are concerned with what the system can be used for. They are external to the system. A central concern of HCI is how they can be mapped onto the functions which are available in the application. Internal tasks on the other hand are high level, logical descriptions of the activities which have to be done in order to achieve that goal. Tasks are dependent on the design of the system and on the functions which are available. For example, the goal of producing a letter is mapped onto the tasks page 17
of start the word-processor, edit the letter, print it, etc. However, in some systems in may be unnecessary for the user to start the word-processor if the application has been configured in a particular way. Tasks may be allocated to human or machine depending on the facilities available. The physical actions are the actual keystrokes, mouse clicks, cursor movements and so on necessary to enact the tasks. These are dependent on the style of the presentation. These three levels are echoed in Rasmussen's consideration of mental models and HCI (Rasmussen, 1986; 1987). Based on his experience of technicians and operators in complex systems such as power plants, Rasmussen proposes that five levels of abstraction are required to capture the problem solving behaviour. Models at low levels of abstraction are related to a specific physical world. The physical form of an object is the lowest level used. The next level up concerns physical functions. Humans model these in a language related to their physical properties (e.g. electrical, mechanical, etc.). These two levels relate to the physical actions of the task analysis model (Figure 4) and the presentation level of the UIMS model. Rasmussen's generalised function level involves models of the function of system objects without concern for their physical arrangement. An engineer recognises that some object is a multiplexer, say, and therefore it performs certain functions. The multiplexer may be physically realised in a number of configurations of actual physical components. This level is a description which is unconcerned with the physical arrangement (the syntax) of a particular system. It is the application level of the UIMS or the (internal) task level of the task analysis model (Figure 4). Above the level of generalised function is the level of abstract function. Here the model is expressed in a language which depends on universal laws and symbols and not on the local physical and functional properties. The human’s mental model at this level can only be formed by considering the purpose or reasons behind a system structure. At the highest level of abstraction, the system may be modelled in relation to its purpose within its overall environment. These two levels thus concern the goals which the system can be used for - the system’s purpose. Rasmussen argues that in human systems higher level functions are naturally derived from the purpose of the system whereas physical systems are responding to the laws of nature. Physical systems are causal systems as opposed to intentional systems. Rasmussen's model is similar to the philosophical arguments of Pylyshyn (Pylyshyn, 1984) and Dennett (Dennett, 1989). Pylyshyn argues that what ‘might be called the basic assumption of cognitive science...[is] that there are at least three distinct, independent levels at which we can find explanatory principles....biological, functional and intentional’ (Pylyshyn, 1984, p.131, Pylyshyn’s italics). He relates these terms to those used by other writers. For example, Newell (Newell, 1982) calls them the device level, symbol level and
page 18
knowledge level. Pylyshyn himself prefers the terms physical, symbol and representational (or semantic) level. The levels are distinguishable from each other and necessary because they reveal generalisations which would otherwise not be apparent. A functional description is necessary because different functions may be realised through the same physical states and different physical states may realise the same function. Although Pylyshyn is primarily concerned in this discussion with human cognitive systems, it is apparent that both physical and functional descriptions are appropriate for computer systems also. For example, the physical action of pressing ^D will result in the application performing different functions depending on the system. Similarly, the function of moving a cursor can be accomplished in a variety of different ways. The semantic, or representational level of description is required because certain principles or constraints are revealed. We interpret behaviours of systems not only through function, but also through relating function to purpose - by relating the representations of the system to external entities. The purely functional view of someone dialling 911 (or 999 in the UK), does not reveal that that person is seeking help. There is no simple causal connection between dialling a certain number and the seeking of help, it is a semantic, or representational relation. It is this level - of intentions on behalf of the user of a system - that also needs describing. We need a representation of the goals, ‘beliefs’ and ‘desires’ which the system has Dennett also recognizes three levels of description. We can understand the behaviour of complex systems by taking a physical view, a design view or an intentional view. The physical view, (also called the physical stance or physical strategy) argues that in order to predict behaviour of a system you simply determine its physical constitution and the physical nature of any inputs and then predict the outcome based on the laws of physics. However , sometimes it is more effective to switch to a design stance. With this strategy, you predict how the system will behave by believing that it will behave as it was designed to behave. However, only designed behaviour is predictable from the design stance. If a different sort of predictive power is required then you may adopt the intentional stance. This is summarized as follows; 1. Treat the system as a rational agent 2. Figure out what beliefs it ought to have given its place in the world and its purpose 3. Figure out what desires it ought to have 4. Predict that this rational agent will act to further its goals in the light of its beliefs and hence 5. Predict what the agent will do on the basis of what it ought to do. It is clear that the intentional stance provides a useful strategy for understanding humans and for predicting what will happen. It is therefore,
page 19
an important part of the domain model which we require since that model will form the basis of our inferences and predictions of future behaviour. The conclusion of this analysis suggests that a cognitively valid domain model should capture descriptions of the application at three levels which we shall refer to as the • Task level • Logical level • Physical level.
At each of these levels, the structure (the objects and relationships which exist) and the processing of the application need to be described. A comparison of the various domain representations which we have considered is shown in Table 2. Our model
Task level
Logical level
Dennett
Intentional stance Design stance
Physical stance
Pylyshyn
Representational level
Symbol level
Physical level
Rasmussen
Purpose and Abstract function
Generalised function
Physical form and Physical function.
Task Analysis
External Task or Goal
(Internal) Task
Action
Abstraction level
Presentation level
User Interface Software
Physical level
Table 2 Comparison of domain levels The task level describes the application from the perspective of what it can be used for and the general components or subsystems which it has. This is the level at which a system is related to external entities, to achieve goals in the outside world. The task level is important because the user needs to be aware that (for example) an e-mail system can be used to send messages and that it can be used for elementary word processing, but that it is not ideally suited to that purpose. Unless users understand the purpose of a system they will be left frustrated by many of its shortcomings. The logical level of the domain model equates with the semantic (in task analysis terms) or functional level. It is the level of generalised functions. The term ‘logical’ is used here because it emphasizes that in order to achieve some purpose, certain functions h a v e (logically) to be performed and certain objects h a v e to exist. This is the design stance where the system is described in logical functions/concepts; a description which concentrates on how something works. In agent-based interaction this level is particularly important. Logically
page 20
something has to be done or some data has to be provided in order to fulfil a purpose (Benyon, 1992b). Whether it is done by human or agent is a design decision. The physical level provides a causal description concerned with how simple tasks are sequenced and how objects and displays are laid out. It is concerned with the presentation of the system, dialogue control and physical actions. The domain model must also describe the mappings between the levels. It does not contain everything about the application. The domain model represents the aspects of the application which are to be used in providing the adaptive capabilities. 3.3
The Interaction Model
The third component of an adaptive system is its representation of the actual and designed interaction between user and application - the interaction model. This use of the term is thus very different from the interaction model proposed by Norman (1986) which is a theoretical representation of humancomputer interaction in general. It is much closer to the notion of a discourse model, or dialogue model. However, there is still much debate over exactly what is meant by a discourse model and what type of data it contains (see the discussion in Computational Linguistics, vol. 14(3), 1988). We will remain with the term interaction model and consider its relationship with discourse models in the Section 3.4. An interaction is a user making use of the system at a level which can be monitored. Data gathered from monitoring this interaction can be used to make inferences about the user’s beliefs, plans and/or goals, long-term characteristics, such as cognitive traits, or profile data, such as previous experience. The system may tailor its behaviour to the needs of a particular interaction or, given suitably ‘reflective’ mechanisms, the system may evaluate its inferences and adaptations and adjust aspects of its own organization or behaviour. In some other representations (Alty and McKell, 1986) the interaction model is seen as a part of the domain model. However, we like to see the components explicit and separated from other parts of the architecture because of the gains which arise from understanding the roles of the different models. For example, Alty and McKell's application model can trap errors. However, an error can only be understood with respect to some notion of what is correct at some level of description. The definition of an error belongs with the representation of domain concepts which is rightly kept in the domain model. It is the role of the interaction model to identify and deal with errors. There are two main aspects to the interaction model; •
capturing the appropriate raw data
page 21
•
representing the inferences, adaptations and evaluations which may occur
Raw data is obtained through maintaining a dialogue history or dialogue record (DR) which is a trace of aspects of the user’s observed behaviour. The dialogue record kept for as long as is required according to the needs of the adaptive system and is then deleted. It may contain details such as the sequence of keystrokes, mouse clicks and mouse movements made, timing information and system messages. It is an abstraction of the interaction since it cannot capture everything which takes place. A portion of the dialogue record from the Basser data project (Thomas, Benyon, Kay and Crawford, 1991) is shown in Figure 5, with an accompanying interpretation. This project is monitoring the use of the editor sam (Pike, 1986) by over 1000 users at the University of Sydney. The purpose of the dialogue record is to enable inferences to be made about how commands are learnt, how they are used, the difficulties which different groups of users have and so on. The dialogue record contains the mouse movements and other commands used by the users. Mouse commands map directly to menu selections and so the data can be sensibly interpreted. However, it was not possible to record various aspects of the interaction such as the size and shape of windows created or the actual position of the mouse on the screen. Other aspects of the interaction left out of the dialogue record were such information as the command arguments entered. Similarly it was decided to record only the total number of keystrokes and backspaces entered when inserting text. The dialogue record is an abstraction of the actual dialogue suitable for the purpose at hand. Although the dialogue record contains only low level data, it is surprising how much can be inferred if the knowledge necessary is available (i.e. provided by a human or stored in a domain model). For example, the command sequence MdMe (position and cut) indicates a possible error since there is no ‘drag’ (or ‘highlight’) operation which indicates the data to be cut. A timestamp (code T) is output every five minutes which shows the total number of characters inserted. This allows us to calculate how many were typed into a command window even though such data is not recorded directly. McMeMdMeMfMdMcMeMdMdMcMe I2 0 3 Md I1 0 10 MaAlA,A$KsAlA,KsEvAlA,KsEvAlA,KsEvM} T667538030 19 292 M{M{M{M}AlA,A$KcKxKwWf
page 22
Explanation
Monitor Output
Drag out (highlight) some text
Mc
Cut
Me
Position and cut (NB no drag means no change in the file).
MdMe
Paste
Mf
Position, drag, cut
MdMcMe
Position, position, drag, cut.
MdMdMcMe
Insert 2 characters, no backspaces in 3 seconds
I2 0 3
Position and Insert one character in 10 seconds.
Md I1 0 10
Select command window
Ma
Substitution command
AlA,A$Ks
Substitution command
AlA,Ks
Substitution error displayed
Ev
Two more attempts with errors
AlA,KsEvAlA,KsEv
Scrolling up and down in command window with timestamp output showing time,
M}
consistency and 292 characters typed.
T667538030 19 292 M{M{M{M{M}
global substitution
AlA,A$KcKx
Write out file
Kw
Warning message occurred
Wf
Codes indicate commands e.g. M} is mouse button 3, Me is select ‘cut’ from pop-up menu, Ks is command ‘search’, addresses e.g. ‘A,’ is ‘to end of file’, warnings, errors, etc.,
Figure 5 Portion of the dialogue record from the Basser data project, with associated explanation. The second part of the interaction model is a description of the 'stereotyped' interaction; the Interaction Knowledge Base (IKB). This describes the inferences that can be made from the dialogue, the evaluations of the interaction which are possible and the changes (adaptations) which the system can accomplish. The user model and domain model define what can be inferred. The IKB actually does the inferencing by combining the various domain model concepts to infer user characteristics or by combining user model concepts to adapt the system. The IKB represents the relationship between domain and user characteristics. It provides the interpretation of the dialogue record.
page 23
For example, given the dialogue record in Figure 5, an attribute in the student model part of the user model KSub = knowledge of substitute command
values {1...5}
and definitions in the domain model for Ks (substitution command), Ev (substitution error), Kc (change command), Kx (global substitution) the IKB might contain a rule IF KsEv OR KsEv occurs in dialogue > 3 times AND KcKx does not occur THEN KSub = 1 OR KSub = KSub - 1 which infers the level of a user’s knowledge of the substitute command (by noting the errors resulting from Ks) and updates the user model appropriately. Adaptations may be accomplished in a similar fashion. For example IF KSub < 2 AND KsEv occurs in dialogue THEN SubHelp which calls a routine (SubHelp) to offer advice on the substitute command if users have a low value of KSub (knowledge of substitute command) and if they make a substitution error. Notice that SubHelp and any ensuing interaction now becomes part of the dialogue record. A sophisticated adaptive system may subsequently evaluate its own recommendation by analysing the dialogue record further and performing some check and repair activity (a routine called Check) on the SubHelp routine. IF SubHelp has been called AND KsEv occurs in dialogue THEN Check SubHelp The interaction model is a vital part of an adaptive system. The developer of adaptive systems has to decide the level of abstraction which is required for the dialogue record, the individual user data and the interaction knowledgebase. In our approach, the interaction model must be constrained so that all attributes used in the representation are defined either in the domain model or in the user model as appropriate. Automatic mechanisms are required for this through software support (Section 5). 3.4 Relationship between the models We have described a conceptual structure for adaptive systems which consists of three models; the user model, the domain model and the interaction model as illustrated in Figure 6. Although the discussion has generally been as though there is only one of these in each adaptive system this may not be the case. page 24
Psycholgical model
profile model
task level
student model user model
logical level
physical level domain model
dialogue record evaluation mechanisms
inference adaptation mechanisms mechanisms interaction knowledge base interaction model
Figure 6. Overall architecture for an adaptive system For example, in a given system there may be several user roles and hence several user models. The focus of the system may be about a person. In general there may be a large number of user models representing the individual agents in a multi-agent co-operative system. Similarly there may be more than one domain model. Conceptually our approach is that we want to model users, or agents who are characterised by having one sort of data the psychological model, personal profile and the student model - and the domains which are characterised by having another sort of data - the task, logical and physical levels of description. The interaction model as expressed through the adaptive, inference and evaluation mechanisms may be extremely complex, embodying theories of language, pedagogy or explanation. Morik (1988) emphasizes the difference between the interaction-independent theory of language and the interactiondependent nature of the actual dialogue. This reflects the distinction which we have drawn between the dialogue record and the mechanisms in the IKB which use that data. In general, the interaction model will represent the strategies and theory of the particular type of system. 4.
METHODOLOGY FOR ADAPTIVE SYSTEM DEVELOPMENT
We take the view that whether or not a system should have an adaptive capability is essentially a human-computer interaction (HCI) activity. Alternatives to adaptive systems will often exist for system developers including training users in the use of a system, the use of other support mechanisms and customisation facilities. The suitability of each of these solutions will depend on the nature of the system, the environments in which it will be used and the characteristics of its users. Adaptive systems are one solution to usability problems (Benyon, 1993a). However, we do expect page 25
that incorporating more knowledge into the interface and providing systems with an adaptive capability will become an increasingly attractive option for designers. As an HCI activity, developing adaptive systems shares problems with all HCI development. Fischer (Fischer, 1989) summarises these as; •
there is no real theory of HCI
•
current ‘life cycle’ models are inadequate because of the ill-structured nature of HCI problems
•
there is a great degree of ‘design instability’ in HCI and hence the need to evaluate and re-design systems
•
requirements cannot be fixed and used as the basis for deriving a formal design equivalent
Hartson and Hix (Hartson and Hix, 1989) also emphasize that designers take an ‘alternating waves’ approach to design rather than a strict top-down approach and that evaluation becomes central. HCI needs to deal with cognitive design issues which do not have any generally accepted notations. These problems of designing effective HCI systems have led various researchers to suggest alternatives. These are generally characterised by •
a changed view of the systems life cycle
•
an approach to systems development based on prototyping, evaluation and iteration
•
the need for effective software support tools
We concur with the spirit of these suggestions and the current discussion should be seen in this context. However, the fact that text is a linear medium can make descriptions of iterative processes difficult. Readers should bear in mind that the textual description inevitably hides the process of continual refinement and iteration which underlies systems development. We deal with the important role of software tools in Section 5. 4.1
A general approach to HCI development
The alternative to the traditional life cycle recommended in (Hartson and Hix, 1989) is a star representation which has evaluation as central. We have changed the labels in some of the components in order to correspond to our terminology which is explained below. The star model is shown in Figure 7, but it is important to remember that all the activities are highly interdependent and that two or more activities may occur at the same time.
page 26
Implement Systems Analysis Prototype
Evaluate
Design
Specify User Requirements
Figure 7 The star approach to interactive systems development (adapted from Hartson and Hix, 1989) Systems analysis Systems analysis is the process of understanding the problems of any current system and the requirements for any replacement system. Systems analysis inevitably involves some design considerations. It consists of five interrelated and interdependent activities •
functional analysis which aims to establish the main functions of the system.
•
data analysis which is concerned with understanding and representing the meaning and structure of data in the application. Data analysis and functional analysis go hand in hand to describe the information processing capabilities of the system (Benyon, 1990)
•
task analysis which focuses on the cognitive characteristics required of the users by the system e.g. the search strategy required, cognitive loading, the assumed mental model etc. Task analysis is device dependent (Benyon, 1991) and hence requires some design to have been completed before it can be undertaken.
•
user analysis which determines the scope of the user population which the system is to respond to. It is concerned with obtaining attributes of users which are relevant to the application. This includes aspects of each of the components of a user model identified in Section 3.1.
•
environment analysis which covers the environments within which the system is to operate. This includes physical aspects of the environment and ‘softer’ features such the amount and type of user support which is required.
page 27
Specify User Requirements During the development process, user requirements will emerge. There are three aspects to the user requirements; •
functional requirements describing what the system must be able to do
•
data requirements describing the data which the system is to store and manipulate
•
usability requirements which specify the effectiveness, learnability, flexibility and attitude (Shackel, 1990). Design
In line with the arguments presented in Section 3.2, the design of a system can be seen as consisting of three levels of description - task, logical and physical - and the mappings between them. Once the requirements have been obtained at a suitable level of detail, the application can be specified at these three levels. The application may be represented in terms of structure and/or functions at each of these levels and any of a number of notations may be used. •
the task level describes the (external) tasks, or goals which the system is to support
•
the logical level describes the logical functioning and structure of the application.
•
physical design is concerned with the layout of screens and the construction of icons, etc. (the representational aspects) and with the operational aspects (how objects behave)
•
the task/logical mapping describes the cognitive processing required by users as they translate their goals into the system’s functionality
•
the logical/physical mapping describes the consistency, learnability and other aspects of the actual system use.
The transition from logical to physical design is particularly important since it is here that functions are allocated either to the human or to the machine and it is here that an adaptive capability is most likely to be considered. A decision to be made by designers is that if a function is allocated to the user, it will require them to face additional cognitive processing, but if it is allocated to the machine, then the system will require an adaptive capability. Prototyping The feature of any prototype is that it does not have a generalised lifetime. It may be built and thrown away or it may evolve into the final system.
page 28
Prototypes must be quick, easy and cheap to construct and hence rely on the provision of good software tools. Implementation This is concerned with completing the system, programming it in an implementation language, thorough testing and other quality assurance processes, acceptance and completion of the documentation. Evaluation Frequent evaluation is central to the methodology. It may take the form of a very quick discussion with users or a formal review procedure. It may require a lengthy experiment or it may demand the skills of an HCI expert. It is also important to recognize that evaluation is applicable at all stages of development - analysis should be evaluated as well as requirements and design. Evaluation may take place in order to help generate ideas or to clarify details later in the development process and thus be primarily formative rather than summative. 4.2
Adaptive system development
Section 4.1 describes the development of any interactive system and the issues which must be addressed. However, adaptive systems differ from other interactive systems in that they are characterised by design variety. Rather than the designer trying to obtain a single solution to a problem, the designer specifies a number of solutions and matches those with the variety and the changeability of users, environments and tasks. At some point in the system development process, the designer may decide that the system requires an adaptive capability. This may be to meet a specified user requirement, it may arise from the analysis or it may happen during the transition from a logical to physical design. Once this option has been identified, the adaptive system must be developed in parallel with the application and using the same development approach. The specification of the adaptive system part of the application requires the specification of the domain, user and interaction models. This in turn requires the designer to focus on what features of the system are to be adaptive, what characteristics of the users it will adapt to and how the adaptations will be accomplished. It may be that the whole system is to be adaptive. In this case, the domain model and the system design are the same thing. However, usually only a part of an application will require an adaptive capability and so it is this part which must be isolated and specified in the domain model. We expect there to be considerable experimentation and refinement during the development of the adaptive component of a system and that software support is vital to the success of adaptive systems (see Section 5). page 29
An important issue in adaptive system development is ensuring that the adaptivity can be controlled and measured and that the reasons for having an adaptive mechanism are known and clear. We believe that the architecture and methodology presented here contribute to this. However others (Browne, Totterdell and Norman, 1987, 1990) suggest that metrics are necessary in order to control the development process. Their experience of developing a number of prototype adaptive systems in the UK led them to the conclusion that the whole issue of how, where and when to adapt needed to be explicitly stated. They recommend that there should be the following measures; •
an objective metric which describes the purpose of the adaptive facility
•
a trigger metric which identifies what will trigger the adaptive mechanism
•
an assumption metric which describes the rationale behind the objective metric
•
a recommendation metric which describes what recommendations are possible
•
a generality metric which describes the generality of the findings
•
an implementation metric which relates to the effect that the adaptive mechanism will have on the implementation.
The notion of metrics in HCI is currently receiving much of attention, but is still in its formative stages. We generally accept that metrics are an important development in HCI and that they have a role to play in adaptive systems. However, we believe that it is unnecessary to have separate adaptivity metrics. Adaptivity is a usability issue and can be accommodated by usability metrics. In considering adaptive systems as solutions to interface problems, the designer must consider an appropriate level of adaptivity. Totterdell, Browne and Norman (1987; 1991) examine adaptivity in natural and computer systems and identify four levels of adaptive system. (Simple) adaptive systems are stimulus-response systems or simple rule-based systems. They are characterised by their ability to produce a change in output in response to a change in input. Their adaptive mechanism is ‘hard wired’. These SelfRegulating systems monitor the effects of the adaptation on the subsequent interaction and evaluate this through trial and error. A mechanism is required which maintains a history or at least provides immediate feedback. This evaluation mechanism then selects from a range of possible outputs for any given input. Self-mediating systems monitor the effect on a model of the interaction. Possible adaptations can be tried out in theory before being put into practice. Self-modifying systems are capable of changing their representations. This allows self-modifying systems to reason about the interaction.
page 30
In terms of the architecture which we have developed in this paper, we can see the levels of adaptivity in respect of the models possessed by the adaptive system. 'Simple' adaptive systems possess a dialogue record and some adaptation mechanism. They do not have to have explicit user and domain models, though refining the mechanisms is made more difficult if they do not. Predictive interfaces (Greenberg et al., 1991) and many natural language systems fall into this category. The move to a learning system - a self-regulating adaptive system - is significant. Such systems now require inference mechanisms. They have to be able to abstract from the dialogue record and capture a logical or task level interpretation of the interaction. Similarly the system must now include a representation of its own ‘purpose’ in its domain model. Self-mediating systems require a more sophisticated model of the interaction and of the other system and require evaluation mechanisms in order to determine how effective a possible adaptation may be. Self-modifying systems are metaadaptive systems in that they can change the user, interaction and domain models. An important difference between Totterdell et al.’s approach and our own is that they see adaptation as being adaptation to an environment whereas we see adaptation as being to another system. Whereas Totterdell et al. argue that: if X occurs in the interaction then the system will do Y our approach is to say: if X occurs then Y is a characteristic of the other system if Y is a characteristic of the other system then the system should do Z In other words Totterdell et al. do not explicitly model the other system. They only react to the behaviour of the other system. We believe that this behaviourist view of interacting systems is restrictive, principally because it is difficult to transfer effects from one situation to another. By building realistic, humanistic representations of people, we are able not just to make better long-term use of the inferred characteristics, but also discover any human cognitive processing requirements demanded by particular system features. 4.3
Software Support
The adaptive system developer needs to consider the opportunities for adaptivity provided by the framework of the three models and their relationships identified in Section 3. Developing the user model forces the designer to focus on the psychological, profile and domain knowledge which users will require. The domain has to be described at the task, logical and physical levels. Specifying the interaction model requires the designer to consider what data is available from the interaction (the dialogue record) and the inferences, adaptations and evaluations which can be made from the dialogue record within the context of the domain and user models. page 31
As with many aspects of HCI design, the designer cannot be expected to code all the contents of the interaction knowledge base at specify the user and domain models using a low-level language. Indeed our experience with the first project (section x) demonstrated this quite convincingly. Hence, software support became an important and integral part of the adaptive system methodology. The Adaptive Systems Development Environment (ASDE, Benyon, Murray and Jennings, 1990; Benyon and Murray, 1993) is a designer’s tool-kit which tailors the general-purpose nature of an AI Toolkit to the specific needs of an adaptive system developer and which exploits the UIMS facilities of the underlying system at run-time. At present, we do not seek to develop the ‘self-modifying’ systems described in Section 4.3. Accordingly, the software does not facilitate the development of such systems. Changing the representations is still the job of the designer. The ASDE is similar to the concept of an expert system shell. During the development phase of building an adaptive system, the developer employs the ASDE to specify the characteristics of the target adaptive system and its users, their interaction and the mechanisms which will guide the inferencing, adaptations and evaluations which are to take place. Once the structure of a particular adaptive system has been established, the developer uses the ASDE to specify the values of relevant attributes of individual or classes of user and the inferences, adaptations and evaluations which are to be made from these values. These are instantiated in the target system with which the user interacts. The ASDE reflects the architecture of adaptive systems described earlier. The domain model describes the structure and functioning of the domain of the target (adaptive) system at the three levels of abstraction and provides the basis of an ‘overlay’ model which is used as a the representation of the user’s knowledge of the domain. The student model automatically inherits the domain model. Each user (individual or class) is represented in the adaptive system as a system object. When individuals use the target system, they are allocated to one or more stereotypes on the basis of the knowledge which the system has acquired about the user. The individual then becomes a member of that class and inherits the characteristics of the class. Conflicting classifications can be handled through the conflict resolution mechanisms provided by the AI Toolkit. As the system learns more about the user, so the user model becomes increasingly individual. The adaptation and inference rules are specified through the AI tool-kit's own mechanisms. An important aspect of the tool is that the models are explicit and can be displayed and edited if necessary. This facility is vital for the user model, both in respect of the privacy of individuals and their rights under data protection legislation, and in order to maintain the accuracy of the model. Making the interaction and domain models explicit and separable from the other components of the adaptive system facilitates changing the mechanisms as page 32
details of the interaction are understood. The designer describes these mechanisms through the ASDE. There are several other software systems concerned with developing aspects of adaptive systems. GUMS (Finin, 1989) is strictly a user modelling system and appears strong on the implementation of user stereotypes. These stereotypes include inference rules (which are part of the interaction model in the ASDE). However, GUMS does not place so much emphasis on the domain modelling or adaptation and evaluation rules, preferring to see these as parts of the user model. Kay’s system (Kay, 1991) is more wide ranging in that it seeks to provide tools for allowing users access to their user models (for viewing, editing etc.). It also provides an architecture to handle various levels of confidence which the system has in its inferred data. As with the system described here, Kay identifies a conceptual and physical level of knowledge which the user may have of the domain. Kobsa’s system (Kobsa, 19xx) employs a graphical function/concept tool similar in structure to the domain model described here. The UM-tool described by Branjik (Branjik, et al. 1990) takes a similar approach to modelling users and attaches methods of obtaining information to the frame slots. 5.
EXPERIMENTAL WORK
During the first project in 1986 - 1988, we were primarily concerned with the mechanisms and feasibility of adaptation. A small ITS was developed and implemented in GC Lisp. This system directed students through the learning material according to their inferred learning style and level of expertise and their performance during the current interaction session. The dialogue was all predetermined and so navigation could be controlled by directing the user to the next node in the system (Benyon and Murray, 1988). Some pilot experiments with this system (Benyon, Innocent and Murray, 1987; Benyon, Milan and Murray, 1987) confirmed that whilst the mechanisms were effective in that they successfully tracked a user through the dialogue, were able to make inferences from this behaviour and hence were able to adapt the dialogue based on these inferences, the inferences which were made did not correlate with either the users own assessment of themselves nor other more objective measures. The mechanisms were laborious to code in Lisp and this represented a serious developmental restriction. As a result of these experiences, the second project was designed which would tackle three main areas. 1.
There was the need to find reliable metrics of cognitive factors which genuinely affected the interaction.
2.
We wanted to implement the system in a much richer environment in order to exploit the facilities such as automatic inheritance mechanisms, rule specification, frame creation and user interface management offered by an AI toolkit.
page 33
3.
We needed to attend to the development of a shell type system which would facilitate the creation and amendment of adaptive systems by providing a set of tools to enable rapid and incremental changes to be made. The provision of such a toolkit meant that the architecture of an adaptive system had to be carefully understood and specified.
Items 2 and 3 above have been well described in this paper. The experimental work was aimed at finding more out about item 1 and was undertaken with the following aims. •
To identify one or more reliable and measurable individual differences which affect the interaction
•
To identify aspects of the interaction which could accommodate these differences and improve the performance of the disadvantaged group
•
To identify a way of inferring which group an individual user was in
•
To develop a mechanism which automatically adapted the system appropriately. 5.1
Pilot experiment
In the first experiment, reported fully in (Jennings, et al., 1991) an on-line shopping catalogue database system was developed. The database contained details of all the items which were available from the catalogue. The user could query the catalogue to find out, for example, if there was a vacuum cleaner costing less than £65. All items in the catalogue had three attributes such as colour, size and cost. Five functionally equivalent interfaces - iconic, button, command, menu and question - were developed for the database. These were chosen as being typical of interface styles which are widely available. Twenty-four users performed similar tasks using each of the interfaces. The time taken to complete a number of standard tasks was recorded and taken as the measure of performance. Each user was tested for five individual differences - spatial ability, verbal ability, field dependency, logical/intuitive thought and short term memory ability - using standard psychometric tests. Those with the top 12 scores on each test formed the high group and those with the 12 lowest scores into a low group. The main results of this experiment are shown in Figure 9.
page 34
A. Spatial Ability 400 task completion time (secs)
B. Verbal Ability 400
p
