Multi-component User Models of Team Members Zoë Lock and Daniel Kudenko QinetiQ, Malvern Technology Centre, St Andrews Road, Malvern, WR14 3PS, UK
[email protected] Department of Computer Science, University of York, Heslington, York, YO10 5DD, UK
[email protected]
Abstract. To date, most research on user models has been viewing the models as a single homogeneous entity. In contrast, a small number of recent modular approaches to user modelling permit the separate (and possibly heterogeneous) representation and acquisition of different aspects of the model, corresponding to different, competing user interests. This modularisation increases the flexibility, portability, and extensibility of the user models. This proves especially useful for modelling interests of users who are members of a team. We propose a modular approach based on different user perspectives such as team, role and current task. In our approach, an explicit, team model is contained within a single user model and components can be reused and shared by a number of users. We describe this approach in the application context of Personalised Briefing Agents, which are software agents that can automatically and autonomously filter and present information to a decision-maker.
1. Introduction Most user modelling research to date has been focused on models that are represented as a single homogeneous entity (e.g. [2, 3]). This approach has difficulties in applications where some components of the user model need to be changed over time, while others stay the same. Consider, for example, a team of decision-makers in which each team member fulfils a specific role (e.g., logistics officer). While certain information requirements are defined by the nature of the team, others are defined by the role of the team member or the task at hand (e.g., a peacekeeping operation or a planned take-over of another company). If all these requirements are represented in a single user model, each time the task changes, the whole user model needs to be updated. However, if the components are represented separately, only the one that specifies task-dependent information requirements needs to be changed. Similarly, if a decision-maker maintains the same role but changes team, then the role and task components can stay the same, and only the team component needs to change. A further advantage of a component-based representation of user models is that the components can be represented using different formalisms. In this paper we propose a multi-component user modelling approach, in which each user model contains a single, explicit team model, and discuss the issues involved. The research will be presented in the context of Personalised Briefing
Zoë Lock and Daniel Kudenko Agents (PBAs), which are software agents that can automatically and autonomously filter and present information to a decision-maker according to his or her own requirements and preferences. PBAs can be employed, for example, in military headquarters to automate the briefing process, reduce information overload and enhance situational awareness. In contrast to other multi-component user modelling approaches (see section 4 for more discussion), our proposed approach has the following properties: − Most components are user-independent and thus highly reusable in different user and team contexts. − All components contribute to the relevance measure of information items according to user-dependent weights. − Components are acquired initially from a large number of users, and then used and adapted within an individual user model. The remainder of this paper is organised as follows. First, we introduce the military PBA context in which our multi-component user modelling approach is being applied and evaluated. Section 3 describes the different perspectives of military commanders that dictate information requirements, each of which translates into a separate user model component. This is followed by a discussion on selected existing multicomponent user modelling approaches. In Section 5, we introduce our multicomponent user modelling approach and discuss some of the main issues that will need to be resolved when developing our models. Section 6 describes how the multicomponent user models will be evaluated in the PBA context. Finally, Section 7 concludes the paper with a summary and outlook.
2. Application Domain: Military Briefings Military briefings are our focus application area for PBAs. Military commanders at different command levels have various information requirements in order to make informed and timely decisions. Commanders must collaborate to achieve mission success. Large amounts of information are available, including some information items on the WWW, but not all of it is relevant to all commanders. Briefings are commonly used to transmit important information within military command. From our observations and understanding of the military context, a number of problems with the current manual briefing process can be identified: − Use of staff time: Briefing preparation can be labour-intensive. Time spent by a briefer on filtering information and constructing briefings could be made available to his other tasks. − Information overload: Frequently, commanders receive too much irrelevant information. − Lack of personalisation: Commanders differ in terms of their information requirements and briefing preferences and yet all team members currently receive the same briefings. Although this has the advantage that everyone receives a wide
Multi-component User Models of Team Members
3
range of information, there are risks both of presenting irrelevant information and of presenting relevant information in an inappropriate manner to team members. − Lack of timeliness of briefings: Frequently, other tasks must be interrupted so that staff can gather together for staff briefings. For this reason, such briefings cannot be held too frequently, and personnel often have to wait for staff-wide briefings to be scheduled to receive information that may support their decision-making. We expect PBAs to significantly improve all of the above problems. By automatically and correctly filtering out irrelevant information and providing briefings on demand, the problem of information overload could be alleviated. In addition, by storing and providing information electronically, synchronisation between commanders could be achieved with no extra effort. As long as the information is stored consistently (i.e. with no duplication or logical contradictions) then all commanders could access the same up-to-date information as required for their individual decision-making cycles. However, the focus of this paper is on the user models that will be required to represent commanders’ information requirements and predict relevance values for new information items.
3. Changing Commander Perspectives There seem to be several aspects or perspectives of the commander that can dictate his information requirements and preferences. For example, requirements can depend on: − − − −
the team to which the commander belongs; the role that the commander adopts/has within his team; the current operation; any personal preferences the commander may have.
The first three perspectives dictate information content requirements, in other words, the relevance of information items given a commander’s team, role and current operation. The fourth represents the desired style and format of information and does not dictate information relevance. Over time an individual commander’s perspectives may change, for example, as he changes role within a team, moves between teams or as the operation changes. Numerous commanders can assume the same command role at different times either in the same team over shift changes or across different teams. Within the same role, commanders have some requirements in common but they may differ in terms of other requirements that depend on other perspectives such as team membership or the current operation. It would therefore be useful to associate each information requirement with the associated user perspective rather than with the user as a whole. Each user perspective corresponds to a user model component.
Zoë Lock and Daniel Kudenko
4. Related Work Some multi-component user models have been developed in which a user’s different topics of interest can be represented by separate components. In some applications, only one component is used at any one time and so the relevant component must be selected to suit the current query or interest. In other applications, all components may contribute to the relevance value of a single information item, so their contributions are combined in some way. Billsus and Pazzani [1] describe a system that automatically compiles personalised spoken daily news programmes for users using a dual-component user model. The user models consist of two components, a short-term model and a long-term model of a user’s interests. The system constructs a short-term model using only the most recent items. To classify a new news story as interesting or uninteresting to a user, its corresponding Boolean feature vector is presented to the short-term model first. The distance between the new vector and every other vector in the short-term model is measured. If the new vector is deemed close enough to other items then it is interesting to the user and added to the model. Any stories that are not considered similar enough to any of the items of short-term interest are then classified by the long-term model. Billsus and Pazzani’s hybrid model outperformed either of the two individual classification approaches. McGowan et al. [2] have developed ‘Web Personae’ to model a user’s different set of interests, for example, golf or theatre. The individual components are learnt using hierarchical clustering techniques over web page content. As only one Web Persona is relevant at any one time, the cluster with its centroid most similar to a user query will be selected and used to search for relevant pages. In the PBA setting, it is clear that the different perspectives of a commander will all contribute to the relevance value of a new information item. In this case, the relevance values from a number of separate components will be combined to derive a single overall relevance value. Baudisch and Brueckner [3] describe a television recommendation system that models various user interests, which can be expressed using several query types such as genre and text search. The user can execute all queries at once by selecting a query called all favorites and a linear combination of the results from the queries (executed by separate subsystems) is used to derive a single ranking of TV shows. The parameters of the linear combination can be adapted to alter the output. These parameters are initialised by assigning higher weights to the most specific queries. Implicit feedback is then used to adapt the weights. In addition, users can manually adjust the weights. Buczak et al. [4] describes another television show recommendation system that employs multiple user interest profiles to model all the user’s television show interests. The profiles are learnt using a range of different machine learning techniques. In addition, some are learnt using implicit feedback while others are developed from explicit feedback. Some of the profiles use feedback from individual users whereas others use feedback from user groups, in this case, households. The various recommender components achieve different levels of accuracy and a radial basis function (RBF) is used to combine the outputs of all the components into a single interest value for a user. However, in the system described, the learnt (RBF)
Multi-component User Models of Team Members
5
is the same for all users so the components contribute to every user’s interests in the same way. Buczak et al. discuss a user for whom the system did not accuracy predict his interests. This was because his interests were quite different from those of the rest of his household. So when the components learnt on the basis of feedback from the entire household have higher contributions to the output, then it performs poorly for such users. The system described in [4] performs well for users whose “behaviour conforms to the mainstream.” In the military domain, as well as many others, there may be many users who do not always share common interests with team colleagues. Indeed, some commanders may have more general interests than others and may have many interests in common with fellow team members, whereas others are specialists who share few team interests. We propose a multi-component user modelling approach in which each user model contains a single and explicit team profile in addition to other distinct components. Each component will represent a user’s interests from one perspective and components will be reusable/plug-and-play to prevent the cold start problem1. The approach will be illustrated in the next section. Unlike the systems described in [1] and [2], all components can contribute towards the rating of a new item, and the individual ratings will be resolved. Like [3], we will employ a linear combination of individual ratings but the weightings of the components will be user-dependent. This means that there will be no outliers like the user described in [4].
5. Multi-component User Models for PBAs In this section, we describe our multi-component user modelling approach in more detail. We first discuss the different components and their representation, and then describe their acquisition, which is carried out in three different training phases. 5.1
Component Profiles
In order to construct and investigate multi-component user models it is necessary to define the individual components. These will be based on commander perspectives introduced in section 3. The team perspective of a commander will relate to the information requirements common to all members of the same command team. The role perspective of a commander will relate to the information requirements common to all commanders in the same command role irrespective of team membership. The operation perspective of a commander will relate to the transitory information requirements of a commander that are particular to the current operation. The information requirements of different commanders, in the role and operation perspective components, may overlap but we assume, for simplicity, that they are independent. The personal preferences component will represent the briefing style and format preferences of the commander. 1
By the cold start problem, we are referring to the problem of predicting the relevance values for a new user, of which little is known about his interests [5].
Zoë Lock and Daniel Kudenko The remainder of this paper will focus on the first three perspectives, which dictate the information content requirements of commanders – team, role and operation. Although we assume that the component profiles are independent, it is likely that the team perspective component will contain information requirements that are more general in nature than those in the other components. The role component is likely to consist of requirements that are, in turn, more general than those in the operation perspective component. For example, a commander interested in the enemy’s location because of his role, may be interested in the location of a particular, named enemy unit because of the particular enemy involved in the current operation. Any generality ordering may not apply to all requirements in the whole user model. By developing separate and explicit perspective components, the relationship between the perspective requirements can be studied. 5.2
Representation of Multi-component User Models
All components will contribute to a new item’s relevance value. So, as in [3], a simple linear combination of their outputs will be used to obtain an overall relevance value for a new item. The weighting for the combination is therefore required. In [4], it was assumed that this weighting would be user-independent, but our approach dictates that this weighting is user-dependent. This weighting scheme will be discussed briefly in section 5.4. Various user model representation schemes have been used in the past [e.g. 1, 6 and 7]. A multi-component user model could be implemented using multiple representation schemes and multiple classification methods. In particular, it is likely that the personal preferences component will have a different structure to the others as it will be less concerned with the informational content of items and more concerned with aspects of briefings such as style or format. The personal preferences component may be more suited to attribute-value representations whereas Boolean feature vectors or term frequency – inverse document frequency (TF-IDF) may be more appropriate for the other components. Here a single representation and classification scheme has been used, but this need not be the case for other applications. The multi-component user models will be represented using the formalism of multi-attribute utility theory (MAUT) [6]. The MAUT formalism has been used previously for applications in which the value dimensions are product attributes or features, such as mileage or airbags [6]. In our system, the perspective components described earlier constitute abstract value dimensions along which a commander can judge a new item in terms of relevance. The attributes in the bottom layer are weighted Boolean feature vectors. The vector space model is used to represent information items where each feature is a word (selected during a pre-processing stage2). When measuring the relevance of an item, two sets of variables in the model can be adjusted: the weights of individual features (the intra-component weights) and the weights of the perspective components (the inter-component weights). Figure 1 shows the basic structure of a value tree that 2
In preprocessing, high-frequency words are removed because they are not usually indicative of the topic of the item or its relevance.
Multi-component User Models of Team Members
7
can be used to represent the relevance of information items to commanders. A single user model consists of the appropriate perspective components (the weighted feature vectors) and corresponding inter-component weights. To be immediately relevant to the user, information must be relevant to the current operation. Information requirements that relate to the current operation are more transitory than those relating to the other components. We therefore anticipate that the operation component will generally have a higher weight than the other components. The operation component may therefore bear some similarity to Billsus and Pazzani’s short-term user model. Relevance to commander
α
β
Relevance to team
wt1 ft1
wt2 ft2
…
wti fti
γ
Relevance to role
wr1 fr1
wr2 fr2
Relevance to operation
wrj …
frj
wm1 fm1
wm2 wmk fm2
…
fmk
Fig. 1. MAUT representation for evaluating relevance of information items to commanders. Each relevance box in the tree represents a linear weighted sum of the children
5.3
First Training Phase: Acquisition of Initial Components
Most commanders’ information requirements are not easy to enumerate prior to operations as they are in fact “highly variable and human-intensive elements” [8]. For a whole command group, consisting of personnel with different roles and goals, the range of information requirements may be wide, covering much of the battlespace. There may be some commanders who can explicitly state their critical information requirements, and if these are available then they could be used to construct initial user models. Here, we do not assume that such pre-defined requirements exist. To construct the training sets required for our approach, commanders will be presented with items and will be asked to provide binary relevance feedback (relevant or irrelevant). Some of the items together with the feedback will form the training set. Other item-feedback pairs will be used for a testing set for evaluation purposes (see section 6). However, the training set must be analysed further to distinguish between the impacts of the different user perspectives.
Zoë Lock and Daniel Kudenko To learn each initial component, the team membership, role and current operation will be identified for each commander and these labels will accompany the itemfeedback pairs in the training set. The training process must be performed for a whole team of users over multiple operations so that the information requirements of the team, role and operation perspectives can be learnt. Training data from members of the same team is used to induce the corresponding team perspective component. Thus, the work involved in collecting feedback can be distributed amongst a group of users, and the inconvenience to each user can be reduced. Training data coming for a single commander over many operations is used to induce the respective role component. All other requirements are represented in the operation perspective component, which is induced from feedback given by multiple users executing the same operation.
5.4
Second Training Phase: Acquisition of Initial User Models
Once the initial components have been learnt and each user model contains the appropriate perspective components for its commander, the inter-component weights will be learnt during a second training phase. Part of labeled training set will be used to learn the inter-component weights so that the weights will be user-dependent. The intra-component weights will remain static during this phase.
5.5
Third Training Phase: Gradual Adaptation of User Models
Once the initial user models have been acquired, these can be used to filter future information items but will undoubtedly require adaptation as the users’ requirements change. Baudisch [9] discusses three types of interest change: − abrupt changes take place suddenly in response to an event such as a change in team, role or operation; − gradual changes are the result of a slow adaptation over time due, for example, to the natural evolution of the information requirements of a command role; − repetitive changes can be abrupt or gradual, but the state is temporary. In this context, an abrupt change corresponds to a change in perspective and when such a change occurs, the corresponding component can be isolated and changed. If a commander changes team, then the team component, common to all members of the new team, can be used as the new team component for the commander. If a commander changes role within the same team, then the role component can be replaced with that of another commander with the same role in another team. When a new operation begins, we propose the following to create a new operation component: − the initial operation component is taken from a similar past operation and subsequently adapted;
Multi-component User Models of Team Members
9
− the information provision to each commander is initially guided mainly by his team and role components while a new operation component is learnt or specified by a commander. The first option is preferable and but the second could be used as a fall back position (and will be used in early development). In order to learn multiple operation components at the start, the training set will be labeled with operation phase information (e.g. pre-deployment, recovery, etc.). The gradual evolution of requirements will be modelled by adjusting the intra- and inter-component weights shown in figure 1 according to feedback from the commander using a learning algorithm such as back propagation. This evolution constitutes a third training phase that could continue throughout the system’s deployment. Neither [3] nor [4] suggest such a comprehensive user model adaptation phase. One major issue with this training phase is that feedback from a single commander may suggest that the intra-component weights of the team component need to be adjusted. If such a change is made in the commander’s user model, his team component would differ from his fellow team members. We propose that any adjustments to the team component are made globally, throughout the team, so that the explicit team model remains consistent and user-independent. This is not an issue for the other components, and so intra-component weights changes may be made locally for the individual user. This will be useful for shift changeovers when two commanders fulfilling the same role over different shifts, can share the same initial role component. When one of the commanders works on his shift, his role component may be updated in response to feedback. When the next commander relieves the shift, the updated component can automatically replace his role component. The newcomer need not know about the changes made over the previous shift. Another issue associated with the third training phase is that it may involve a lot of processing. We will assume that unless we receive relevance feedback from a commander, the automated classification is correct. The weight adjustment will only occur when feedback is received. Feedback may be frequent to begin with but we expect it to decrease over time.
6. Evaluation of User Models We hypothesise that the multi-component user models proposed in this paper will be more robust than existing single-component user models and multi-component user models, such as those described in [3] and [4]. This is because the components can be adapted and reused to facilitate abrupt, gradual and repetitive changes in user interest. In addition, we anticipate that, as inter-component weights will be user-dependent, accuracy will be acceptable for all users. To evaluate our claims, we will compare the performance of our user models with that of a single-component user model and a multi-component user model in the style of [3]. This comparison will be made over many interest changes. We also plan to demonstrate how component reuse can solve the cold start problem.
Zoë Lock and Daniel Kudenko
7. Conclusions Multi-component user models based on user perspectives have been proposed for PBAs for military commanders. Two hypotheses have been stated and will be evaluated thoroughly once the PBAs have been fully developed. Several issues arise when considering the representation, combination and adaptation of user model components. It is likely that many more will arise during future development and evaluation. It is anticipated that this avenue of research will contribute to user modelling research by incorporating explicit team models within robust, modular user models. We also aim to highlight particular issues that arise when employing user modelling in the military environment. Although, we have focused on the military collaborative environment, the multicomponent user models described in this paper could be exploited in other collaborative environments in which information requirements or user attributes are linked to different perspectives including team membership and assumed role. References 1. 2. 3.
4. 5. 6.
7. 8. 9.
Billsus D. and Pazzani M. (1999). A Hybrid User Model for News Story Classification. UM99. McGowan J., Kushmerick N., and Smyth B. Who Do You Want to Be Today? Web Personae for Personalised Information Access. LNCS 2347. Springer Link. Baudisch, P. and Brueckner, L. TV Scout: Lowering the entry barrier to personalized TV program recommendation. In Proceedings of the 2nd International Conference on Adaptive Hypermedia and Adaptive Web Based Systems (AH2002), May 29-31, Malaga, Spain. Buczak A., Zimmerman J. and Kurapati K. Personalization: Improving Ease-of-Use, Trust and Accuracy of a TV Show Recommender. TV'02:Workshop on Personalization in TV, AH2002. Salton G. and McGill M. Introduction to Modern Information Retrieval. McGraw Hill. 1983. Jameson A., Schäfer R., Simons J. and Weis T. (1995). Adaptive Provision of EvaluationOriented Information: Tasks and Techniques. In C. S. Mellish (Ed.), Proceedings of the Fourteenth International Joint Conference on Artificial Intelligence (pp. 1886-1893). San Mateo, CA: Morgan Kaufmann. Langley P. (1999). User modeling in adaptive interfaces. Proceedings of the Seventh International Conference on User Modeling (pp. 357-370). Banff, Alberta: Springer. Kahan J., Worley D. and Stasz C. (1989). Understanding Commander’s Information Needs. Baudisch, P. Dynamic Information Filtering. Ph.D. Thesis. GMD Research Series 2001, No. 16. GMD Forschungszentrum Informationstechnik GmbH, Sankt Augustin. ISSN 1435-2699, ISBN 3-88457-399-3.
Acknowledgements This work was carried out as part of the UK Ministry of Defence Corporate Research Programme. © Copyright QinetiQ ltd 2003
