Using Workflow, User Modeling and Tutoring Strategies for Just-in ...

Using Workflow, User Modeling and Tutoring Strategies for Just-in-time Document Delivery J. Davis, J. Kay, B. Kummerfeld, J. Poon, A. Quigley, G. Saunders, K. Yacef School of Information Technologies University of Sydney, Australia {jdavis,judy,bob,josiah,aquigley,gsaunder,kalina}@it.usyd.edu.au

Abstract. New employees in an organisation typically undergo a period of relatively intense training when they commence their employment. Often the quantity of information imparted is too large for the newcomers to assimilate during the short training period. Moreover, much of the information may not be relevant until months or even years after the initial training period, by which time it has long been forgotten. We propose to address these problems by creating a smart personal assistant which delivers training documents to its user in a just-in-time manner. The proposed system uses workflow technology to drive the delivery of documents in a timely manner based on organisational processes through which the user is working. User-modelling is incorporated into the system to ensure that redundant or previously known information is not delivered, thereby reducing the problem of information overload. Finally, tutoring strategies are used to prioritise the available documents based on their present relevance and importance.

Introduction There are many learning challenges within organisations. Assimilation of organisational knowledge is a problem encountered in almost any organisation [1]. A particularly important group of problems relates to helping people learn things that are already well documented within the organisation. The challenge in supporting such learning is that the people within the organisation often have real difficulty in gaining access to that information at the right time. In cases where the person is aware of their lack of knowledge, the primary challenge is to help them find the documents which can answer their needs. Even more challenging is the common case where the person is unaware that they have a learning need. This problem is compounded when people are feeling overloaded, either by the demands of their work or information overload. One important aspect of this workplace learning relates to the learning load. The timing of teaching is critical to the success of learning. In an organisational context, it is important to take account of the problems of information overload,

especially in the case of people who are new to the organisation or who have recently changed their roles and responsibilities. Such newcomers to roles or organisations have a great deal to learn and are only able to absorb a limited amount of information at one time. More generally, teaching should be paced so that a person is given information in manageable chunks, so as not to overload their short-term memory [2]. It is particularly appealing to improve the delivery of documents at just the time that the learner is ready for them, when they need to learn. So, for example, suppose an employee has been in their job for six months at an organisation which requires them to take annual leave within the next six months. This employee would do well to begin planning that leave. So this would be a good time to ensure that they are aware of this leave policy. A simple way to achieve this would be to deliver a short prompt, perhaps via email, pointing them to the particular paragraph that they need know about in the leave policy document. This is just one simple example of a learning need that the employee may not have been aware of. Another crucial aspect of workplace learning is the appropriateness of the information content given to the user: it should be adapted to the current context of their work, to their pre-existing knowledge. There is a need to understand the user’s needs in order to adapt the delivery of information. Some researchers have adopted a task-driven approach to model the context of the users’ needs and build heuristics to tailor the information to them and present it in a just-in-time manner [3, 4]. In our project, we use a workflow approach to determine the context of the user’s needs. In organisations where there are automated processes for managing workflow, these can provide a framework for supporting learning. For example, our university has a process for managing examination scripts. This process could well be supported by a conventional workflow system. A new academic who is responsible for running a teaching unit in our organisation would need to learn about several aspects of this and related processes. Since newcomers need to learn so many other things when they first join our organisation, it is not surprising that they generally have problems learning what they need to know to get through this process. It is typical of many learning processes in that it runs over a period of several months and has several stages. It requires completion of several forms and there are several relevant policy documents. There are many support learning documents, including the policy documents of the institution, and our own school and there are examples of ways to complete sub-tasks. It is very appealing to tap into a workflow for a process like this so that it supports, not only the normal management of the documents and approval processes, but also that it should support learning about those processes. This should be in good time as well as just-in-time. Both the examples mentioned, leave and examination script management, are also typical of many aspects of workplace planning in that there is considerable tacit knowledge involved. It would be valuable to facilitate learning of these aspects by helping newcomers and those coming to new roles by helping them make contact with people who have recent relevant experience and willingness to help. This paper describes the current version of JITT, a Just-In-Time Training system which tackles these problems by building upon workflow models and technology [5]. Section 2 describes the overall architecture of JITT and then we describe its elements, the interface in Section 3, the user model in Section 4, the tutoring strategy in Section 5. Section 6 summarises the current state of the work and concludes.

1. Architecture The overall architecture of JITT is illustrated in Figure 1. At the top is the Human Resources staff team. They need to initialise the system when a new employee starts. This process can be triggered by the typical administrative tasks associated with this starting stage. HR initialises user roles and profiles

My Just In Time Training system

User Model

Activity Engine

Traditional workflow + teaching strategy layer

Tutoring strategy prioritiser Interface(s) {web, email}

Figure 1 Architecture of JITT

For example, the new employee will have a role within the organisation and that will be reflected in an obvious way in their user model. This will show their position and level. Normal duty statements within the organisation would naturally feed into the definition of default tasks for each user. For example, a new member of academic staff would typically have responsibility for teaching and research. Each of these roles can be used to create a default user model for these aspects of a new member of staff. This would include the elements that are a normal part of these roles. For example, the teaching role would generally include the need to set examination papers and submit these and associated paper work. The initial model would include a teaching context and within that, there would be a context for handling examination scripts. For a new staff member, these would initially be modelled with a stereotype [6] to reflect that the user does not know the procedures. At the very bottom, we show the user. S/he can interact with JITT and update the user model. For example, although the HR staff instantiates the default model, the user can alter that. Users interact with JITT via the interface layer shown at the bottom of the main block in the figure. When the Human Resources staff member establishes a new user in JITT, s/he also instantiates the activity engine. This activates a set of workflows, each associated with a context in the JITT user model structure. So, for example, JITT creates a user model context for the user's knowledge of leave processes within the organisation. This starts one workflow for that user for each of the forms of relevant leave. Similarly, the user model context for handling examination scripts is associated with a workflow for that process. As indicated in the figure, there are two distinct types of workflow managed by the activity engine. One important class of these is based on the typical process workflows that are currently used in organisations. One of these could be used for the

management of examination scripts. A quite different class of workflow is purely part of JITT. This structures the ordering of teaching within one context. The final part of the architecture is the tutoring strategy prioritiser. This takes the information from the Activity Engine and the user model in order to formulate a prioritised set of teaching goals. It makes the top ranking ones available to the user at the interface. We now discuss each of these elements in detail.

2. Interface We now describe the interface element of the architecture. This also provides an overview of JITT from the user's perspective. As indicated in Figure 1, we have designed JITT so that it could, potentially, use a range of mechanisms to communicate with the user. At this stage, our prototype implementation uses just two interface elements: email and a web site. We will illustrate the operation of the interface in terms of the pacing shown in Figure 2. This shows the user who starts employment on Jan 1, interacts with JITT on her first day on the job, then on a set date, April 30 of her first year on the job and then again six months after starting. Human Resources

Activities A B …. Profile

Roles Stereotype user model + Activities : A, B

Activities A B …. Profile

Day 1

Activities A C D Profile

+6 months On April, 30 Doc Y

Figure 2 - Pacing of screenshots

As indicated in Figure 1, an individual users' JITT agent is initiated by the actions of the Human Resources staff. This means that the new employee's JITT agent can establish an initial user profile and student model and it can use these to formulate an initial set of teaching goals. Figure 3 shows an example of a JITT screen with a set of initial recommended learning goals. These are organised into categories. From the user's point of view, these are intended to structure the presentation in terms of major activities in the workplace. In terms of our architecture, each main category maps to one workflow within JITT. JITT is intended to work in either push or pull mode. We first describe JITT's push mode of operation. As soon as JITT is initialised for a user, it would send mail directing her to the JITT web site. In general, the email interface in our current prototype simply sends short notifications to the user, suggesting that they consult the JITT web site and providing its url. In the case of the example shown in Figure 2 , such notification mail would be first sent at the start of employment, then on April 30 and again 6 months from the start of employment. These latter two notifications illustrate the two main JITT trigger mechanisms. The April 30 case is tied to a

specific date. By contrast, the learning process model that triggered the other notification operates in terms of times relative to events within the user model: in this case, the event is the user's start of employment but in general, it might be any other event that is held for this user.

Figure 3- JITT, day one for user Jane Simmons

Figure 4 indicates the types of changes in the interface in the later visits to the JITT site. The overall learning categories, as active workflows, will tend be stable over quite long periods of time. However, at each stage, different learning objectives will be suggested for some of them.

Figure 4 - JITT, at a later date for user Jane Simmons

JITT will also operate in a pull mode, with the user initiating a request for advice on learning. This operates in two ways. Firstly, as the previous discussion

indicates, the user can access the JITT web site whenever she chooses. She can then scan the list of recommended learning topics. As shown in the interface, we also allow the user to enter a search request. This will search both the documents that are linked to the currently active workflows for this user and other collections of organisational documents. The results will be prioritised by the tutoring strategy component of JITT. Essentially, the user should see learning recommendations based on a combination of the request and the user's current learning context. They will be presented in essentially the standard JITT format. The difference is that there will be at least one additional category. This is needed to present documents that matched the query but were not part of a workflow that is currently active for that user. We have not yet built this part of the interface and need to explore some of the range of possible structures. First, there are documents that are indexed as part of JITT but belong to workflows that are not currently active for the user. These can be presented in much the same structure as the current interface but need to be distinguishable from those learning recommendations. Secondly, there may be organisational documents that can be searched at this stage but are not part of any JITT workflow. The nature of the learning documents deserves some discussion. It is typically true that much of the information that employees need to learn is actually documented, somewhere, within the organisation. Often, the particular element that is relevant to a person's current learning need is captured in a small piece of text, perhaps a single paragraph, in a quite large document. In that case, the document that JITT delivers should be the relevant snippet of the full document. For the current prototype, we have added tags to our test collection of documents so that we can index relevant parts for the user. In the longer term, we would like to perform customisation of the documents so that the user can be shown just the relevant snippet and an indication of where this is located in the actual document. We would also like to extend the work to deal with a broader range of documents, so that we can deliver arbitrary customised multimedia.

3. Activity engine The workflow technology lies in the activity engine. In the literature, when workflow technology is applied to teaching, it is used to handle the administration side of teaching or to let students pace their learning activities by relaxing the time constraints. The Flex-eL project [7] is such an example. In JITT, we use workflow differently. Rather than modeling the teaching workflows, we use the existing business workflows and deduce the pacing of the training from them. The activity engine consists of workflows, which can be of two types: extended business workflows and time projection workflows. We will describe them in the context of an example. Figure 5 shows the example of a workflow for managing examination scripts in our university. Many nodes have concepts attached to them. These are depicted with a dashed box. We have only shown here some of them, so as not to clutter the diagram. For example the node “Book room” has the concept “Room booking procedures” attached to it. “Large scale copying” is a relevant concept for the two “large scale copy” nodes. (One of these is at the lower right of the figure and is indicated with an ellipse. The other is straight above it, near the top.) Another important dimension, not shown for this process here, is the corresponding timescale. For example, the “Write exam” node, in the case of a formal examination, has a specific deadline by which it should be done (and it appears at the

lower left of the figure). This deadline is different for the other “write exam” node (above and to the right) in the case of an in-house examination. Book exam supervisors Book room

in house

Course associate check

in house

Students sit exam

Draft marking scheme

Write exam

Large scale copy

Inform students of assessment conditions (eg open book)

Submit exam request form

Handle scripts

Students sit exam non confidential paper

•Room booking procedures

script ok

•Exam writing procedures •Marking issues •Past exams

Internal submission of exam script

Course associate check

Admin check