A Multiple Student and User Modelling System for Peer Interaction Susan Bull Language Centre, University of Brighton, Falmer, Brighton, East Sussex, BN1 9PH, UK.
[email protected]
Abstract: This paper presents a combined student and user modelling system (S/UM) to facilitate educational interaction between peers. Students may use S/UM to find a collaborative or cooperative partner, to request help on a specific problem, or to seek more general feedback on their work. An individual, inspectable student model is maintained for each student, based on feedback received from peers. An inspectable, modifiable user model is also constructed for each individual to indicate their availability for assistance should a request for help be made, and also the areas of the task on which they feel able to offer useful feedback. The user model also contains information about areas where students would themselves like help, and about whether they would like to collaborate or work cooperatively if an appropriate partner can be found. Representations for the user model are provided directly by the user. The main aim is to promote reflection in the users of the system.
1. Introduction Arguments have been made for the benefits of opening a system’s student model to the student - to encourage learner reflection; to facilitate discussion of the contents of the model; or to allow editing of the model (Bull & Pain, 1995; Cumming & Self, 1991; Kay, 1994; Paiva et al, 1995; Self, 1991). A number of computational systems have also been developed to take advantage of peer interaction - in the context of face-to-face interaction at one computer (e.g. Bull & Broady, 1997; Puntambekar, 1995); and distributed interaction via a computational learning environment (e.g. Chan et al, 1992; Hoppe, 1995). Further, peer modelling has been developed, where an inspectable student model is constructed in part from a peer assessment (Bull & Brna, 1997). Possibilities exist for uniting these approaches to take advantage of the benefits of each, with a system to encourage learner reflection through an inspectable student model created from peer feedback, which itself then leads to further peer interaction. S/UM, a student and user modelling system, comprises multiple inspectable student and user models - one of each type for each individual student - which are used by the system to help match potential collaborative partners and mediate between helpers and those seeking help on some aspect of a task. Student and user models are inspectable by those they model, and the user models are more widely available for viewing. User models are maintained by the user they model, and student models are updated based on feedback from peers. System modelling is restricted to an overview of representations given by non-expert humans, which is designed specifically to prompt interaction between human users to encourage reflection. System support is in the form of managing the selection of partners.
2. The Origins of S/UM
S/UM has much in common with a number of recent systems which use inspectable user or student models, but it is unique in its particular combination of the various benefits of these systems. The systems on which it is based are briefly introduced below: 1. See Yourself Write (Bull, 1997) is a vehicle through which expert tutors provide feedback to students via an inspectable student model, with the aim of promoting learner reflection through enabling them to interact with their feedback. 2. PeerSM (Bull & Brna, 1997) is a system for two peers to give feedback to each other, the aim again to enhance reflection, but in this case also to promote reflection in the giver of feedback. 3. 2SM (Bull & Broady, 1997) provides a pair of students at the same screen, with their two student models, in order to encourage rich peer discussion in a face-to-face situation. 4. PairSM (Smith & Bull, 1997) calculates appropriate interaction types for a pair of students, according to information in their respective student models. 5. PHelpS (Collins et al, 1997) is concerned with mediating peer help when more than two people are involved, through multiple inspectable user models. S/UM unites these approaches. It suggests peer interactions based on the user and student models of multiple learners. The recommendations which can be made are as follows: suggestion of a peer who is willing and able to offer feedback on some aspect of the task; suggestion of a peer who would benefit from feedback; suggestion of a collaborative partner; suggestion of a cooperative partner.1 The interactions which result from these recommendations may take place either on-line or off-line.
3. The Domain of S/UM The framework of S/UM is designed to be domain independent: peer interaction and awarenessraising through inspectable student models is applicable to a variety of subject types. The current implementation uses writing as the task, though the writing may take different forms as a number of aspects of writing are relevant in different contexts. S/UM is not itself a writing environment: students may use any word processor, or a writing environment such as Composer which is designed to provide support in writing a document (Pemberton et al, 1996), but students should make their document available to others in a form accessible to all, in order not to restrict the number of potential partners for peer interaction. (The current version of S/UM has a WWW site where writers can submit their document.) S/UM helps learners to select suitable working partners once their document (or a draft of their document) has been written. Interaction through S/UM concerns four main aspects of writing: content of document; structure of document; argumentation skills; style of writing. A further category (’other’) enables feedback on aspects of a writing task which are not not covered by the above. Feedback may also be given (and received) in a more general manner - i.e. not targeted to a particular problem. The aim is to increase learner awareness of issues to consider when writing. The more learners interact with their peers, the more suggestions they receive, and the more work they see and evaluate by others - the more likely they are to improve their understanding of what is involved in the writing process.
4. Construction of the S/UM Models
1
In this context ’collaboration’ concerns peers working together on the same area in order that they may together increase their understanding of the requirements of the category, resulting in each being able to improve their document. ’Cooperation’ describes a situation where peers help each other in different areas according to their relative strengths.
S/UM combines the modelling approach of See Yourself Write with feedback from peers. Although See Yourself Write differs strongly from most student models in its purpose of promoting learner reflection rather than existing as a source of information for system adaptation, it has in common with the majority of learner models its one-to-one modeller-modellee status. S/UM differs in that there can be a number of peer modellers contributing to each student model, and each modeller may contribute to any number of student models in S/UM. S/UM also contains separate user models (updated by the individual user) to represent the kind of feedback an individual is able to give; would like to receive; their availability for interaction; the kinds of interaction they would prefer (in addition to giving or receiving feedback - collaboration or cooperation). The contents of the user models are therefore more similar to preferences than to knowledge states, or skills acquired, or routine paths followed, etc. However, these preferences are not permanent, and for many users are likely to change frequently. An example of the sources of information for user and student models is given in Figure 1. Each user model (UM) is constructed and maintained by its owner (and only by its owner). Representations in student models are obtained from peer feedback (quantitative and often also qualitative). In this example S1’s student model (SM1) has had information recently entered in two categories: content and argument. These representations have been constructed from two sources: feedback on content from S2 and S3, and feedback on argument from S3. SM2 contains information on the style of S2’s document, which was provided some time ago by S1. Similarly, SM3 holds information on content previously provided by S2. This distinction of recency of representations is relevant because feedback provided a while ago may no longer be such an accurate representation if the recipient has since altered their document. S2 and S3 are currently cooperating (off-line) on the areas of structure and argument, but have not yet reached the stage where the summaries are submitted to the student models. Once this occurs, SM2 will contain information about argument as well as the already represented style, and SM3 will hold information on structure in addition to content. content
S1
S2
UM 1
UM 2 content
SM 1
argument structure argument
S3 UM 3
content
SM 2
SM 3
style Figure 1: Sources of information for the user and student models of S/UM
In this example S3 was seeking, and received feedback on content a while ago, and then recently provided feedback on content him/herself. This may be because, since receiving this feedback, s/he has improved and now feels more confident in this area, or it may be that the original request for feedback was more for reassurance that the topics included in the document were adequate or acceptable. In this case the content coverage may have been good anyway. 4.1 Updating the User Model Users udate their user model by clicking descriptions on or off (see Figure 2).
Figure 2: The user’s view of their user model
4.2 Contributing to a Student Model Peers provide information for student models as shown in Figure 3. The interaction is initiated when a peer selects ’give feedback’ from the main menu. They then receive two menus from which they select the recipient of feedback and the category of feedback. (The options offered are based on the various student models.) Peers are asked to select a quantitative evaluation of performance in the category selected. They may also provide further (qualitative) information from menu selection and/or by typing text.
S/UM: Who would you like to give feedback to? • Amelia • Humphrey • Oscar • Walter Mildred: Humphrey S/UM: What category do you wish to give feedback on? • Content • Structure • Argument • Style Mildred: Style S/UM: How do you assess the document for style? • Good • Okay • Weak Mildred: Okay S/UM: Please select comments which apply. • Style Inconsistent • Style Inappropriate And enter any other comments in the area below. Mildred: Style Inappropriate
• Other
Your writing is good, but informal. This is not really appropriate in this context.
Figure 3: Providing information to a student model
5. Contents of the S/UM Models Using an example of three students, the contents of the models might be as portrayed in Figure 4. USER MODEL 1 availability × collaboration √ cooperation × seeking: cont, arg offering: ______ STUDENT MODEL 1 content: WEAK structure: argument: WEAK style: GOOD
USER MODEL 2 availability √ collaboration √ cooperation √ seeking: arg offering: cont, struc STUDENT MODEL 2 content: GOOD structure: GOOD argument: style: OKAY
USER MODEL 3 availability √ collaboration × cooperation √ seeking: struc offering: cont, arg STUDENT MODEL 3 content: OKAY structure: WEAK argument: GOOD style:
Figure 4: Example of some of the contents of user and student models for three students
5.1 Contents of the User Models User models hold very simple representations, since they only need to contain information on which areas of writing students wish for assistance, or can give help, and their preferences for type of interaction and current availability for peer interaction. At the level of the system representation, this is composed simply of preferences with associated yes/no states, and lists of categories, where appearance on a list indicates a positive state for that category (i.e. If X is on list Y, then student X seeks/offers feedback on category Y). 5.2 Contents of the Student Models Student models contain a quantitative evaluation (GOOD, OKAY, WEAK)2 of each category for which feedback has been received. This is obtained from the quantitative feedback given by all peers who have contributed information. The algorithm used to calculate the quantitative evaluations of the student models is similar to that used for calculating the summaries of feedback given in See Yourself Write - essentially, all feedback offered in a particular category is ’averaged’, i.e. GOOD + WEAK = OKAY, with greater weighting awarded to later entries in the student model. In S/UM this occurs through the assignment of numbers to peer evaluations: GOOD = 6
OKAY = 4
WEAK = 2
For example, if there were two peer assessments for a particular category, the following calculation would be used: if peer evaluation 1 = GOOD (6) & peer evaluation 2 = WEAK (2): 6 + 2 = 8, 8 / 2 (number of evaluations) = 4. The overall evaluation of 4 is translated back to ’OKAY’. In less clear-cut cases, when the two individual peer evaluations are closer, but not identical, more weighting is awarded to the later entry in the student model, e.g.: entry 1 = GOOD (6) & entry 2 = OKAY (4): 6 + 4 = 10, 10 / 2 = 5. Since 5 is not an exact match for any of the possible peer evaluations, S/UM uses the most recent evaluation as the overall summary (i.e. OKAY). Should there be more than two peer entries in any category, the first two are evaluated as above. Each subsequent entry is then evaluated against the combined previous overall evaluation, e.g.: entry 1 = WEAK (2) entry 2 = GOOD (6) entry 3 = GOOD (6) entry 4 = WEAK (2)
=2 (2 + 6) / 2 = 4 (4 + 6) / 2 = 5 —> GOOD (6) (6 + 2) / 2 = 4 (overall = OKAY).
Thus later evaluations have more weight, in order that earlier ones which may no longer be relevant if the writer has since altered their document, are not represented so strongly in the student model. Student models will usually also contain qualitative textual information which is not used by the system, but is viewable by the owner of the model. Figure 5 is an extended example of SM 3 above. 2
Planned future work includes an investigation into the range of choices to offer.
Categories: Summary: Feedback 2:
content structure argument style other OKAY WEAK GOOD GOOD WEAK I find the structure of your paper a little hard. Instead of introducing related work as a starting point in the introduction, you have weaved it in throughout the paper. It would be easier to follow if you linked your main point to your statement of the deficiencies of previous studies, and built upon this. So, I suggest starting with related work and at this point indicating where the problems lie. Feedback 1: WEAK Although you support your argument well, there are some counter-arguments that you have not mentioned. Many could be overcome by your own claims, but you should probably show that you have considered them (e.g. you should probably mention more of the studies which produced different results to yours, and say why this did not occur in your experiment). OKAY GOOD The way you argue for your position is strong. You support it with evidence. OKAY GRAMMAR In general ok. But there are some problems with the choice of definite/indefinite articles. Figure 5: Example contents of a complete student model
Figure 5 illustrates a situation of feedback on three main categories (content, structure and argument) for a novice writer of a research paper. There is also an entry under ’other’ in the area of grammar, as this non-native writer of English has difficulties choosing between definite and indefinite articles. There are two separate entries in the categories of content and structure, and one piece of feedback given for argument and grammar. Therefore, as well as the overall summary for each category, information is stored as to the make-up of the summary, i.e. the individual evaluations provided by peers. As stated above, summaries are obtained by ’averaging’ the evaluations, as illustrated for content, with greater weight given to later feedback, as is the case here for structure. If there is only one piece of feedback in the model for a particular category, this is the sole source of the summary for that category (e.g. for argument in the above example). If there are no entries for a category (see style), no summary is given. There is also no summary for the category ’other’, as entries in this category may cover a wide range of issues, therefore a summary would be inappropriate. In addition to the quantitative evaluations, peers may provide supplementary qualitative descriptions. The aim here is to provide clarification to the recipient of the quantitative feedback, and also to promote reflection in the giver of feedback (see Bull & Brna, 1997). Since those offering feedback are not experts, the feedback cannot be relied upon as accurate. However, because the primary aim (of this implementation) of S/UM is to enhance awareness of important issues to consider when writing a document, if the recipient of feedback disagrees with the feedback, this disagreement will necessarily have occurred as a result of some degree of reflection. Indeed, the recipient may wish to challenge the feedback, as occurs with See Yourself Write, and this may, in turn, lead to the giver of the feedback also reflecting further. The two peers may wish to discuss the situation in greater detail in a face-to-face situation, as in peerSM. Such discussion is likely to be very rich even though neither learner is an expert, and it is likely to be beneficial to both partners (see Bull & Broady, 1997).
6. Viewing the S/UM Models
Figure 6 gives an example situation of the availability of models for viewing when two students are involved. For more students the situation is more complex, but follows the same principle.
UM 1
UM 2 S2
S1 SM 1
SM 2 S/UM
Figure 6: Access to the student and user models of S/UM
Figure 6 shows how each student (S1 and S2) has access to their own user and student models. Neither may directly inspect the student model of the other (though permission may be given for information in a student model to be disclosed - see below). S1 has direct access to a small part of the user model of S2 (UM2), because S2 is seeking help (or collaboration or cooperation) on an aspect of the task for which S1’s user model (UM1) indicates that S1 is willing to interact. S2 has no access to UM1 because there is no match between S1 asking for help on a problem where S2 can offer help (or collaboration/cooperation): only parts of a user model which are relevant to someone else are open to them. 6.1 Viewing the User Models Viewing the user models occurs in a different form depending on whether the model is being viewed by its owner or by a peer. The model’s owner sees the model as in Figure 2 above. Contents are shown in groups: areas where the user can give feedback; areas where they would like to receive feedback; their availability to provide feedback at the current time; whether they wish to cooperate with someone else; whether they wish to collaborate. In the example for the user Humphrey (Figure 2), the user is available to give feedback, and will give feedback on content and structure. He is seeking specific feedback on the style of his document, and is also open to receive general feedback. He would be happy to work cooperatively with someone, but not collaboratively. Humphrey may change any of these representations at any time (but no-one else can effect any changes in his user model). A peer viewer will receive a textual summary indicating those learners seeking help on an area where the peer has declared him/herself available for assistance through their own user model, and information about any suitable collaborative or cooperative partners, if appropriate (i.e. if the peer viewer has indicated a wish to collaborate and/or cooperate). The peer viewer, therefore, does not see the relevant part of the user model for each student as a separate entity, but descriptions from a number of models are combined. This grouping of information means that students may easily make a selection from amongst names offered in each section. Further information may be obtained about an individual if this is available (i.e. if that individual has given permission for further details to be disclosed). Such information consists of a summary of relevant representations in the individual’s student model. The summary of the relevant components of individuals’ user models appears to the peer viewer on request. Figure 7 shows an example for a user who has selected ’offer areas’ from the main menu. As seen in Figure 2, Humphrey is offering feedback on content and structure. If he chooses to view information from the user models of other individuals seeking feedback, he first receives a textual summary of his potential areas of contribution as represented in his own model. This is followed by specific statements drawn from the user models of others, about who is seeking feedback on areas where he could help. If any matches are possible for collaboration or cooperation, these are listed.
In this example Mildred is offering feedback on style, an area on which Humphrey would himself like feedback. He could receive this in return for helping Mildred on the content of her document.
Humphrey:
You are currently offering feedback on content and structure. You would like to cooperate.
The following people are seeking feedback on content: Mildred, Oscar. The following people are seeking feedback on structure: Amelia, Oscar, Walter. Cooperation:
Mildred is offering feedback on style, and would like to cooperate with someone. You could receive feedback from Mildred, and give Mildred feedback on content.
Figure 7: Information from user models as presented to potential helpers/collaborators/cooperators
6.2 Viewing the Student Models The owner of a student model receives the information in the model (see figure 5), but usually with the name of the giver of feedback included.3 For example, if Humphrey’s student model so far contained only the feedback on style given by Mildred (see Figure 3), his student model would be presented to him as illustrated in Figure 8 below.
FEEDBACK:
HUMPHREY
Categories: Summary:
content
Feedback 1:
OKAY (Mildred) Your writing is good, but informal. This is not really appropriate in this context.
structure
argument style
other OKAY
Figure 8: Viewing one’s own student model
If a user has granted permission for peers to have access to their student model, the information is displayed as above, but without the names of contributers.
7. Peer Interaction After receiving information about potential interactions, the responsibility to contact another person lies with the user, not the system. Humphrey may decide to contact Mildred to suggest a cooperative exchange of information. This may occur in any form (e.g. in person, telephone, email, S/UM4). Feedback may also occur by whatever method the participants choose (though a quantitative summary is always provided to S/UM). This multiplicity of forms of interaction is similar to the approach adopted for PHelpS (Peer Help System), where workers may receive help from others on aspects of their job with which they are unfamiliar (Collins et al, 1997). However the peer selection methods of the two systems differ: PHelpS selects potential helpers who have not recently provided a lot of help in preference to those who have been more active, in order to spread the burden more evenly amongst staff. Since S/UM is used in an educational setting, the question of burden is different. In some cases giving feedback might be seen by an individual as a burden however, because of the reinforced understanding which can occur through giving feedback, 3 4
Peers can block the appearance of their name if they wish, but this will prevent any subsequent discussion, and is therefore not encouraged. The mechanism by which peers may interact with each other through S/UM is not yet in place - in the present version of the system it is easier for peer communication to occur in some other way.
providing a lot of feedback could be a positive experience for many students. Thus the decision of how much feedback to offer is left to the students themselves. Another difference is that PHelpS uses task hierarchies. For the provision of feedback on writing as implemented in S/UM, this is less desirable. Categories for feedback (e.g. content, structure) and preferences (availability, interaction types), are each consulted and dealt with by S/UM independently. Humphrey may be working with Mildred in the categories of content and style, and may also be following some kind of interaction with somone else about either of these, or about a different area. If he is working on a particular category with more than one person, the interactions and aspects addressed within that category might be completely different in each case. On commencing some kind of peer interaction a learner may choose to edit their user model in some way. Continuing with the above example, when he starts working with Mildred, Humphrey may decide to reduce his availability to ’possibly available’ in order that he is less likely to be suggested by S/UM for future interactions. Mildred, on the other hand, might choose to focus other feedback she receives onto areas where she has, as yet, no feedback. She may therefore delete ’request feedback on content’ from her user model. This will, of course, not necessarily result in her receiving more offers of feedback on other areas. However, in cases where someone is offering feedback on content and on another area where Mildred would like help (e.g. argument), because Mildred has deleted her request for help on content from her user model, she is more likely to receive it on the argumentation strategy of her document. Because of the ease of directly editing user models, the models may change frequently, and potential recommendations of peer interactions will therefore also be changing frequently.
8. The Selection of Potential Partners by S/UM A few indications have already been given of features involved in selecting potential partners. In this section the method is described in greater detail. S/UM uses both the user and student models in its matching of partners. Imagine a case where there are 6 students (A - F) using S/UM, and 2 categories where feedback is currently being requested (X & Y). Figure 9 shows the underlying representations in the relevant parts of the user and student models. (The lists shown for some predicates are, in fact, intermediate stages constructed from separate representations for each individual.)
DESCRIPTION Student A requests feedback on category X B, E and F are available C is possibly available B, C and F are offering feedback on X B and F have representations for X in SMs
REPRESENTATION request(A,[X]). available([B,E,F]). poss_available([C]). offer(X,[B,C,F]). repSM(X,B,’GOOD’). repSM(X,C,none). repSM(X,F,’WEAK’).
MODEL user model user model user model user model student model student model student model
Figure 9: Example representations in S/UM models
Scores are assigned to the states of availablilty represented in the user models: available: +2
poss_available: +1
not_available: +0
The suitability of a peer to give feedback on a particular area is calculated from their performance in that area as represented in their student model. Where there are no representations for a category a mid-way score is given - they may not have requested feedback in this area because they feel they do not need it (i.e. they are ’GOOD’), or they may simply have not yet received any feedback on this area. However, the fact that they are offering feedback indicates that they believe themselves able to
do so, so the score awarded is between that for OKAY and GOOD. The suitability rating is calculated as follows: GOOD: +3
OKAY: +1
WEAK: +0
no representation: +2
S/UM then calculates the following, from these scores: B = candidate to provide feedback on X: C = possible candidate to provide feedback on X: E = unlikely candidate to provide feedback on X:
availability 2 + suitability 3 = 5 availability 1 + suitability 2 = 3 availability 2 + suitability 0 = 2
The highest possible score is 5, which can only be achieved if an individual is fully available (score 2), and has the maximum score for competence in an area (3). Anyone scoring 5 will definitely be recommended for interaction. In the above example, S/UM will therefore suggest student B as a potential giver of feedback to student A. In cases where S/UM is required to make multiple recommendations the situation becomes more complicated. For example, it would be unwise to recommend someone to interact with more than one person at a time, in case that particular student has not had time to update their user model for availability. The following example illustrates the selection mechanism where there are multiple cases. Table 1: Example of multiple requests REQUESTER A A A C C C
AREA X X X Y Y Y
OFFERING B C F B E F
AVAILABILITY 2 1 2 2 2 2
SUITABILITY 3 2 1 3 3 2
Student A is requesting feedback on area X, and C is requesting feedback on Y. B and F are offering feedback on both these areas; C is offering feedback on X; E is offering feedback on Y. For A’s request for feedback on X, the scores are:
For C’s request for feedback on Y, the scores are:
B: availability 2 + suitability 3 = 5 C: availability 1 + suitability 2 = 3 F: availability 2 + suitability 1 = 3
B: availability 2 + suitability 3 = 5 E: availability 2 + suitability 3 = 5 F: availability 2 + suitability 2 = 4
In the above example5, B scores the maximum (5) for both categories. However, as stated above, only one recommendation can be made for each person. In this case, for Y there is another user also scoring 5 (E), so S/UM will suggest B as a potential helper for X, and E as a potential helper for Y. To give another example, imagine E scored 4 instead of 5 for Y. This time S/UM will suggest B as a potential helper for Y, and E and F as potential helpers for X. This is because B again scored the highest in each area, but there was no second person with the same score in either category. S/UM will then check the next highest score for each. In this case, for X the second highest score is 3, and for Y it is 4. Those scoring 4 are either more likely to be able to offer useful feedback than those scoring 3, or are more available than those scoring 3. 5
Although these calculations for individuals are made, the algorithm for selecting between them is not yet fully implemented. The description that follows relates to how selection will be made once this algorithm has been completed.
A final point to note here is that learners are not restricted to the suggestions of S/UM. As in most settings, students are free to approach each other without using the system. S/UM is designed for use by those who wish to use it, and students who do choose to use it are likely to receive useful suggestions for peer interaction in part because all those participating have themselves selected to take part, and therefore want to be working with their peers.
9. Summary and Conclusion S/UM has been designed based on a number of systems which use inspectable student models as a source of learner reflection, and systems which mediate peer interactions. The main aim of S/UM is to encourage peer interaction in a way that suits the particular students, taking account of their preferred type(s) of interaction, and their relative abilities in different aspects of the task. This interaction should encourage greater understanding of the task requirements, thereby leading to improved results. S/UM draws from information in user and student models to suggest: potential givers of feedback to someone seeking feedback; potential cooperative partners to someone who seeks cooperative interaction; potential collaborative partners to someone who wishes to collaborate. The kind of information consulted includes the availability of an individual to participate in peer interaction at the current time; the categories in which they have declared themselves willing to interact; their performance in these categories to date. After completion of the algorithm for selection amongst multiple requests, the next stage of the work will be to evaluate the system with a variety of users writing different kinds of document. Extension of S/UM to other domains will also be considered.
References Bull, S. (1997). See Yourself Write: A simple student model to make students think, in A. Jameson, C. Paris & C. Tasso (eds), User Modeling, Proceedings of the Sixth International Conference, Springer Wien New York, 315 - 326. Bull, S. & Brna, P. (1997). What does Susan know that Paul doesn’t? (and vice versa): Contributing to each other’s student model, to appear in Proceedings of World Conference on Artificial Intelligence in Education, Kobe. Bull, S. & Broady, E. (1997). Spontaneous peer tutoring from sharing student models, to appear in Proceedings of World Conference on Artificial Intelligence in Education, Kobe. Bull, S. & Pain, H. (1995). ’Did I say what I think I said, and do you agree with me?’: Inspecting and questioning the student model, Proceedings of World Conference on Artificial Intelligence in Education, Washington DC, 501 - 508. Chan, T-W., Chung, I-L., Ho, R.G., Hou, W. & Lin, G-L. (1992). Distributed Learning Companion System: WEST Revisited, ITS-92 Lecture Notes in Computer Science, SpringerVerlag. Collins, J.A., Greer, J.E., Kumar, V.S., McCalla, G.I., Meagher, P. & Tkatch, R. (1997). Inspectable User Models for Just-In-Time Workplace Training, in A. Jameson, C. Paris & C. Tasso (eds), User Modeling, Proceedings of the Sixth International Conference, Springer Wien New York, 327 - 337. Cumming, G. & Self, J. (1991). Learner modelling in collaborative intelligent educational systems, in P. Goodyear (ed), Teaching Knowledge and Intelligent Tutoring, Ablex Publishing, Norwood NJ, 85 - 104. Hoppe, U. (1995). The use of multiple student modeling to parameterize group learning, Proceedings of World Conference on Artificial Intelligence in Education, Washington DC, 234 - 241.
Kay, J. (1994). The um toolkit for reusable, long term user models, Report 94/3/36.2, Systems Centre, University of Sydney. Paiva, A., Self, J. & Hartley, R. (1995). Externalising learner models, Proceedings of World Conference on Artificial Intelligence in Education, Washington DC, 509 - 516. Pemberton, L., Shurville, S. & Sharples, M. (1996). External representations in the writing process and how to support them, Proceedings of Euro-Conference Artificial Intelligence in Education, Lisbon, 343 - 349. Puntambekar, S. (1995). Investigating the effect of a computer tool on students’ metacognitive processes, PhD Thesis, University of Sussex. Self, J. (1991). Formal approaches to student modelling, Tech. Report AI-59, Lancaster University. Smith, M. & Bull, S. (1997). Computer mediation of peer interaction: Choosing recommendations, to appear in Proceedings of International Conference on Computers in Education, Sarawak.
