1 AIED'99. Lemans (July, 1999)
An approach to analyse collaboration when shared structured workspaces are used for carrying out group learning processes Barros, B. & Verdejo, M.F. Departamento de Ingenieria Electrica, Electronica y Control, Escuela Técnica Superior de Ingenieros Industriales (U.N.E.D) Ciudad Universitaria s/n, 28040 Madrid, Spain Email
[email protected] phone: 34 - 91 - 398 64 84 fax: 34- 91 - 398 60 28 Abstract In this paper we present an approach to characterize group and individual behaviour in computer-supported collaborative work in terms of a set of attributes. In this way a process-oriented qualitative description of a mediated group activity is given from three perspectives: (i) a group performance in reference to other groups, (ii) each member in reference to other members of the group, and (iii) the group by itself. In our approach collaboration is conversation-based. Then we propose a method to automatically compute these attributes for processes where joint activity and interactions are carried out by means of semi-structured messages. The final set of attributes has been fixed through an extensive period of iterative design and experimentation. Our design approach allows extracting relevant information at different levels of abstraction. Visualization and global behavior analysis tools are discussed. Shallow analyses as presented in this paper are needed and useful to tackle with a large amount of information, in order to enhance computer-mediated support.
1.Introduction Collaborative learning research has paid closed attention to study pupils interactions during peer-based work in order to analyze and identify the cognitive advantages of joint activity [6]. As [5] points out, the benefit of the collaborative approach for learning lies in the processes of articulation, conflict and co-construction of ideas occurring when working closely with a peer. Participants in a problem-solving situation have to make their ideas explicit (assertions, hypothesis, denials..) to other collaborators, disagreements prompt justifications and negotiations, helping students to converge to a common object of shared understanding. The computer provides opportunities to support and enhance this approach in a number of ways, for instance, offering computer-based problem spaces for jointly creating and exploiting structures of common knowledge and shared reference. Moreover, networks make possible opening the collaborative framework to distributed communities providing remote access to these spaces as well as computer-mediated communication to support interpersonal exchange and debate. An increasing number of collaborative learning environments for open and closed virtual groups have been built for a range of learning tasks [8][13][11][12], and experiences of use are reported from school to university level [2][4].
2 In this paper, we are focusing on the analysis of computer mediated interaction in collaborative learning processes. In our approach collaboration is conversation-based. Conversation consists of turn taking, where each contribution both specifies and grounds some content [3]. The type of contributions and their constraints can be defined to establish an explicit structure for a conversation. For learners the benefit is twofold (1) they receive some support for the process of categorising and organising their ideas when contributing to the debate and (2) further inspection of the process is facilitated because the system can take into account the type of units. Most of the automatic analysis of computer mediated interactions have been up to now based on quantitative terms, a variety of qualitative analysis have been performed mainly manually (see for instance [6]), with a few attempts to carry out automatic natural language processing [9]. Our goal is to design a tool to facilitate the handling of a large amount of data, in order to have a view, at an abstract, qualitative level. Our proposal aims to describe the individual and group behaviour in terms of a set of features to characterize collaboration when performing a task. This is a first step towards the complex goal of understanding how interactions are related to learning processes and outcomes. The following section gives an overview of our system. A process-oriented visualization tool is presented in section 3. The qualitative analysis is discussed in section 4 and elements for further research are pointed out in the conclusions. 2. The collaborative environment The main metaphor in our system for sustaining a learning activity is the concept of space, a virtual structured place with resources and tools to perform a task. Three types of spaces are available, an individual workspace, private for each user, and two types of shared spaces: one workspace for debate and joint construction, the other for coordination purposes. The information handled is mainly textual, so a variety of editing tools and file management facilities are available. Links to other relevant electronic sources of further information for the task at hand can also be included. A shared workspace provides support for conversation in the form of semi-structured typed messages. When learners express their contributions they have to select a type from a predefined set. Workspaces are defined in the configuration mode, an authoring tool is provided to define the workspaces associated to a learning task. The architecture of the system is organized into four levels: configuration level, performance level, reuse level and reflection level [1] For example, let us consider an experience comprising two collaborative activities: writing an essay and designing a case, where the first one, in terms of Activity Theory [10] is described as follows: The object of the learning activity is to make the survey in a focused research topic on Educational Technology. The outcome of the activity is an essay, pointing out the key ideas. The schema for the essay has been previously defined by the tutor, and included in the shared workspace. The community involved in the activity is made up of pairs of graduate students in a Ph.D. distance teaching programme. Students are part time, geographically distributed, usually interacting in disjointed time slots. The subject, each individual student, has either a technical or non technical profile, but groups are formed by people with the same background. Mediational tools include humans, a tutor and a technical assistant, and artifacts: the phone, a set of documents and the system (a networked computer environment including the script of the activity, private and shared structured task spaces for peer argumentative elaboration and coordination, as well as access to other electronically available sources of information). Rules for the activity are stated explicitly and have to be accepted before starting. They include the commitment to finish the work, the script for the activity and the protocol for the collaborative debate. Some of these rules are embedded in the system. For example, the conversational graph defines the way one can respond to another peer’s contribution. Agreement has to be explicit and reached by consensus. Others rules are the full responsibility of the learners, such as the way to organize the discussion, as well as aspects of time management and deadlines. The only guideline given to the groups is to use the coordination
3 space to deal with all these matters, using the elaboration workspace only for the topic discussion. Students in each group have the same responsibility, there are no predefined roles or division of labour. ENTRY FOR THE GROUP
Menu with links to workspaces
Accesses to workspace subspaces
• ELABORATION SPACE
‚ Selected contribution subtask
Form for replying to QUESTION34
Contribution types related to QUESTION Links to contributions related with PROPOSAL10
Contribution link
RESULT SPACE
Votes for first section
ƒ
Final document
Figure 1. Interfaces of the Elaboration Space and Result space of final version
The learner interface provided by the system for this experience is shown in figure 1. The upper window shows the interface for a groupwork. The bar shows the number of activities and the associated workspaces. For accessing activity1 one’s elaboration space the user clicks in the space name (labeled 1 in the figure) and the elaboration space window appears. This interface is organized in three areas. The upper area is a menu for accessing activity workspaces. In the left area we can see the task schema which has been automatically updated by the system and on the right side - after clicking on one of the contribution names the whole content of the contribution appears. Labels on the window explain each element of the interface. For moving to the result space, the user has to click the checkbox with the space name (labeled 2 in the figure) and the result space appears showing the state of the votes for each section on the right. The final document can be seen, on the right of the window, pressing the option labeled 3 in the figure. 3. Visualizing the process The system records all the accesses and the actions performed by each user. The information automatically registered includes user identification, time and date, host computer, learning experience, group, activity, task, workspace, and type of action. A relational database stores
4 this information so a variety of queries combining a selection of criteria can be easily formulated and the results are displayed either in textual or graphical format. A web-based interface with buttons is provided for selecting the options for a query. Examples of queries follow: • • • • •
• •
Evolution of the number of users contributions in an experience along a period, using a graphic display. Plot the number of hourly accesses for a group in an activity. Number of contributions by user, for all the group members in an activity. Number of contributions by user, by type, for each substask of a task. Contributions of a group, related to a workspace task, by type of contribution, displayed as a chart bar. The discussion process is summarised in term of the type units, in this case proposal, contraproposal and agreement have been the backbone of the process. Only a few comments, questions and clarifications have been performed. This graphic together with the previous question provides a good overview of the dynamics of a groupwork. Number of contributions, by member, for an activity (all the workspaces supporting the activity), graphic display. Evolution of a discussion for a substask, graphic display. We can observe here a first proposal followed by a contraproposal and a comment from the same author. A turn taking happens with a peer contraproposal and then two contraproposals plus a comment from the first author, a question from his peer follows, and after some time without receiving a response this student takes the initiative making a new contraproposal. In this way they finally reach an agreement
The question of who many request this data is a matter to be decided at the setting-up of the collaborative learning experience, when rules for the group and division of labour are established. The system, in configuration mode, allows us to define restrictions and permissions related to roles. Roles can be assigned to users for each activity. The tool for visualizing group processes becomes available for authorized users by selecting the reflection mode. This process oriented data analysis can be used for a variety of purposes. For example, during the activity, an insightful presentation of individual contributions to the group task (as in figure 3) can increase presence awareness for the rest of the group and stimulate peers to contribute just in time. Furthermore, it may support pedagogical decisions. For instance, to carry out an external intervention, either by artificial or human agents, which suggests relating contributions to each other. 4. Qualitative Analysis Criteria and methods to evaluate whether a collaborative learning process has been carried out is a controversial and open question in the field. But al least, from a practical point of view, we need to identify, even roughly, if and when students have been addressing each other and working together. For this purpose we can exploit some of the evidence provided by the student contributions while performing a task. We propose three kinds of analysis to characterize: ⋅ Group behavior compared to other groups performing the same task. ⋅ Group behavior in itself ⋅ Individual student behavior compared to the rest of the group members We will express the result of the analysis in terms of a set of features (attribute-value pairs), subjectively established but tested and refined through an extensive experimental period of system use.
5 Figure 4 shows the attributes considered for group comparative behavior. Values for these attributes are either (i) calculated from data of the task definition and the process performed or (ii) concluded from fuzzy inference using a set of evaluation rules, for those attributes appearing with a dark background in the figure. These rules, which are obtained from relationships among attributes, as shown in the figure, can be applied to the calculated variables to infer new ones. Those variables, in turn, can be the entry for another rules giving rise to a chained inference process. Next we will describe in detail these features. Coordination Messages Initiative
MContributionsNumber
Coordination
MContributionsSize Work Elaboration
Argumentation
Colaboration
Interactivity Deph tree
Conformity
Cooperation
Creativity
Figure 4. Relationship among attributes
4.1 Analysing group behavior Data for the process analysis of a learning experience comes from two sources: •
The definition of the experience, embedded into the system configuration, i.e. tasks workspaces associated to each activity. In particular their conversational graphs and the group definition structure. The conversational graph specification includes the definition of values for a set of attributes for each type of contribution. We propose four attributes taking values in a range of (–10,10) ⋅ Initiative, indicates the degree of involvement and responsibility required to perform a contribution. ⋅ Creativity, relates to the degree of originality required to produce this type of contribution ⋅ Elaboration, qualifies the workload needed for making a contribution. ⋅ Conformity, establishes the degree of agreement of a contribution in relation to another selected and linked contribution. For instance making a contraproposal to a proposal indicates a low degree of conformity to the proposal.
Figure 5 shows the feature structure associated to the conversational graph of the elaboration workspace presented in figure 1. CONTRIBUTION A
P
CO
CN Q CL
initiative
creativity
10 10 2 4 4 0
10 9 1 3 2 0
j=(1,k)
Proposal (P) Contraproposal (CN) Question (Q) Comment (CO) Clarification (CL) Agreement (A)
elaboration
conformity
Vji
10 9 1 2 2 0
-10 -10 3 0 0 10
Figure 5. Conversational graph and a table with the attributes and values for each contribution type
6 •
The learners’contributions organised as tree-like structures for each task workspace, and the set of messages interchanged through the coordination space associated to each activity.
From this data we consider and compute the following attributes: For each elaboration workspace Mean Contribution’s number: number of contributions from the group/number of group members ⋅ Mean Contribution’s size: mean size of contribution contents (in characters) ⋅ Depth of the Contribution’s tree: Maximum depth of the trees related to the workspace ⋅ Interactivity: Percentage of contributions responded or linked to other contributions made by a student, other than the contributor. The values Vai for each i attribute: Initiative, Creativity, Elaboration and Conformity are computed by the formula below: Vai = Σkj=1 (Nj * Vji) where Nj is the number of contributions for each type j, and Vji the value of the attribute for that type of contribution as defined in figure 5. ⋅
For the coordination space, total number -by type- of messages. Conclusions are established by a fuzzy inference process. The logical product of each rule is inferred to arrive at a combined magnitude for each output membership function, by the MAX-MIN method. Then a defuzzification process is carried out for the output generation. The defuzzification function is performed by mapping the magnitudes to their respective output trapezoidal membership functions and computing the fuzzy centroid of the composite area of the member functions. In order to perform fuzzy inference, numerical values of the computed attributes have to be first of all mapped from their numerical scale to a linguistic label expressing their degree of membership. This is performed by a fuzzification function. As the scales have to be comparable and adjusted to a task, the range defining each attribute is not the same for all the tasks, but calculated for each task related to a mean reference point (MRP). For instance, for a normal task 50 contributions is a typical number of contributions, so 55 is the top value for the adequate linguistic label, while for a long task this could be in the range of low. Mean reference points can be dynamically calculated taking into account all the similar tasks performed in all the learning experiences carried out with the system. Figure 6 shows the results of the global analysis for a particular group learning experience.
7
Figure 6. Results of group behavior for an experience, comparing each task with other groups performing the same task
4.2 Individual behavior This analysis is done for each student in a group for a learning task, the method is the same as for the group behavior, but in this case we consider a different set of attributes and rules as illustrated in figure 7. For instance the attribute promote discussion is inferred from the attributes: (i) Answered, contributions responded to others, (ii) Was Answered, own-contributions responded by peers, (iii) continued, contributions answered by the same user and (iv) proposal moves made by this person along the process. Figure 8 shows the result of an analysis where attributes are displayed, for more details see [1]. MContributionsNumber MContributionsSize
Work
Elaboration Answererd Was Answered Valuation
Promote Discusion Continued Initiative Propose
Attitude Creativity Conformity
Figure 7. Relationship among the attributes in individual analysis
8
Figure 8. Results of individual analysis of a group member
4.3 Group task behavior summary This analysis focuses on two aspects: distribution of work between group members and evolution of group activity in a period of time. We use the same data as before, (i.e. definition of the task and process contribution trees), but in this case we have to consider a clustering of attributes as indicated in table 1 for the definition of a task. CONTRIBUTION CATEGORY Proposal propose Contraproposal propose Question argue Comment argue Clarification argue Agreement agree Table 1. Category attributes considered for group task behavior
Conclusions are reached in this case by comparing computed data from each user with the rest of the group. Figure 9 shows an example of results obtained from a particular task and a group of students. On the left we can observe the evolution of the task performed in three periods of activity, the first being where more contributions occurred. On the right, for each
9 subtask, a conclusion of student’s participation is given. The result in this case may suggest that collaboration, but also some division of labour, has really happened. Group name
task name
Work evolution by time
subtasks
Conclusion sabout the way of working by subtasks
Figure 9. Intensity and evolution of work on the left, and participation by substask on the right
5. Summary and conclusions Computer mediated collaborative learning allows the recording of a large amount of data about the interaction processes and the task performance of a group of students. This empirical data is a very rich source to mine for a variety of purposes. Some of practical nature like, for instance, the improvement of peer awareness on the on-going work. Others of a more long-term and fundamental scope such as to understand socio-cognitive correlations between collaboration and learning. Manual approaches to fully monitor and exploit these data are out of the question. A mixture of computable methods to organize and extract information from all this rough material together with partial and focused in-depth manual analysis seems a more feasible and scalable framework. In this paper we have presented first of all an approach to characterize group and individual behaviour in computer-supported collaborative work in terms of a set of attributes. In this way a process-oriented qualitative description of a mediated group activity is given from three perspectives: (i) a group performance in reference to other groups, (ii)each member in reference to other members of the group, and (iii) the group by itself. Then we have proposed a method to automatically compute these attributes for processes where joint activity and interactions are carried out by means of semi-structured messages. The final set of attributes has been fixed through an extensive period of iterative design and experimentation. We do not make theoretical claims about this particular set, the only worth of this proposal is practical value and therefore it is open to further refinement. The method uses the feature structures associated to the conversational structure of shared workspaces as data, therefore the attributes considered can be easily redefined. Moreover in our system these are specified in a declarative way when configuring the computer environment for a learning experience. The results about the processes can be used in many ways for pedagogical purposes. Promote reflection is an interesting one, for instance, one could extend the experience with a peer reflection phase, making visible this information either within or among the groups of students performing similar tasks. Furthermore, it is quite useful input for teachers and designers, to evaluate whether the definition and the support provided for the collaborative task has been adequate. To combine process-oriented evaluation together with manual evaluation of the final product has proved for us a reasonable approach to deal with assessment matters in our distance learning courses. A comprehensive theoretical perspective for analysing collaborative learning has been and would be a trend for the research community, partly depending on the development of computable methods for analysing process-oriented data. Natural language processing would allow a categorization of contributions from a content analysis, but current NLP techniques require expensive resources and processing. To consider semi-structured interactions are a non-intrusive way to break this complexity at least for a broad range of tasks. Our design approach allows extracting relevant information at different levels of abstraction. Visualization and global behavior analysis tools are just two of them. A fine-grained study of collaborative interactions would require a more extensive
10 modeling not only dealing with task and communication aspects but also with learner’s beliefs. Nevertheless shallow analyses as presented in this paper are needed and useful to tackle a large amount of information, in order to enhance computer mediated support. Acknowledgements The work presented here has been partially funded by CICYT, The Spanish Research Agency, project TEL97-0328-C02-01. We would like to thank our UNED, UCM and UPM students, for their participation in the experiences carried out while designing the system prototypes. References [1] Barros, B. (1998) Aprendizaje Colaborativo en Enseñanza a Distancia: Entorno genérico para configurar, realizar y analizar actividades en grupo. Technical Report. STEED Project. [2] Bell, P. & Davies, E. A. (1996) “Designing and Activity in the Knowledge Integration Environment”,1996 Annual Meeting of the American Educational Research Association, New York. (http://www.kie.berkey.edu/KIE/info/publications/AERA96/KIE_Instruction.html) [3] Bobrow, D. (1991) “Dimensions of Interaction: AAAI-90 Presidential Address”, AI Magazine, Vol.12, No. 3, pp. 64-80. [4] Collis, B. (1998) “Buidling evaluation of collaborative learning into a WWW-based course: Pedagogical and technical experiences “, Invited paper for the Special Issue on Networked and Collaborative Learning of the Indian Journal of Open Learning, New Delhi, India. (http://utto237.edte.utwente.nl/ism/ism1-97/papers/col-eval.doc ) [5] Crook, C. (1994) Computers and the Collaborative Experience of Learning, Routledgem International Library of Psychology. [6] Dillenbourg, P. & Baker, M (1996) Negotiation Spaces in Human-Computer Collaborative Learning. Proceedings of COOP'96. (Juan-Les-Pins, France,June). http://tecfa.unige.ch/tecfa/publicat/dil-papers/LastJuan.fm.ps [7] Dillenbourg, P., Baker, M., Blaye, A. & O’Malley (1996) “The evolution of research on collaborative Learning”, Learning in Humans and machines, Spada & Reimann (editors). http://tecfa.unige.ch/tecfa/research/lhm/ESF-Chap5.ps [8] Edelson, D.C., Pea R.D. & Gomez L.M. (1996) “The collaboratory Notebook”, Communications of the ACM, Vol 39, No. 4, April, pp 32-33. [9] Henri, F. (1992) “Computer Conferencing and Content Analysis”, Collaborative learning through computer conferencing: the Najaden papers, Kaye, A.R. (Editor), Springer-Verlag, pp. 117-136. [10] Nardi, B.A. (1996) (Editor) Context and Consciousness. Activity Theory and Human-Computer Interaction, MIT Press. [11] Scardamalia, M. & Bereiter, C. (1994) ”Computer Support for Knowledge-Building Communities”, The Journal of the Learning Sciences, Vol. 3. No.3, pp. 265-283. [12] Suthers, D. & Jones, D. (1997) "An architecture for Intelligent Collaborative Educational Systems", Proc. AI-ED'97, B. Du Bovlay and R. Mizoguchi (editors) pp.55-62. [13] Wan D. & Johnson P. (1994), “Experiences with CLARE: a computer-supported collaborative learning environment” Int.J.Human-Computer Studies, vol. 41, pp 851-859.
