Re-Arranging: usually protocols are ordered by chronological criteria, such as time stamps. For specific purposes it may be more interesting to re-arrange the ...
Processing and Transforming Collaborative Learning Protocols for Learner’s Reflection and Tutor’s Evaluation Andreas Harrer and Lars Bollen and Ulrich Hoppe1 Abstract. This paper discusses the potential of processing and analysis of collaborative learning protocols, which in the first place are represented as low-level logfiles of actions in collaborative learning environments. These protocols can be abstracted, analysed, and augmented to provide the users with feedback about the collaboration process. We present several techniques we implemented which may be combined to support learner’s self-reflection as well as tutors in evaluating the process.
1
INTRODUCTION
While in the beginning CSCL-Systems focussed on the mediating function of the system, i.e. to provide an environment where collaboration can take place, in recent years the importance of functionality actively supporting the collaborative learning process has grown. For that the system has to incorporate some techniques for an understanding of (parts of) the collaborative processes. Some of these CSCL-Systems considered mainly the pragmatic level of linguistic contributions, while others concentrated on activities on the domain level and the resulting artefacts of collaboration. The first line of research used either techniques for understanding of textual contributions [7] or classification of contributions on pragmatic level with semi-structured interfaces, such as sentence-openeners [8]. The second line of research investigated in an understanding of domainrelated actions in shared workspaces with techniques like plan recognition [5] or analysis of the resulting states of collaboration [1] with checking of structural properties. Because in our scenarios of using CSCL-systems we typically cannot assume the system to be aware of the complete interaction (e.g. in classroom examples the students usually have computers on their own or in small groups, but they can talk to each other unmonitored by the CSCL system), our approaches focus on an understanding of actions and states on the domain-level of shared workspaces [2]. In this paper we describe techniques we designed for processing and transformation of collaborative learning protocols, i.e. we take an action-based viewpoint for the domainlevel.
2
TECHNIQUES FOR PROTOCOL TRANSFOMATION
Learning Protocols are readily available in CSCL-systems simply by logging the activities that take place in the collaborative environment. Especially in distributed architectures with a communication server these logfiles are produced automatically by logging the messages 1
Universit¨at Duisburg-Essen, Institut f¨ur Informatik und interaktive Systeme, Lotharstr. 63/65, 47057 Duisburg, Germany, email: {harrer, bollen, hoppe}@collide.info
sent from learners’ machines. In the first place these logfiles are at a very low abstraction level and of little use for teachers, learners, and even the system for creating an understanding of the collaboration going on: Depending on the content of messages transmitted the actions can be either related to domain objects (such as changing the content of a node in a concept map) or even lower-level as user-interface events (such as mouseclick at position (100,50)). In our collaborative applications the MatchMaker [4] communication server creates object-oriented logfiles, i.e. actions are related to the objects they operate on (e.g. creating, deleting an object, changing attributes of an object). These logfiles contain information about the user conducting the action, the original creator of the object and other issues, that may be processed to interesting information about the collaboration going on. Even though this format is not as low-level as pure user-interface-events, the logfile itself is not very useful for learners and teachers in its raw form (here in XMLrepresentation): typeOfAction="setTokens"
Table 1. Excerpt from a MatchMaker logfile
To gain a better understanding of the collaboration, different techniques are available to transform low-level protocols to higher level protocols: • Abstraction: each logfile entry is looked at in an abstracting, classifying manner; for example some actions can be classified as structure-changing, some as meaning-changing and some purely with graphical purpose. Abstraction helps to avoid information overload caused by too detailed protocol formats. • Filtering / Querying: Processing the logfile and deciding for each entry if it is kept or dropt in the resulting protocol we call filtering; more general, querying describes the process of keeping or dropping user-specified patterns, called queries. In that sense filtering can be seen as a special variant of querying, where the pattern consists of just one entry. With these techniques specific interesting events or sequences in learning processes can be found. • Re-Arranging: usually protocols are ordered by chronological criteria, such as time stamps. For specific purposes it may be more interesting to re-arrange the whole protocol according to other criteria, such as all entrys associated to an object are succeeding, or all entrys generated by a specific user. With this transformation no information is lost in the protocol. • Augmenting: In combination with querying techniques a protocol can be augmented with information that indicates specific situations/phases. For example a recurring pattern of create/delete object may indicate a trial-and-error strategy of the learners; if specified by a teacher and searched for in the protocol the protocol could be annotated and thus augmented for further use or transformations. This augmentation can be event-oriented (add an indicator at a specific location) or scope-oriented (attribute an information to a whole block or sequence of entries). Each of these steps of protocol transformation results in another learning protocol, so the different techniques can by used in combination by applying transformations succesively. Abstraction, filtering, and querying potentially lose information, while re-arranging preserves and augmenting even enriches the information, so the sequence of transformations should be chosen carefully. In the next section we present different implementations of protocol transformations that may be used within or in combination with our modelling environment Cool Modes [6].
3
IMPLEMENTATIONS
As we already mentioned in the previous section, the primary source of learning protocols is the XML-based logfile produced during a MatchMaker session. Since some of our transformations described in the following are done on-the-fly, we use as secondary source the synchronisation messages broadcasted from the MatchMaker server to each connected learner’s application (these broadcasted messages are used to synchronize the states of several instances of our application and represent the learner’s actions, which are also stored persistently as logfile entries in XML-format). This avoids incremental parsing and transformation of the XML-document which would be needed for adequate performance of on-the-fly transformation(s) on logfiles. • Abstraction: to provide immediate feedback we implemented an on-the-fly feedback tool within the Cool Modes environment. Figure 1 shows a so called protocol node in a Cool Modes workspace. This protocol node is notified on each action ocurring in this workspace and displays a textual abstract classification of this
action. At the moment we have 4 different action classifications (simulation relevant, semantical, specific for learner’s preferences, optical), but the number and categories are easily exchangable [3]. One interesting aspect of classifying actions is that, depending on the modelling method used in Cool Modes, actions may have different meanings: for example moving the position of a place node in a Petri-Net is not relevant for the meaning or even for simulation of the model, and therefore is primarily optical, while in a concept map moving one concept node closer to another usually has meaning for the users. Considering this we allow the specification of classification categories for each modelling method (realized as a plugin with the relevant domain semantics) separately.
Figure 1.
Visualization of protocol abstraction in Cool Modes
• Filtering / Querying: in a student’s seminar we implemented a stand-alone logfile processor, which searches for patterns of learner activities in MatchMaker logfiles. The patterns are specified with a graphical user interface, where the steps of the pattern and the users conducting the actions are defined (e.g. different users interacting on one object). Technically these patterns are compiled to JESS rules (Java Expert System Shell) and the JESS engine is used for extracting instances of the specified patterns. The results are then visualized with information about users making the activities and time information. An example with patterns of actions found in a collaboration of three students can be seen in figure 2 where time is the dimension along the x-axis and the students are along the y-axis with the pattern’s elements plotted with the time/user combination.
Figure 2.
Visualization of pattern instances in protocols
At the moment the querying process searches for exact matches; partial or similar patterns are not yet considered, but are a topic of future development. • Re-Arranging: for specific purposes, such as one student’s self reflection of activity traces, a re-arrangement of protocols may be useful. We implemented some scenarios for re-arrangement (such as grouping user specific entries or object type specific ones) of logfiles by using XSL-transformations on XML-based logfiles. These rearranged protocols are presented directly in tabular format. • Augmenting: In the student’s seminar mentioned above the technique of augmenting logfiles with indicators was implemented, too: for each pattern found in the XML-based logfiles a breakpoint was included at the log’s position where the pattern was completed. This breakpoint is used for the replay-mechanism [4] of our Cool Modes environment: With that replay mechanism a stored logfile can be used to review the temporal development of a learning session. Breakpoints enable a direct navigation to the situation for which the breakpoint was set (e.g. an important partial solution). Usually these breakpoints are set manually, but in the method presented here the breakpoints are introduced automatically for each pattern instance found. To further highlight the pattern, a node with a textual description representing the pattern was added to the workspace with relations to all the objects that are used in the instance of the pattern. Thus a teacher can review the session by defining specific patterns and can jump directly to the situations where the patterns are completed with highlighting of the relevant objects. In our implementations, the abstraction method is strongly domain-dependant, while querying, re-arranging, and augmenting at the moment do not use any domain-specific representation within the system; so the specific interpretation has to come from the user of the analysis methods.
4
USE OF PROTOCOL TRANSFORMATIONS
For the use of combining several of the techniques of logfile transformation we sketch two use cases for the following learning scenario: Two students collaborate in a spatially distributed system to model the dynamic behaviour of a system, e.g. in System Dynamics notation. After creating a first solution the students explore their model by simulating repeatedly and calibrating the value of constants, so that in the end the model shows the desired behaviour. This scenario can be analysed by using the processing techniques mentioned above in variuos ways and for different users’ purposes: 1. Awareness support for students: a student who is especially interested in the changes and evolution of an object she created can use the following sequence of transformations to get the desired information: (a) Filter the protocol with the object as filtering parameter: filter(object=MyObject) (b) Abstract the low-level information to get information better understandable to the user: abstract After this sequence the student gets a list of actions related to the specific object on a level, that is comprehensible for the student; the following text paraphrases the students interpretation of the abstracted logfile: After the object’s creation by myself, it was repositioned several times by my peer learner, then I made some
changes to the content and finally it was targeted by the execution of the model by my peer. After some iterations of changing values and executing the model done mostly by me, she fixed the graphical position of the object for the submission to the teacher. 2. Evaluation Support for teachers: a teacher who is especially interested in specific phases of the problem solving process learners have been in during the session, can use the following sequence: (a) Rearrange the original time-stamped order of actions in the protocol, so that all actions of each user are in sequence: rearrange(Users) (b) Abstract the low-level information to get information better understandable to the teacher: abstract (c) Filter the protocol with the action classification simulation relevant as filtering parameter: filter(classification=simulationRelevant) (d) Augment the protocol by assigning the attribute phase of model exploration to the action sequences given by the previous step: augment(scope=sequence, attribute=modelExplorationPhase) With this sequence of transformations the teacher gets a protocol that is explicitly attributed with phases of model exploration, which may support comparison of different students problem solving behaviour (e.g. length and timing of exploration phases). This attribution is similar to the technique of coding or scripting protocols with attributes, which is frequently used by researchers in social sciences for interview analysis.
5
CONCLUSION
In this work-in-progress paper we presented a classification for transformations of learning protocols to process, extract and create information about actions and interactions in collaborative environments. We described shortly implementations of the different transformations types within the context of our collaborative modelling environment Cool Modes. Then we showed in two use-cases, how transformations can be used in combination for specific support to learners and teachers in understanding the collaborative processes. At the moment our implementations are restricted to specific purposes and domains to be used in our modelling environment. For a full-fledged empowerment of the users to get the information they are interested in, we plan to extend and generalize our implementations in such a way, that we can parameterize our transformations more flexibly, as shown in the two showcases. Providing an appropriate user interface, to relieve users from reading low-level protocols and having to understand technical aspects of our transformations, is also essential for our future work to enable learners, teachers, and researchers to use the full potential of protocol processing in collaborative learning (and working) environments.
ACKNOWLEDGEMENTS We would like to thank all students involved in implementation of the techniques described in this paper: Martin Roffmann, Michael Vetter, Jens Brauckmann, Stefan Th¨ur.
REFERENCES [1] Maria de los Angeles Constantino-Gonzalez and Daniel Suthers, ‘A coaches collaborative learning environment for entity-relationship modelling’, in Proc. of Intelligent Tutoring Systems ITS 2000, eds., Gilles
[2]
[3]
[4]
[5] [6]
[7]
[8]
Gautier, Claude Frasson, and Kurt VanLehn, pp. 325–333. Springer, (2000). Katrin Gaßner, Marc Jansen, Andreas Harrer, Kai Herrmann, and Ulrich Hoppe, ‘Analysis methods for collaborative models and activities’, in Designing for Change in Networked Learning Environments - Proc. of CSCL 2003, volume Volume 2, pp. 369–377. Kluwer Academic Publishers, (2003). Andreas Harrer and Lars Bollen, ‘Klassifizierung und Analyse von Aktionen in Modellierungswerkzeugen zur Lernerunterst¨utzung.’, in Modellierung als Schl¨usselkonzept in intelligenten Lehr-/Lernsystemen, Workshop Proceedings Modellierung 2004, Marburg, (2004). Marc Jansen, ‘Matchmaker - a framework to support collaborative java applications’, in Shaping the Future of Learning through Intelligent Technologies. Proceedings of the 11th Conference on Artificial Intelligence in Education, eds., Ulrich Hoppe, Felisa Verdejo, and Judy Kay, pp. pp. 529–530, Amsterdam, (2003). IOS Press. Martin M¨uhlenbrock, Action-based Collaboration Analysis for Group Learning, Ph.D. dissertation, Gerhard-Mercator-Universit¨at Duisburg, 2000. Niels Pinkwart, Heinz Ulrich Hoppe, Lars Bollen, and Eva Fuhlrott, ‘Group-Oriented Modelling Tools with Heterogeneous Semantics’, in Proceedings of Intelligent Tutoring Systems (ITS2002), ed., Stefano Cerri and Guy Gouarderes and Fabio Paraguacu, volume Lecture Notes in Computer Science (LNCS) 2363, pp. 21–30. Springer, (2002). Carolyn P. Rose, Andy Gaydos, Brian S. Hall, Antonio Roque, and Kurt VanLehn, ‘Overcoming the knowledge engineering bottleneck for understanding student language input’, in Shaping the Future of Learning through Intelligent Technologies, Proc. of AIED 2003, eds., Ulrich Hoppe, Felisa Verdejo, and Judy Kay, Frontiers in Artificial Intelligence and Applications, pp. 315–322. IOS Press, (2003). Amy Soller, Frank Linton, Brad Goodman, and Alan Lesgold, ‘Toward intelligent analysis and support of collaborative learning interaction’, in Proc. of the World Conference on Artificial Intelligence in Education AI– ED 99, eds., Susanne P. Lajoie and Martial Vivet, pp. 75–82, Amsterdam, (1999). IOS Press.
