Therefore, there is a need to develop methods and tools for the formative evaluation of cooperative e-learning, which support tutors and learners to monitor and ...
Continuous Evaluation of Web-based Cooperative Learning: the Conception and Development of an Evaluation Toolkit Shirley Holst & Torsten Holmer GMD – German National Research Center for Information Technology Integrated Publication and Information Systems Institute (IPSI) Dolivostr. 15, D-65293 Darmstadt, Germany {holst, holmer}@darmstadt.gmd.de
ABSTRACT Traditional evaluation approaches are not sufficient to tackle the evaluation of web-based cooperative learning. Evaluation in this area is difficult because many factors influence the cooperative learning process, which are always changing. This makes it necessary to make adjustments to the cooperative tools and learning methods to fit particular settings and emerging requirements. Therefore, there is a need to develop methods and tools for the formative evaluation of cooperative e-learning, which support tutors and learners to monitor and optimise their learning process at runtime. We propose the concept of continuous evaluation, which combines existing evaluation approaches in the construction of an evaluation toolkit consisting of guidelines, methods and software tools for the monitoring, analysis and optimisation of cooperative learning. The concept of continuous evaluation is interesting to those who wish to make systematic evaluations of CSCL systems. It describes which evaluation activities are appropriate when planning and designing CSCL, during early field studies, and throughout the ongoing maintenance of established courses. The aim of continuous evaluation is to build up a suite of evaluation methods and tools to be used by researchers, tutors and learners, which are tailored to a specific setting and are iteratively improved over time. Keywords Evaluation, CSCL, web-based learning, formative evaluation, data logging, participatory evaluation, distributed learning
INTRODUCTION The current explosion in use of new web-based technologies to support cooperative learning brings many new challenges for those evaluating the effects of these new tools and methods on the learning process. These effects are by no means easy to recognise and many factors influence the quality of learning. The newness of these technologies means that we are not in a position to prescribe exactly the ingredients, which make up the ideal cooperative learning process in advance. Although we can already make general guidelines about how to proceed with cooperative learning, there are so many variables which influence the process, that it is equally important to provide means by which the ongoing learning process can be assessed and adapted as it proceeds. We should aim to provide tutors and learners with sufficient information, so that they can take effective action to improve their learning activities themselves. In this paper we propose our continuous evaluation approach to evaluating webbased cooperative learning, which addresses exactly these needs. The aim of continuous evaluation is to build up a suite of evaluation methods and tools, to be used by researchers, tutors and learners, which are tailored to a specific setting and are iteratively improved over time. The evaluation toolkit supports tutors and learners in monitoring the events and patterns of interaction that are occuring during learning cooperations. A range of methods and tools, including data logging, online questionnaires, together with tools for their analysis, provide a representation of the ongoing learning process such that the participants can assess its satisfactoriness. In this way, participants are supported to reflect on their behaviour and are in a position to intervene in order to optimise it. Guidelines explaining how to arrange and manage cooperative learning, and how to use the monitoring tools are also an important part of the continuous evaluation toolkit. In the remainder of the paper, we firstly discuss selected aspects of the state of the art in CSCL evaluation and criticisms that have been made of it. Based on our own experience and well-established literature, we then propose our continuous evaluation approach as a solution to these problems. We illustrate the continuous evaluation approach in detail, with a description of how this approach is applied to the L_ learning platform. In particular, we
describe the evaluation of so-called Points of Cooperation (PoCs). current project work and our future plans.
We end by summarising the status of our
STATE OF THE ART Our perspective on evaluation extends a number of well-established approaches. Firstly, that evaluation should not be purely summative, but contribute to a continual improvement in the learning process has been said by many (Ramage,1999; Cullen et al., 1993; Issing et al., 1997). We go a step further and implement evaluation tools and methods, which serve to monitor and optimise the learning process at runtime. Secondly, Cullen et al. describe the evaluation of learning technology innovations as being a process of learning for particular 'stakeholders'. Stakeholders are individuals in particular roles (e.g. tutors, learners, educational providers and researchers) who care about the outcome of the evaluation study. In our case, we focus particularly on tutors and learners, who are the closest to the learning process and the most involved in it. Visualisations of learning events and their own behaviours allow them to take a step back and to reflect on the effects of their actions and strategies. Thirdly, we subscribe very much to participatory evaluation approaches (Bodker, 1996) in which the stakeholders are involved at every stage of evaluation. We place the tutors and learners in a central role, both in the development stages of the quality suite, seeing the evaluation toolkit as a series of tools designed to support them in their existing roles. Finally, we extend data logging methods, which have effectively been used to analyse online behaviour in groups (Holmer and Streitz, 1999). Data logging has so far mainly used to reconstruct and analyse the learning process afterwards, and we extend its use to support the ongoing monitoring and optimization of learning at runtime. A variety of criticisms have been levelled at previous evaluation concepts. We outline three of these here, which we want to address in particular. Firstly, evaluation methods that rely mainly on an assessment of learning outcomes have been criticised (Baumgartner, 1996). Learning gains are inherently difficult to assess, and in particular it is difficult to statistically prove an increase in knowledge due to particular interventions (e.g. one sort of learning method versus another). This is especially true for cooperative learning, where many more factors come into play, and where we know that knowledge is distributed among the members of the cooperating group (Salomon, 1994). The knowledge of individuals is typically displayed during interactions with other learners (e.g. in conflict situations or when explaining to other learners, rather than in pre- and post-tests of individual understanding). Therefore, we find it appropriate to focus on identifying events, behaviours and interaction patterns that demonstrate effective cooperative learning, rather than solely relying on the results of knowledge tests. A second key criticism, which has been made of previous evaluation approaches, is their lack of foundation in theory. Baumgartner (1996) stressed the need to build up the theoretical foundations of cooperative web-based learning. Without a descriptive model of the cooperative learning process, we cannot begin to find indicators that will tell us whether effective learning is taking place. We should consider how to create models of learning that explain and predict the behaviour of learners and learning groups with a particular technology. We can draw on established theories of cooperative learning in order to describe the learning processes taking place. Those theories that thave been most widely applied to cooperative learning systems (e.g. constructivism, distributed cognition and socio-cultural learning theories) are extremely general and typically under-specified in their application to virtual cooperative learning scenarios. General concepts, such as ‘social conflict’ or the ‘mediating’ role of tools are frequently used to motivate the design of cooperative learning tools and environments. However, often little effort is made to identify and analyse examples of ‘social conflict’ and ‘mediation’ taking place when learners actually use the tools. Such general concepts should be operationalised in terms of specific behaviours, which researchers and educators can observe as learners complete a specific cooperative task for a particular learning domain. Designers and educators should agree on a model of what the ideal learning process for these cooperative tasks would entail, defining indicators of successful versus unsuccessful learning at the level of observable behaviour and events (Holst, 2000). For particular design features they can then predict how learning behaviour is altered. Of course it is not always possible to justify in such detail the use of new tools in terms of established theory. In this case (and as we illustrate with our examples below) the didactical concept behind new tools should be well described in terms of the predicted effects that will be produced both within cooperative learning activities, as well as on the course as a whole. This leads to the creation of mini or 'local' theories of the learning process, which can then be tested. Finally, we should try to predict the influence of moderating variables, which are seen by stakeholders to be particularly relevant in their educational context. The moderating variables could be, for example, type of content (e.g. highly technical versus soft skills training), learning styles or learner preferences regarding cooperation.
The third criticism addresses the fact that existing criterion catologues for evaluating computer-based learning, of which to our knowledge none directly address cooperative e-learning, are based on plausibility arguments given by experts, which are then not empirically validated (Fricke, 2000). In order to establish the validity and reliabilty of quality criteria and their respective measures, we can ultimately only rely on empirical proof of their effect on learning strategies and (but not solely) on learning outcomes. We now describe our approach to evaluation in more detail, addressing each of the above criticisms, and ending with a summary of the resulting three components of our continuous evaluation toolkit (which we call the Quality Suite).
THE CONTINUOUS EVALUATION APPROACH We distinguish three evaluation activities that contribute to building the evaluation toolkit. The process of continuous evaluation starts before the application of the learning system in a particular setting. It begins with the designing and testing of communication and cooperation tools independently of any specific course. At this stage of evaluation, it is appropriate to carry out usability studies and experimental studies, which look at whether the interface and the application design are appropriate. Such studies also assess the predicted effects of particular features of the tools and cooperative learning methods on the learning process, learner interactions as well as learning outcomes. The second phase of the continuous evaluation process focusses on the application of the cooperative learning system in the context of a particular course. Here we are interested in the effect which cooperative learning episodes have on the progress of individual learners through an existing course. This type of evaluation should start while tools are being conceptualised and developed, and in collaboration with educational experts, tutors and learners. Ideas for the design of tools that fit to the overall learning process and philosophy are thereby captured. At runtime, quasi-experimental designs (field studies with systematically manipulated variables, but without full random allocation of participants) are also appropriate to compare the effects of introducing cooperative learning elements at various stages during the learning process. In the context of a particular course, we can establish empirically if one learning method or tool is more appropriate for specific learning goals than another. Thirdly, by means of correlational studies we can look at the interaction between cooperative elements and moderating variables. We can gather information via questionnaires about the learning style, abilities and preferences of the learners, and use this information to find out if such factors influence how they deal with cooperative tools. For example, do learners with particular learning styles prefer one form of cooperation rather than another? From each of these three types of evaluation activity, we gather experiences and knowledge, which can then be used within the evaluation toolkit. Our concept for this toolkit consists at the moment of the following three elements: Guidelines, Monitoring Tools and the Questionnaire Generator. •
•
•
The guidelines for how to arrange effective cooperative learning draw on knowledge and best practise examples, which emerged from experimental, usability and field studies. These guidelines are used by tutors and learners in order to plan or improve their learning activities, to give new ideas for ways in which the tools can be used, and to provide a solid (and at least in part empirically founded) basis on which to know which interventions produce desired effects. These guidelines may be available as a handbook, or a more flexible online version, which could be continually enhanced by the learning community over time. The monitoring tools gather information about the learning process via both unobtrusive data logging and via brief online questionnaires that learners and tutors fill out at particular stages of the learning process. The questionnaires can be made to appear automatically after certain events (e.g. immediately after performing a cooperative activity). The data is then automatically analysed and made available in summarised form to the tutors and learners, giving them a useful overview of how cooperative learning is taking place across the whole course. The monitoring tools also present selected screen-shots and results of student discussions. It may be appropriate to offer tools, which support tutors quickly and easily to code and summarise this qualitative data. The questionnaire generator helps those evaluating a course to apply known quality criteria in order to generate questions about the learning process, which can be answered by the learners, giving feedback to the tutor about the course. While the monitoring tools are designed to provide a continually available stream of information about the level and type of cooperation going on in the course, the questionnaire generator is meant for occasional use, when the tutor wants to get more in depth feedback from course members about their experience
and satisfaction with cooperation during the course. The evaluator selects those quality critieria that are particularly relevant at the time, and the software selects appropriate, pre-defined questions from a database of quality criteria and an associated pool of questions. Such criterion catalogues are currently available for computer-based learning (see Schenkel et al., 2000), but they do not provide a specific focus on cooperative elearning opposed to computer-based learning in general. As yet, there are no universally recognised quality standards for cooperative e-learning. The questionnaires can also form a checklist of critical incidents, which may have occurred. The learners should indicate which incidents have occurred (e.g. problems in finding cooperation partners, or difficulties in carrying out particular cooperative tasks). This gives the tutor an overview of how learners are experiencing cooperation and goes beyond the monitoring tools, which simply gather non-reactive observations of events and behaviours. The questionnaires are used to gather an understanding of the learners' perspectives and perceptions. The three components of the quality suite are tightly interwoven and complement each other. The guidelines accompany the monitoring software and the questionnaire generator, explaining their purpose and how they should be used. Since the development of the toolkit is iterative, the monitoring tools can generate data about the ongoing cooperative process in a particular setting, which can later be integrated into the guidelines. Results from the questionnaires could also be used to enhance the guidelines and to give new ideas for interpreting the significance of particular events and behaviours that are captured by the monitoring tool. Notably, the full empirical validation of such guidelines and tools is a huge undertaking. We address this in two ways. Firstly, through a reliance on participatory evaluation and a participatory approach to building up appropriate measures and representation of the learning process. Secondly, we choose selected parts of the quality suite for validation through empirically stringent methods (experiments, quasi-experiments). As yet, we know of no reliable external scales against which we can validate our own measures. At this stage, we therefore compare the results of our own measures against the ratings of expert tutors, educational specialists and learners. We go a step further towards addressing Fricke's criticisms, in that we make explicit what we are trying to measure, and do not simply rely on the unexplained and often implicit expert opinions. We place emphasis on building models of the learning process, which can then be empirically tested.
APPLYING CONTINUOUS EVALUATION TO THE L_ PLATFORM L_ stands for life long learning and is a learning platform, the development of which is being funded by the German Ministry of Education and Research (BMBF). The L_ project aims at developing an integrated infrastructure for continued education, training and retraining, based on internet technology. The L_ system supports cooperation among learners and tutors in various ways. The cooperation functionality of L_ combines both more traditional and new concepts for online cooperative learning. The so-called PoC concept (Points of Cooperation) has been developed in L_ to distinguish and support different modes of cooperation, which differ in the extent to which they are bound to specific courses, and course elements (Wessner and Pfister, 2000). At the most general level, GPoCs (Generic Points of Cooperation) provide functionalities for making contact with other learners in the learning environment irrespective of what course or content they are studying. For example, mail and news facilities which allow learners to get in touch with each other about any topic of their choice. SPoCs (Spontaneous Points of Cooperation) are related to a specific course and support learners to contact their own tutor or peer learners within the course. For example, by providing options such as 'send message to tutor' or 'ask a peer learner'. Chat rooms and newsgroups may be assigned to a particular course, for use by course members only. This allows learners to spontaneously ask for help or discuss issues arising from their ongoing studies. IPoCs (Intentional Points of Cooperation) go a further step towards binding cooperative episodes directly to specific course elements. They are incorporated into the course at a specific place, as specified by the course author. The learner comes to a point in the course material at which they are required to register for a cooperative activity. These activities range from group discussions with a chat tool, to brainstorming in groups with a shared whiteboard and pro-contra disputes in which the learners take particular stand points on a topic and use audio-video conferencing or chat tools to exchange their views. We now apply three evaluation activities described above to the evaluation of the IPoC concept. IPoCs are positioned into the learning process at strategic points and therefore we can clearly describe and distinguish their predicted effects on the learning process. As described above, we approach the evaluation of these effects through three evaluation activity types:
(i) (ii)
(iii)
Internal IPoC Evaluation assesses the usability and utility of the cooperative activity taking place independently of a specific course setting; Evaluation of Course Effects assesses the effect that IPoCs have on the ongoing learning process. In particular, we want to establish what effects the cooperative learning activites have on individual learning activities. We monitor changes in learner behaviour immediately before cooperation (e.g. preparations that are made before taking part in an IPoC) and also afterwards (e.g. the reviewing activities of the learners after the IPoC is completed); the Influence of Moderating Variables investigates how other factors, such as learning style or type of content, interacts with the acceptance, appropriateness and effectiveness of particular types of cooperative learning activity.
Internal IPoC Evaluation The cooperative learning methods supported by each IPoC have been developed according existing didactical concepts, methods and theory (Flechsig, 1996). In order to support online and distributed learners to carry out these methods, we have built a range of tools. They display background information to the task (provided by the course author) and guide the learners in their progress through the stages of cooperation. In some cases they also specify the pattern of social interaction that takes place. For example, in the Pro-Contra Dispute, learners are automatically assigned to roles and are provided with background material and discussion strategies, which help them to prepare for beginning the discussion. During the discussion itself, the tool regulates turn-taking between pro and contra representatives. A strict turn-taking algorithm is applied, in which pro and contra must alternate. We call these algorithms learning protocols (Miao et al., 2000). The effect of carrying out the Pro-Contra Dispute should be that learners learn to represent a specific point of view and to defend this stand point against other arguments from colearners. They learn how to formulate consistent arguments based on available information, irrespective of their personal views. In internal IPoC evaluation studies, we firstly examine the usability of the tools and the effect of the cooperative activities on the learners. These studies are conducted at various levels. Firstly, and more informally we carry out initial trials to run through the cooperation activity. We identify obvious usability problems and gather feedback for adaptations to both the interface, as well as to the background material and the learning protocols which guide cooperation. It may be necessary, for example, to provide alternative protocols which guide the learning process to a more or less strict extent. We also show sketches of the interface, and working prototypes to tutors for their comments. Secondly (and due to the large effort required we only do this in selected cases), we carry out controlled experiments to investigate the influence of parameters such as group size, or turn-taking strategy on learning. In a related project, a series of experimental studies are being carried out on another type of IPoC, the ExplanationDiscourse IPoC (Pfister and Mühlpfordt, submitted). In this cooperative activity, the learners are presented with a text or other initial material. They may then ask the tutor questions about this material using a chat tool. A learning protocol guides the discourse, in that students are forced to contribute in a round robin format. The participants must indicate to which part of the text their contribution refers. They also label their contributions as either a comment, a question or an explanation. Every time a question (rather than a comment or explanation) is made, the tutor must answer. Once the answer is given, the protocol continues the round robin where it left off. This is based on the theory that learners will retain more knowledge when they make use of protocols than when they are simply left to chat. Deriving from script theory (Schank and Abelson, 1977), the prediction was made that learners would be supported in recalling the learning process because they could remember the conversation that they had in terms of a script. In this case the script is very simple, but nevertheless, the labelled contributions, which occur in a regular order, should give learners an anchor on which to hang their memory of the learning process. In addition the protocol reduces the need for learners to coordinate their learning process themselves. Initial studies to evaluate this and other protocols have been carried out. The results already indicate that there are positive effects of the protocols on learning. Although students made many more contributions in the free-chat sessions, there was still a greater learning effect (in terms of pre- and post- tests of knowledge) for those learners who used the learning protocol. Future studies will be used to find out where exactly these effects lie.
Evaluation of Course Effects The evaluation of course effects goes beyond the effects that we expect to occur during the cooperation itself. In the evaluation of course effects we analyse the influence that cooperative episodes (IPoCs) have on the learning
behaviour that occurs before and afterwards. Not all of the IPoCs are carried out immediately. The students are registered for an IPoC, which is then carried out at a time convenient to all participants. Depending on the type of PoC, it is expected that the students prepare themselves to take part in the cooperative session. Therefore we expect to observe that students are carrying out extra research on their learning material that is related to the PoC itself. For example, in order to prepare for a Pro-Contra Dispute, the learners should revise the material, which supports their position and gives them information about the opposing point of view. We expect learners who do more preparations of this form to perform better (e.g. more and higher quality contributions to the discussion) during the IPoC itself.
preparation
individual learning
postprocessing
cooperative learning
cooperative learning individual learning
individual learning
time
Figure 1: Pre and Post Effects of Cooperation The second observable effect of the IPoC on the ongoing course is a change in learner behaviour after it is carried out. We expect learners to discover gaps in their knowledge, which they then try to fill by re-examining previous modules after the IPoC has taken place. The IPoCs should also lead to subsequent more informal discussions among participants. The extent to which students use the PoC tools to invent their own cooperative exercises can be seen as a measure of their perceived effectiveness and usability. Figure 1 illustrates these pre and post effects of cooperation. These effects are particularly important, when we consider that learners may have to wait until there are enough co-learners with whom to carry out particular IPoCs. This could mean that a topic that was studied early in the course is then cooperatively discussed weeks later. It is important to understand the negative, but also positive effects of such delays. Beyond the effects of IPoCs on the individual's learning activities, we also monitor the occurence of spontaneous cooperations (SPoCs) across time. We want to observe whether they increase or change over time, at what points in the course they occur, and whether they are influenced by the occurence of IPoCs. In terms of evaluation, we can see the introduction of IPoCs in the course as interventions. To examine all of these effects, various experimental and quasi-experimental designs are possible. For example, we can run courses in which only SPoCs are available, in contrast to those in which IPoCs are introduced. A third control group can be compared, in which no cooperation takes place. We can also look at the effects of using alternative PoCs or versions of PoCs, or experiment with introducing cooperation at different phases of the course. In this way, we find out what is the optimal combination of course content plus cooperation. Figure 2 illustrates the comparison of an individual learning condition with a cooperative learning condition for which PoCs are introduced at intervals throughout the course.
individual learning condition individual learning
cooperative learning condition
individual learning
cooperative learning
cooperative learning individual learning
individual learning
time
Figure 2: Experimental Comparisons to test the Effects of Cooperation
Influence of Moderating Variables As well as considering factors to do with the design of the learning environment and learning tasks, we have to take account of moderating variables, which in field studies we cannot always control, but which are thought to have a significant effect on the learner's experience and to moderate the effects of cooperation. Some of the moderating variables are learner specific (e.g. learning style, social and cognitive characteristics, or cooperation style), others depend on the content of the material or the context in which learners are taking part. For example, do learners take part on a voluntary basis, within work time or at home, are they working towards a formal qualification or doing on-the-job training? Depending on whether the learners are completing their courses during work time, and on the learning culture of their employing institution, we would expect to see an effect on the amount of time available for taking part in more time consuming cooperative exercises. The duration of the course and the number of learners taking part also affects the amount and nature of cooperation that is possible. Learners need to get used to new cooperative learning methods as well as to the new tools. They also have to get to know each other and develop a style of cooperative learning within their groups. We need to investigate how effectively the IPoCs are carried out among groups of learners who previously did not know each other. How important is it to establish working groups, which remain stable throughout the course? What composition of group works the best for which types of task (e.g. homogeneous, heterogeneous)? Heterogeneous groups may be more desirable for creative tasks or where the learners should learn from one another. Homogeneous groups, who start with more equal knowledge levels, may be more appropriate for the rehearsal and practising of routines or for learning new knowledge for which a common knowledge base for all group members is required. We gather information about learner characteristics through questionnaires at the beginning of the course and monitor the learning and cooperation behaviours, such as those mentioned above, by means of data-logging as well as self-reports from learners. Using correlational and cross-tabulation methods we look for relationships between the hypothesised moderating variables and our measures of the successfulness, preference and acceptance of IPoCs. We could for example expect that learners with higher social ability would express more enjoy of IPoC tasks, prepare more thoroughly beforehand and show higher levels of review activities after the cooperation episode is completed.
Implications for the Quality Suite The three IPoC evaluation activities (internal evaluation, evaluation of course effects, and assessment of the influence of moderating variables) culminate to provide input for the evaluation toolkit. Results from any of these L_ evaluation activities can be incorporated into guidelines and checklists for tutors and learners as to how best to use IPoCs in their learning process. Internal evaluations of the IPoCs in their early stages contribute directly to the design of the IPoC tool itself, but also contribute to explaining why and in what ways the IPoC should be performed. The evaluation of course effects gives advice as to where IPoCs should be positioned in the course and how they are expected to influence learner behaviour before and after cooperative activities. Information about the influence of moderating variables provides advice about optimal learning group combinations, appropriate cooperation tasks depending on the length of the course, or the nature of the content or previous knowledge and experience of the learners. The written guidelines also contain information about how to use the monitoring tools, how to interpret the results, and recommendations for interventions that can be made to improve cooperation processes in certain defined situations. The data logging tools are used during internal evaluation activities to analyse the interaction patterns among learners within the IPoC. They are also used during course evaluations to gather data about when cooperation of the various types occurs, and to trace the projected effects on learner activity before and after IPoCs are carried out. For example, we use data-logging to examine navigation behaviour in the learning material, which we expect to increase immediately before and after an IPoC such as the Pro-Contra Dispute is carried out. The monitoring tools are ultimately incorporated into the Quality Suite. We use the results of our field studies and experiments to establish which aspects of cooperation need to be monitored by the tutors on a continual basis. We also use the studies to establish data analysis methods, initially performed by hand (with statistical software packages external to the evaluation toolkit), but which are later integrated into the quality suite as an automatic analysis component of the monitoring tools. We need to select carefully that information which is presented, since we do not want to overload tutors and learners with too much information. We do, however, want to provide them with indications of points at which they should intervene. The tutors should be supported to browse through a range of options for visualising different aspects of the cooperative process. For example, at the most general level, the tutor can be made aware of the quantity of cooperative exchanges occurring, such as number of emails, number of newsgroup entries, number of IPoCs performed per day. An activity timeline for individual learners can also be examined, showing individual and cooperative learning activities, such as material navigated, as well as pending and completed IPoCs. At the finest level of detail, logs of discussion and screenshots of the results of cooperative sessions could be displayed. Clearly, the learners have to agree that the tutor has access to such data. Self-organising learning groups may also choose to monitor their own cooperative process, thus encouraging them to reflect on their own learning strategies.
CONCLUSIONS AND FUTURE WORK We have described an approach to the continuous evaluation of cooperative e-learning which extends existing literature and addresses some of the problems, which have been identified previously. We have extended the formative and participatory evaluation concepts by building evaluation methods and tools, which are designed to be used at runtime. We recommend a mixture of evaluation approaches in order to create and validate the evaluation toolkit. We go beyond existing criterion catalogues towards a more dynamic way of evaluating, which focuses on monitoring the behaviours and interactions among learners, rather than on the simple measurement of learning outcomes. We stress the need to base the development of the evaluation strategy on a detailed model of the learning process, which has to be continually updated, according to the current requirements of the learning setting. We have developed our concept of continuous evaluation based on the ongoing work in three related research projects. Firstly, as we described above, the L_ (Life Long Learning as a basic need) project, in which our research group (GMD-IPSI) has the responsibility for conceptualising, building and evaluating cooperative services for the L_ learning platform. In the more recently started ALBA project, we have begun to develop the concept of the evaluation toolkit and have already carried out some case studies (Holst and Fleschutz, 2001) with a view to extending existing criterion catalogues for the assessment of cooperative e-learning. In this project, we are cooperating closely with our partners in both corporate and public education settings to gather requirements for the Quality Suite. A third related project, Learning Protocols, is examining the effects of learning protocols in online cooperative learning tasks. These three projects are running in parallel. The youngest of these three projects (ALBA) will integrate the results of all three projects. In a series of field studies, the Quality Suite will be conceptualised, tested, and used.
ACKNOWLEDGEMENTS We would like to thank our colleagues for their discussions and for reading of drafts of this paper. Particular thanks go to members of the projects, on whose work this paper is based. The L_ and ALBA projects are each funded by the German Ministry of Education and Research (BMBF). The Learning Protocols project is supported by the German Research Council (DFG).
REFERENCES Baumgartner, P. (1996) Evaluation vernetzten Lernens. http://www.avmz.uni-siegen.de/extern/GMW/events/VC96/prg/abs/2_7.html Bodker, S. (1996) Applying Activity theory to video analysis: how to make sense of video data in HCI. In B.A. Nardi (Ed.), Context and consciousness: Activity theory and human-computer interaction (pp. 147-174). Cambridge, Massachusetts: MIT Press. Cullen, J., Kelleher, J. and Stern, E. (1993) Evaluation in DELTA. Journal of Computer Assisted Learning. 9: 115-126. Flechsig, K.-H (1996) Kleines Handbuch didaktischer Modelle, Neuland. Fricke, R. (2000) Qualitätsbeurteilung durch Kriterienkataloge. Auf der Suche nach validen Vorhersagemodellen. In Schenkel et al. (2000), p. 75-88. Holmer, T., and Streitz, N.A. (1999) Neue Möglichkeiten der Analyse der Mensch-Computer-Interaktion zur Evaluation von computerunterstützten Gruppensitzungen. In: U. Arend, E. Eberleh, K. Pitschke (Ed.): Software-Ergonomie '99. Design von Informationswelten, pp. 137-148, Stuttgart, Teubner, 1999 Holst, S.J. (2000) Evaluation of Collaborative Virtual Learning Environments: The State of the Art. In F. Scheuermann (ed.): Campus 2000: Lernen in neuen Organisationsformen. Proceedings of GMW 2000. Fachtagung der Gesellschaft für Medien in der Wissenschaft. Innsbruck, 19.-21. September, 2000, pp. 199-212. Medien in der Wissenschaft, Bd. 10, Münster, New York, München, Berlin: Waxman. Holst, S.J. & Fleschutz J.M. (2001) Learning Foreign Languages Cooperatively Across the Internet. To apprear in: Proceedings of the ninth biennial conference on Computer Assisted Language Learning CALL 2001 - the Challenge of Change. September 1-3, 2001. Exeter. UK. Issing, L.J. (1997) Ausbildung für Medienberufe – Perspektiven für den Multimedia-Bereich. In (Ed. Ludwig J. Issing and Paul Klimsa) Information und Lernen mit Multimedia. Beltz Psychologie Verlags Union pp. 451-462. Miao, Y., Holst, S.J., Haake, J.M. & Steinmetz, R. (2000) PBL-protocols : Guiding and Controlling Problem Based Learning Processes in Virtual Learning Environments. In B.J. Fishman, S.F. O'Connor-Divelbis (eds.): Proceedings of International Conference of the Learning Sciences (ICLS 2000). June, 14-17, 2000, Ann Arbor, USA, pp. 232-237. Mahwah, NJ, USA: Lawrence Erlbaum Associates. Pfister, H.-R. & Mühlpfordt, M. (submitted to CSCL 2002). Supporting synchronous learning discourses: The learning protocol approach. Ramage, M. (1999) The Learning Way. Evaluating Cooperative Systems. Doctoral Thesis. Lancaster University. http://www.dur.ac.uk/~dcs1mr/thesis/ Salomon, G., (1994) (Ed.) Distributed Cognition. Cambridge, UK: Cambridge University Press. Schank, R. C., and Abelson, R. P.(1977) Scripts, plans, goals, and understanding: An inquiry into human knowledge structures. Hillsdale, NJ: Lawrence Erlbaum Associates. 248 pages. Schenkel, P., Tergan, S.-O. & Lottmann, A. (2000 (Eds.) Qualitätsbeurteilung multimedialer Lern- und Informationssysteme. Evaluationsmethoden auf dem Prüfstand. Nürnberg: BW Bildung und Wissen. Verlag und Software GmbH. Wessner, M., and Pfister, H.R. (2000) Points of Cooperation: Integrating Cooperative Learning into web-based courses. In Proceedings of the International Workshop on New Technologies for Collaborative Learning. NTCL 2000, Awaji-Yumebutai, Hyogo, Japan, Nov. 27-29, 2000, p. 33-41.
