Combining User-Centered design and Activity concepts for developing

1

Combining User-Centered design and Activity concepts for developing computer-mediated collaborative learning environments: a Case Example M. Felisa Verdejo Beatriz Barros Departamento de Ingenieria Electrica, Electronica y Control, Escuela Técnica Superior de Ingenieros Industriales (U.N.E.D) Ciudad Universitaria s/n, 28040 Madrid, Spain {felisa, bbarros}@ieec.uned.es phone: 34 - 1 - 398 64 84; fax: 34- 1 - 398 60 28

Abstract: Formative evaluation and iterative prototyping seem an interesting approach for building collaborative learning systems. Our approach to create our software system consisted on iterating successive stages of analysis, specification, implementation and evaluation. At the end, the process turned out to be a five-pass cycle, carried out in two years, with a major reimplementation between the third and the fourth prototype version. We describe the main features for each version, as well as the evaluation techniques followed. The first cycle is presented in more detail and relevant aspects are summarized for the rest. Each evaluation includes the design of a learning activity, the configuration of the prototype to carry out the experience, the set-up and observation of the activity performed with real students and tutors, and finally the drawing of conclusions including the improvements to consider for a next design phase.

Introduction For a number of years we have been involved on research and experimentation with collaborative learning at a distance in an effort to identify and deploy appropriate uses of the new media to support a distance learning community. In this paper we focus on the process of designing a software environment to support collaborative learning activities in an integrated way. The design of collaborative software has proved to be hard. Theories and metaphors in the CSCW field have emerged but are still a matter of big debate and further elaboration is needed to produce models supporting the practical design of CSCW applications. The difficulty comes from the complexity of the situations to be considered, where the technical perspective is one part of the picture. In a specific work situation, technology should support human activity taking into account the social setting: an organization with rules, structure and working practices. Formative evaluation and iterative prototyping seem to be an interesting approach for building collaborative learning systems. Conducting experiments to find out how the prototype is used and what is learned from its use provide insights, not only into how to improve the system but also into how to motivate further extensions. A conceptual framework is still needed to analyze and capture how the technical solutions in each version of your prototype will support the group of learners to attain their goals. The socio-cultural framework provides the concept of Activity (Nardi, 1996) as a unit of analysis, with a rich internal structure to make the context of a situation explicit, specially the interlinks between the individual and social levels stressing the role of the tools as mediating artifacts. One of our systems serves here as a case study to illustrate the potential of combining user-centered approach (Gould & Lewis, 1985) with Activity Theory to design collaborative learning software. The next section describes our context, and elaborates on the kind of learning tasks and collaborative framework we have considered. In section three the main features of the iterative design are presented and learners’ experiences discussed. For a detailed description of the system itself the reader can consult (Barros & Verdejo, 1998).

Context, goal, and premises for defining the learning approach Collaborative learning research has paid close attention to studying pupils interactions during peer-based work in order to analyze and identify the cognitive advantages of joint activity. As Crook (1994) points out the benefit of the collaborative approach for learning lies on the processes of articulation, conflict and co-construction of ideas occurring when working closely with a peer. Participants in a problem-solving situation have to make their ideas explicit (assertions, hypothesis, denials..) to other collaborators. Disagreements prompt justifications and negotiations, helping students to converge to a common object of shared understanding.

2 The computer provides opportunities to support and enhance this approach in a number of ways (e.g. offering computer-based problem spaces for jointly create and exploit structures of common knowledge and shared reference). Furthermore networks made possible the opening of collaborative frameworks to distributed communities, providing remote access to these spaces as well as computer-mediated communication to support interpersonal exchange and debate. An increasing number of collaborative learning environments for open and closed virtual groups have been built for a range of learning tasks (Scardamalia & Bereiter, 1991) (Edelson & O'Neill, 1994) (Wang & Johnson, 1994) (Suthers & Jones,1997), and experiences of use are reported from school to university level (Bell, Davis & Linn, 1995) (Collis, 1997). The potential of collaborative learning, within the framework of a distance learning institution, has to be explored taking into account the traditional practice of individual and stand-alone study in distance education. Not only faculty but also students are reluctant to carry out collaborative practice. Network facilities are widely appreciated as a way to improve the efficiency of how course material is delivered rather than an opportunity to change patterns of teaching and learning. So the challenge for developing collaborative activities in a traditional distance learning university is not only to build appropriate tools but also to transform established practices in the community. For these reasons, it is quite critical to look for opportunities of implementing collaborative learning experiences where collaboration could be perceived by students as a clear added-value to existing learning practices. So we want to propose that our distance learners, geographically distributed, work together in small groups to prepare an overview of a topic related to their professional profile. There are a number of ways in which this collaborative activity can be organized, one of the procedures that we have implemented consists of a phase of joint involvement in the preparation of the work within each group, followed by a period of individual reading. Then participants engage in a dialogue to elaborate collaboratively a synthesis according to a schema. Variants can include a phase of planning and the negotiation of the schema before to the collaborative writing. A further step could be mutual presentation of the final products between groups. From the learning point of view the task fulfills the requirements of been relevant (it corresponds to a real practice in a research context) and co- constructive in the sense that they have to build their own shared understanding of the topic. Furthermore the discursive nature of the procedure and the deferred mode for the debate afford opportunities for the students to reflect on the way they perform the collaboration process.

The iterative design process Our approach to create the software system consisted of iterating successive stages of analysis, specification, implementation and evaluation. At the end, the process turned out to be a five-pass cycle, carried out over two years, with a major reimplementation between the third and the fourth prototype version. Next we will describe the main features for each version, as well as the evaluation techniques followed. The first cycle is presented in more detail and relevant aspects are summarized for the rest. Each evaluation includes the design of a learning activity, the configuration of the prototype to carry out the experience, the set-up and observation of the activity performed with real students and tutors, and finally the drawing of conclusions including the improvements to consider for a next design phase. The first cycle. The prototype The first prototype, installed on a web server, offered the same services to all clients. The access was remote and the mode asynchronous. Each user had a login and a password and belonged to a group. The system supported a number of small groups operating at the same period of time. The main metaphor in the prototype for sustaining a learning activity is the concept of space, a virtual structured place with resources and tools to perform a task. Three types of spaces were available: an individual workspace, private for each user, and two shared spaces for each group (one workspace for debate and joint construction, the other for coordination purposes). The information handled is mainly textual, so a variety of editing tools and file management facilities was available, as well as links to other relevant electronic sources of further information for the task at hand. The shared workspace provided support for conversation in the form of semi- structured typed messages. When learners express their contributions they have to select a type from a predefined set. These types include: proposal, contraproposal, comment, question, and clarification. The system dynamically builds a representation of the discussion process going on in each workspace, in the form of an index of related contributions for each subtask. This index was fully deployed and represented on the left part of the workspace screen. The evaluation phase The first evaluation addressed the central point of the adequacy of the shared workspace to support a group of learners to perform a discursive task. The design of the learning activities was carried out by our team, as well as

3 researchers we are also teachers. Two kind of learning situations were considered. We describe one of them, characterized in terms of Activity Theory (Kuuti, 1996) (Cole & Engestörm, 1993) as follows: The object of the learning activity was to synthesize the state of the art in a focused research topic. Two topics were selected: one in Artificial Intelligence (AI), the other on Educational Technology. The outcome of the activity was an essay, pointing out the key ideas. The schema for the essay was previously defined by the tutor, and included in the shared workspace. The community involved in the activity were pairs of graduate students in a Ph.D. distance teaching programme, in UNED, our university. Students were part time, geographically distributed, usually interacting in disjoint time slots. The subject, each individual student, had either a technical or non technical profile, but groups were formed by people with the same background. The topic for the groups with technical profile was in AI Mediational tools included: (i) humans, a tutor and a technical assistant both from our team (ii) artifacts: the phone, a set of documents and (iii) the prototype: a networked computer environment including the script of the activity, private and shared structured task spaces for peer argumentative discussion, coedition and coordination, as well as access to other electronically available sources of information. Rules for the activity were stated explicitly and have to be accepted before starting. They include the commitment to finish the work, the script for the activity and the protocol for the collaborative debate. Some of these rules were embedded in the system, for example, the conversational graph defined the way one can contribute after another peer contribution. Others were the full responsibility of the learners, such as the way to organize the discussion, the way to agree, as well as aspects of time management and deadlines. The only direction given was to use the coordination space to deal with all these matters, using the coedition workspace only for the topic discussion. Students in each group had the same responsibility. The task is divided in subtasks but there were no predefined roles for division of labour. Performing the activity Once the learning activity was designed and the prototype ready, the experience went to a set-up phase to organize the formation of groups, fix the training strategy for the use of the tool, prepare the support strategy and establish the evaluation framework. In this case students were recruited on a call for volunteers. The groups were made up by the tutor. A handout for the use of the system was distributed to all participants, and a session of practice on their own was recommended before starting the activity to became familiar with the tool. Moreover an interactive training session, using the phone and the system, was offered and carried out with one of the students. Technical support for the whole period of the experience was provided, a person could be reached either electronically or by phone to deal with questions or problems related to technological aspects. For content or pedagogical related matters, a teacher was also available. Along the running stage of the experience her role was to act only on request, however for evaluation purposes she was observing actively the use of the system. Events and data were all recorded. This information together with teacher and assistant notes, as well as individual interviews for dropping-out cases, were the main sources for the assessment of the prototype. Results Three groups started the activity and only one arrived at a completion state. All the groups performed the preparatory phase, but one of the groups dropped-out early, before engaging in the collaborative step, another student dropped-out in the middle and we had to redefine the learning task for the remaining student to allow him to continue alone. Finally the third group worked well and accomplished the whole activity, having a balanced participation. This 50% ratio of drop-out is comparable to a standard course. In the first group both students changed their job and were not interested anymore in the course. Some students needed help and technical assistance for the communication infrastructure, being the first time they had connected and used Internet. The system was considered easy to use, and students in the third group stated the experience as successful, appreciating the benefits of collaboration, despite the additional constraints this form of work demanded on them. Added-value was perceived in three dimensions: more depth in content related matters, becoming familiar with new technologies, and improving collaboration abilities. The concept of structured shared workspace proved suitable for the learning task, however, as expected, significant problems were pointed out: some regarding either the interface, the functionality of the prototype, or the accessibility and reliability of the server; others to group related aspects of engagement, participation or tutor intervention. Related to the interface, the representation of the discussion as an index fully deployed on the left part was uncomfortable to handle when it became long. Some problems of consistency with buttons for editing operations were also detected. Related to the collaborative task, students missed the explicit agreement option. This was considered to be a natural way to conclude a discussion. An important issue was the lack of a separate representation for the process and the outcome. The possibility to look for what was being achieved -drafts of the essay- at different stages of the process, separated from the discussion was perceived as useful.

4 The coordination space, a public list of messages, suffered from lack of privacy. Students wanted to send messages not only to groups but also to individuals, particularly the tutor or the assistant. Facilities for browsing and reading related messages with different criteria, for instance topic, date, etc., were not provided. The period of the experiences lasted for the '97 spring term. A second experience using the same prototype was carried out, involving three groups of students in a master program of another, non distance, university. In this case students combined face to face teaching sessions with collaborative discussions through the system. Groups consisted of three students who knew each other previously. The patterns and style of the interactions were clearly different and specific problems of mutual awareness and concurrence arose due to a more synchronous use of the system. Comments from students, feedback from tutors, and improvements suggested by designers were discussed in the analysis phase of the second version carried out during the summer, in order to have a prototype ready for the 97/98 autumn term. The second and third cycle The main redesign for the second version concerned the structure and function of the shared workspace. Instead of a unique workspace, three related subspaces were defined, one for the joint elaboration, a second one, the result space, to keep separately the outcome, and the third, the version space, to save the process and the outcome at different stages. The conversational graph included agreement as an option. Once all participants had performed this action for a particular proposal, the system automatically loaded the text on the corresponding subtask section of the result space. Thus, each subtask has to be finished by explicit consensus. The conversational graph, wired in the first version, was configurable from an external data file so that it could be different for different tasks. Moreover, the number of workspaces was an option to be defined at the configuration of the prototype, depending on the needs of the learning activity. Related to the interface, these were the main new features: • As the number of spaces increased, on the upper part of the screen, a button bar enabled the user to go directly either to the information space, the individual space, the shared workspaces or the coordination space. • All the shared working spaces (version, result and elaboration) were directly accessible from each other, by buttons, and they have a similar interface. • The representation of the discussion process on the left part of the elaboration workspace was redesigned in order to allow the folding and unfolding of the content index of the contributions for each subtask. This helped to reduce the amount of visible information, focusing on the task at hand. • The interface was rewritten in English to allow a wider use. • Messages in the coordination space could be one-to-one or one-to-many. As in the first cycle, the evaluation phase consisted of steps for fixing the objectives of the evaluation, designing, setting-up and performing the learning experience. In this cycle we aimed at two different scenarios to test the versatility of the system regarding other communities and activities. The first one was to support the discussion of a group of international experts, distributed all over the world, previous to the celebration of a working conference. The second one involved a teacher and their graduate students, all external to the designer's team, using the system as a complementary activity. The learning activity was fully designed by the teacher. In both cases the prototype was configured by a member of the research team to generate an environment adapted to the activity, i.e., number of workspaces, its structures, associated conversational graphs, and links to other sources, as well as roles and participation rules. All these features were different in each experience. Results From users the main feedback in this cycle was the need of: (1) having notification when new things happened related to the group activity; (2) seeing at a glance where new things were placed. From the monitoring of these and previous experiences, we also concluded: In relation to collaborative aspects • In the case of students, procrastination is a problem, support for explicit time management as well as negotiated intermediate deadlines should be considered. • For full distance learners, a preliminary phase to set up the group, establishing common objectives and protocols is a must to ensure their engagement as well as to enforce their feeling of belonging to a group. • In the case of larger, open groups, a well-specified purpose, shared responsibility and the role of a moderator are key factors to promote interaction and participation. The only existence of the mediational tool does not, of itself, generate patterns of collaboration. • Teacher intervention along the collaborative process remains an open question: whether this should be supporting the group spontaneously or on request, being involved as a member of the discussions groups with

5 an specific facilitator role, or entering as commentator at specific time of the process. Students feelings were quite controversial on that point. In relation to the system • On-line help, guidelines for each activity, and rules of participation should be included in the system, in addition to handouts (i.e. the more integrated and self contained the system, the better its usability). For Spanish students, English for the interface was accepted, if all the rest was available in Spanish. • Facilities, beyond notification, to handle, monitor and analyze the collaboration processes are required. Both during and after the experience. • A configuration mode should be available to generate tailored web sites, as automatically as possible from the data provided by the designer of the learning activity. The fourth and fifth cycle At that point we decided to rethink the whole system architecture, characterizing levels and generic components and implementing the workspaces on a database framework. For further details on this point see (Barros & Verdejo, 1998). Besides the full reprogramming, the improvements for the next version concerned two aspects: • The definition of roles for shared workspaces allowing to distribute responsibilities for a learning activity. For each role a profile of features was defined, where accesses and privileges for different uses of the system could be configured. A simple mechanism of notification was also introduced and could be enabled or disabled in the role profile. • The redesign of the coordination space to provide not only an enhanced e-mail facility including a variety of filters to browse the messages with different criteria, but a new tool, a group agenda to facilitate time management matters. Evaluation Phase The evaluation followed the same steps as in the previous cycles. This time the new issue addressed was to test how easy the system could be adapted by the users themselves, for instance, allowing learners to configure workspaces completely tailored to their needs. This possibility opened the range of learning activities that the system could support. For instance, students performing design tasks or negotiating among themselves their collaboration schemas. So we planned and executed an experience where the learners, using the system, could also participate and decide aspects of the learning activity they have to perform in collaboration, such as the conversational graph structure for the elaboration process, roles and their distribution, or the schema for the outcome. We launched an experience comprising three learning activities. The first one, a collaborative synthesis on an Educational Technology topic. The second one, the selection and definition of a case study; and the third the elaboration of a detailed proposal for the case study. A teacher, a technical assistant and two groups of three part-time students following a Ph.D. course worked together for 6 months. • The objectives for the first activity included acquiring knowledge of a selected topic, and developing collaborative abilities as well as becoming familiar with the use of the system. • The second activity was a preparation for the third one, students had to attain three goals: (1) to select, discuss and agree on a case study, i.e. a learning activity, (2) to write a detailed plan to develop their proposal including the structure of the outcome, and (3) to fix the responsibilities of each one on completing this proposal. • The third learning activity included the configuration of the shared workspaces where they could develop the design of their case study. They started to work with the third prototype. This time as the experience lasted for a longer period as soon as some problems were detected we decided to work simultaneously on the system, so that they started with version 3 but finished with version 4. New features were included. For instance the initial bar included all the activities and there was the possibility to state temporal restrictions between activities so that spaces were accessible only when the time conditions were fulfilled. Agreement methods were extended and could be selected between consensus or majority. Further tools were also provided for global and individual evaluation: the representation of accesses per hour, work evolution in the experience period and number of contribution type. For global analysis there were an option for visualizing the evolution of the discussion by access and by user and the evolution of work by section and user. These results could be visualized in graphics or textual form. Also a tool to create, fill, submit and, automatically process WWW-forms was provided in order to systematically collect student feedback. Version four A major extension in version 4 was the concept of an organizational learning memory. This was a structured repository of collaborative learning activities containing selected events and outcomes (Barros & Verdejo, 1998). This memory was defined and implemented with a range of editing and searching facilities. Initially we filled it with relevant data collected in previous evaluation cycles. This resource can be exploited for a variety of

6 educational purposes, two of them, scaffolding and peer evaluation, were tested on the latest cycle developed during the last three months of the 97/98 academic year. Two experiments were carried out. The first comprised two learning activities. One was fully defined by the teacher, the other only partially predefined. The learners once the first activity was completed had to select outcomes of cases stored in the organizational memory to perform a peer review. The second experiment followed a similar approach but this time students could use previous cases as guidelines for their own work and not only for the peer evaluation step. Results • The experiences in this cycle, as well as the system, were more complex. Close monitoring from the teacher became critical, but also a clear state of affairs was very helpful for students. Notifications provided automatically from the system to learners and the teacher about selected events improved the “what is going on” with respect to the other versions, but the solution to present the “what is new” required some further analysis. In short this information has to be presented not only in quantitative terms but also in a more qualitative flavor. • There were also problems with the generation of tailored versions of workspaces. The configuration procedure demanded additional understanding of the system, and the students were not prepared to invest time on that point. They preferred to specify what they wanted and have the technical assistant doing it for them. This suggested that more training on the tool and better configuration facilities are needed. These two points were addressed on the next, current version, of the prototype. The current architecture of the prototype is organized in levels: configuration level, performance level, reuse level and reflection level. We consider the tool robust and versatile enough to finish the development phase and to proceed to a wider dissemination in order to prepare a summative evaluation phase.

Concluding remarks and further work Formative evaluation has been essential to guide the team in tuning the tool and for suggesting directions for further extensions. Prototype testing with real users helps not only to reach technical validity but more importantly, to face the complexity of implementing collaborative learning. Social and institutional factors are essential when designing collaborative learning activities, specially in a conventional educational framework. In this sense Activity Theory seems a promising framework for modeling learning situations with a global approach. Issues for the next evaluation are, on the one hand, to look more closely at pedagogical and collaborative effectiveness, and on the other to collect more experiences. We perceive that practical guidelines for designing and performing these activities, not only looking at learning objectives but also dealing with social and organizational issues, are indeed needed. Acknowledgments The work presented here has been partially funded by CICYT, The Spanish Research Agency, project TEL97-0328-C02-01. References Barros, B. & Verdejo. M.F. (1998) STEED Project (http://sensei.ieec.uned.es/~steed/produc.htm ) Bell, P., Davis, E.A. & Linn, M. (1995) "The Knowledge Integration Environment: Theory and Design", in Proc. CSCL'95 (http://www.kie.berkeley.edu/KIE/info/publications/publications.html). Cole, M.; Engeström, Y. (1993) "A cultural-historical approach to distributed cognition" en Distributed cognitions. Psychological and edcational consdierations (Salomon, editor), Cambridge Univ. Press, 1-46. Collis, B. & Veen, J. (1997) Project ISM-1.(http://utto237.edte.utwente.nl/ism/ism1-97) Crook, C. (1994) Computers and the Collaborative Experience of Learning, Routledgem International Library of Psychology. Edelson, D.C. & O’Neill, D.K. (1994) “The CoVis Collaboratory Notebook: Supporting Collaborative Scientific Inquiry”, NECC’94. (http://www2.covis.nwu.edu/papers/Papers.html) Gould, J.& Lewis,C (1985) “Designing for Usability: Key Principles and What Designer Think”,CACM 28(3) 300-311. Kuutti, K. (1996) “Activity Theory as a Potential Framework for Human-Computer Interaction Research” en Context and Consciousness. Activity Theory and Human-Computer Interaction (Nardi editor), MIT Press, pp. 17-44. Scardamalia, M. & Bereiter, C. (1991) “Higher Levels of Agency for Childern in Knowledge Building: A Challenge for the Design of New Knowledge Media”, The Journal of the Learning Sciences, 1(1), 37-68. Suthers, D. & Jones, D. (1997) "An architecture for Intelligent Collaborative Educational Systems", Proc. AI-ED'97, (Du Bovlay & Mizoguchi editors) 55-62.

7 Wan D., Johnson P. (1994), “Experiences with CLARE: a computer-supported collaborative learning environment” Int.J.Human-Computer Studies, vol. 41, 851-859.