Learning Effects of Asynchronous Learning Networks: A Comparison of Groups and Individuals Solving Ethical Case Scenarios Raquel Benbunan-Fich Department of Information Systems Stern School of Business, NYU
[email protected]
Starr Roxanne Hiltz Computer and Information Science New Jersey Institute of Technology
[email protected]
Abstract
mastery of the material (actual learning) as measured by grades. Subjectively, by contrast, most students reported that VC is overall a better way of learning than traditional classes [8]. For this particular study, conducted in the context of the VC research program, a field experiment was conducted to compare individuals and groups using a computer conference or a manual mode to solve ethical case scenarios. This experiment measured objective and subjective learning. The analysis of these outcomes can advance our understanding of the benefits of asynchronous learning networks. Whereas prior field studies of ALN’s tend to confound the effects of online learning with those of individuals and group (or collaborative) learning, the factorial design used in this experiment allows us to determine their separate and joint effects.
The field experiment described in this paper explores self-reported learning and actual learning outcomes when Asynchronous Learning Networks (ALN) vs. manual methods are used in the individual and group solution of ethical scenarios. A 2x2 factorial design crosses two modes of communication (manual vs. asynchronous computer conference) and two types of teamwork (individuals working alone vs. individuals working in groups). The results of this field experiment suggest that the use of ALN’s can be as effective as a traditional or face-to-face medium for conducting collaborative activities, such as the discussion and solution of cases in groups. Teamwork combined with asynchronous support can enhance the perception of learning. However, individual work in interaction with online materials negatively affects the perception of learning and subsequent exam performance.
2. Theoretical Background 1. Introduction Recently, a number of empirical studies comparing learning outcomes in computer-supported versus unsupported groups have been carried out ([1], [3], [8], [10], [14]). Most of these studies measured two types of learning: objective and subjective. Objective learning refers to the actual learning, traditionally measured through an exam in which participants are asked to recall as well as apply the concepts learned in the process of completing an experimental task. Subjective learning is concerned with self-reported or perceived learning. It is usually measured by a posttest questionnaire, in which participants report their own perceptions about the learning experience and many other aspects of the experiment. Prior field studies of Asynchronous Learning Networks (ALN), such as the Virtual Classroom or VC (a long term research project on the use of asynchronous communication systems in entire courses), found no consistent significant differences between traditional and VC supported classes in
Different learning models can be placed on a continuum ranging from objectivism to constructivism. The objectivist approach assumes that there is an objective reality and the goal of learning is to understand it and modify the behavior accordingly. On the contrary, the constructivist perspective (based on the work of [13]) assumes that knowledge is created or constructed by each learner. For a recent review of learning models see [9]. One of the derivations of the constructivist approach is the cognitive information processing model. According to this model, learning involves processing inputs to develop, test, and refine mental models in long-term memory [16]. At the cognitive level, learning is a process of constructing, extending and refining mental models, and using them in problem-solving situations [1]. Collaborative learning is one approach towards facilitating constructivism. In this approach, learning emerges from the interaction of individuals with other individuals [17]. Learning occurs as individuals
1060-3425/98 $10.00 (C) 1998 IEEE
exercise, verify, solidify, and improve their mental models through discussion and information sharing during the problem solving process ([1], [9]), and as team members engage in reciprocal teaching [12]. Case studies are effective ways to engage students in problem-solving activities that challenge mental models and allow their refinement. “The goal of the cases is to enable students to process instructional inputs and assimilate the course material. Such cases can be analyzed individually or in the context of a group” [10: 294]. When case discussions take place in a group setting, higher order cognitive skills are developed [8]. Moreover, the contribution of different understandings or the exposure to alternative points of view can enhance learning. Thus, the discussion and solution of case scenarios in groups may accelerate the creation or refinement of improved mental models and augment learning. Collaboration and teamwork can support the development of advanced mental models for a number of reasons. First, there is an opportunity for evaluation and feedback in which group members can monitor individual thinking and provide feedback for clarification and change. Second, the exposure to alternative points of view can challenge understanding and motivate learning [6]. Third, a group structure provides social support and encouragement for individual efforts [1]. Groupware technology can support collaborative learning activities by providing an environment in which group interaction is more effective and efficient. A computer-mediated communication system to support group processes can increase group process gains, such as synergy, pooling of more information, objective evaluation, cognitive stimulation and learning; and decrease group process losses, such as fragmentation, blocking, domination, evaluation apprehension and information overload [11]. Asynchronous Group Support Systems (GSS), in particular, can facilitate self-pacing and self-directed learning and increase the time available to read or reread a message and formulate a comment. This can improve in-depth reflection and development of a topic [7]. Increased opportunity for member input may enhance the quality of decision making [15]. The downside of ALN’s includes procrastination because students do not have to participate at any specific time, they may not participate regularly at all. The anxiety produced by delays and different participation rates (or “login-lags” [4]) may reduce the quality of decision making, because members may go along with an initial suggestion, even if they do not agree with it, in order to accelerate the process and meet a deadline [7]. In addition, students may feel that
the medium is not as warm or personal as face-to-face classes, and this may also decrease motivation.
3. Review of Empirical Studies Various studies in the area of technology supported collaborative learning using synchronous GSS have reported mixed results in objective and subjective learning. Table 1 compares the findings in self-reported and actual learning for each study. Leidner and Fuller [10] observed that students working collaboratively in groups (discussing cases) perceived themselves to learn more than students who worked alone. However, students who worked individually outperformed students who discussed the cases in groups before preparing their individual response to the case, as measured by the quality of the final reports they submitted. In this study, the individual solutions can be considered as a measure of actual learning. Therefore, those who participated in the group discussion, prior to solving the case on their own, learned less than those students who worked individually all the time. Yet, the self-reported learning measure reflects exactly the opposite. Students who discussed in groups perceived that they had learned more than students who worked alone. Another study [14] compared two sections of an introductory MIS course held in two consecutive semesters. One section used GSS support for the discussion of eight ethical scenarios in groups, while the other section carried out the discussions face-toface with no GSS support. This study found that students in the GSS-supported class learned more (measured by the scores obtained in the last question of the final exam) than students in the traditional class, but no significant differences in self-reported learning were observed. Alavi [1] is one of the few studies reporting that GSS-supported collaborative learning leads to higher levels of both subjective (self-reported) learning and objective learning (measured by the grades in the final exam). She also compared the use of GSS support versus a traditional face-to-face approach to case discussions in an MIS introductory course. There are interesting similarities among these studies. They all ran for at least one semester and were focused on MIS introductory courses, using synchronous GSS to discuss several case scenarios. Although the findings across studies seem to be favorable to the use of GSS tools to support case discussions, only Alavi found consistent results between objective and subjective measures of learning.
1060-3425/98 $10.00 (C) 1998 IEEE
Table 1: Comparison between objective and subjective learning in different studies Alavi [1] Leidner and Fuller [10] Reinig [14]
Learning Perception GSS > non-GSS GSS > non-GSS (Individual) No significant differences
4. Hypotheses There is a need to separate the role that working in groups (teamwork) and using an asynchronous computer-supported communication medium play in learning. Therefore, three main research questions motivated the field experiment on which this paper is focused: (1) How does teamwork, virtual or face-to-face, affect learning outcomes? (2) How does the use of a GSS as an Asynchronous Learning Network affect objective and subjective learning in groups and individuals? (3) How do teamwork and GSS interact? (i.e. is online learning dependent on teamwork for good results?) Based on the literature review, the following hypotheses were formulated.
4.1. Self-Reported Learning Hypotheses Based on Leidner and Fuller’s [10] findings, it is expected that participants working in groups will perceive that they have learned more than participants working alone. Thus, H1a: Group participants will perceive higher levels of self-reported learning than individuals working alone Moreover, the combination of asynchronous work, i.e. more time to process a comment and think about personal contributions, linked to the availability of a written transcript of the interaction, can augment learning. Hence, H1b: Computer-supported participants will perceive higher levels of learning than non-supported participants Due to the advantages of working in groups (exposure to different understandings) and the availability of an asynchronous system to support group communication, a positive interaction effect between teamwork and computer-support is expected.
Actual (Observed) Learning GSS > non-GSS GSS< non-GSS (Individual) GSS > non-GSS
On the other hand, working alone online is likely to lack stimulation and motivation for participants, and they may not work as long or as hard as individuals pooling their efforts in a collaborative endeavor. Therefore, H1c: Computer-supported groups will perceive the highest levels of self-reported learning
4.2. Actual Learning Hypotheses Collaborative or group learning involves three attributes of effective learning: active construction of knowledge, cooperation or teamwork, and learning by problem solving [1]. Therefore, it is expected that, as measured by exam scores: H2a: Group participants will learn more than individual participants The online exposure to alternative points of view and different responses will enhance learning [6] in computer-supported individuals when compared to individuals working without computer support. Therefore, H2b: Participants working online will learn more than participants working manually. As in the case of self-reported learning, the availability of a written transcript of the interaction, linked to the flexibility to participate by choosing the most convenient time and place, can augment learning ([7], [8]). Hence, H2c: Participants in computer-supported groups will learn more than will participants in any other condition.
1060-3425/98 $10.00 (C) 1998 IEEE
5. Research Design and Methodology
an actual college course, in the debriefing the students were presented with a model solution to the case.
The experimental design was a 2x2 factorial, crossing two modes of communication (offline-manual vs. asynchronous computer conference on EIES2) and two types of teamwork (individuals working alone vs. individuals working in groups). In the individual manual condition, students solved the case individually, in an inclass exercise like an open-book quiz. In the individual online condition, students submitted their individual responses in the computer conference by using the Question/Response activity software on the Electronic Information Exchange System (EIES2). The Question/Response Activity allows students to submit their individual responses without seeing what anybody else has written, but after their solutions are posted, they can read the answers of others. In the group manual condition, team members discuss and solve the case by interacting face-to-face. In the allotted time (two hours), they discuss the scenario and write the report. In the group online condition, team members interact asynchronously using the computer conference as the only means of communication, in order to solve the case. The participants are 136 NJIT undergraduate students in a core course for computer science majors (“Computers and Society”). Since this course is traditionally offered in two modes -- face-to-face and distance --, students in face-to-face sections could be assigned to any condition, while those in distance sections could be assigned to online conditions only. Subjects were distributed across conditions as follows: 44 in Individual/Manual, 42 in Individual/Online, 28 in Groups/Manual and 22 in Groups/Online. Due to scheduling constraints and the loss of groups because of “no-shows”, fewer participants completed the experiment in group conditions, but each of these conditions ended up with five teams of 4 to 6 students. The task was the solution of an ethical case scenario (“Jane’s Case” in [2]). This kind of task accomplishes two objectives at the same time: it encourages group discussion and allows the practice of ethical analysis. A case worksheet was added to the case scenario to standardize the solution reports and to introduce questions which have correct answers, whose responses could be objectively graded. The task was implemented as one of the assignments in the course. This allowed us to obtain a measure of long-term learning (recall and application of concepts) by including two similar ethical scenarios in the final exam for the course. Upon completing the experiment and the post-test questionnaire, students were debriefed about experimental methods, research design, and hypotheses. Since this experiment was implemented in the context of
5.1. Learning Measures Learning perception was measured immediately after the experiment concluded in the post-test questionnaire through a seven item scale adapted from [8] (see Table 2). The students responded on a fivepoint Likert scale ranging from 1 = Strongly Disagree to 5 = Strongly Agree. A Cronbach’s Alpha of .92 was obtained, indicating a very high reliability of the selfreported learning scale. Actual learning was measured in the final exam with two similar ethical scenarios, two weeks after the experiment ended. Table 2: Learning Perception Scale Items 1. My skill in ethical thinking was increased by solving this case 2. My ability to integrate facts and develop recommendations improved with this case 3. My ability to critically analyze ethical issues improved 4. I learned to interrelate topics and ideas 5. I learned a great deal from this case 6. The case aided my learning 7. I learned to identify central issues in this field 8. Solving this case was a good learning experience
6. Results 6.1. Learning Perception Results The scores of the items in the self-reported learning scale were added up to create a composite variable, which was used to test the hypotheses in this category. Means and Analysis of Variance results are shown in Table 3. Because this was a field experiment, the minimum level of significance for assessing the results as worthy of note was .10. However, a minimum of .05 is required to refer to the results as “statistically significant”.
1060-3425/98 $10.00 (C) 1998 IEEE
Table 3: Means and ANOVA Results for SelfReported Learning1 Condition Mean Individual/Manual 30.47 Individual/Online 26.81 Groups/Manual 30.15 Groups/Online 31.38
was no significant instructor effect (F = 1.14; p = .32). The ANOVA test of exam scores is shown in Table 4. There are no significant factor effects (Teamwork: F=.91; p=.34 and Online: F=.32; p=.57); and no significant interaction effect either (Teamwork * Online: F=.27; p =.61). It is worthy of note that Individuals/Online are once again the lowest scoring condition.
Individual Group Manual Online
28.64 30.77 30.31 29.10
Model
F val. 2.07
p 0.08*
Table 4: Means and ANOVA Results for Actual Learning Condition Mean Individual/Manual 69.43 Individual/Online 66.27 Groups/Manual 73.33 Groups/Online 70.57
Source Teamwork Effect Online Effect Interaction Effect
3.22 1.04 4.23
0.07* 0.31 0.04**
Individual Group Manual Online
67.85 71.38 71.95 68.42
Model
F val. .57
p .64
Source Teamwork Effect Online Effect Interaction Effect
.91 .32 .27
.34 .57 .61
1 Self-Reported Learning: min = 5; max = 40; *=p .1). Hence, H2a (“Group participants will learn more than individual participants”) was not supported. Students in computer-supported conditions scored lower than their manual counterparts, but not significantly so (Online effect: F = .32, p > .1). Thus, H2b (“Participants working online will learn more than participants working manually”) was not supported. An interaction effect was expected. The theory suggested than online groups would reflect higher levels of actual learning than the rest of the conditions, but this was not the case. It is worthy of note, however, that Individuals/Online are once again the lowest scoring condition. Therefore, H2c (“Participants in computer-supported groups will learn more than will participants in any other condition”) was not supported. Since hypotheses 2a, 2b and 2c were not supported by the data, further statistical analysis was conducted with exam scores using the Grade Point Average (GPA) as a covariate. This variable was chosen
1060-3425/98 $10.00 (C) 1998 IEEE
because it measures the students’ academic ability, which is usually a good predictor of exam performance. Results are shown in Table 5. Table 5: Means and ANOVA Results for Actual Learning using GPA as covariate Condition Mean Individual/Manual 69.43 Individual/Online 66.27 Groups/Manual 73.33 Groups/Online 70.57 Individual Group Manual Online
67.85 71.38 71.95 68.42
Model
F val. 6.53
p .0001***
Source Teamwork Effect Online Effect Interaction Effect GPA
2.20 4.23 0.43 21.95
.14 .04** .51 .0001***
**=sig. at p
