only the controlled experiment, with random assignment of subjects to conditions, is an acceptable method for testing hypotheses. In examining this premise, ...
Evaluating the Importance of Collaborative Learning in ALN’s Starr Roxanne Hiltz and Raquel Benbunan-Fich Department of Computer and Information Science New Jersey Institute of Technology Newark, NJ 07102 ABSTRACT - Asynchronous Learning Networks (ALNs) represent a new paradigm for teaching and learning. A basic premise of our research efforts has been that collaborative or group learning, which can be facilitated by anytime/anywhere access to the group communication and work space, is key to achieving superior learning outcomes in this medium. This paper describes some of the major evaluation procedures and results for a recently completed three year project that produced and delivered 26 courses in Computer and Information Science using ALN, to over 1000 students. It focuses on some of the methodological problems and inconsistencies in results that occur in trying to assess the process and outcomes of teaching and learning in ALNs.
1. Introduction Over the last decade, NJIT has constructed a series of computer-mediated communication systems tailored to support “anytime/ anywhere” interaction among students and instructors in an ALN system called “Virtual Classroom,” used the system first in a variety of individual courses and then for full degree programs, and developed various evaluation instruments and approaches. Qualitative approaches, including protocol analysis and standardized reports from instructors, have been used primarily for “formative” evaluation, while quantitative approaches including pre and post course questionnaires, and quasiexperimental and experimental designs for longitudinal studies, have been used for summative evaluation. In this paper, we will present the overall theoretical model that guided the research. We will go on to describe what is meant by “collaborative learning” and why we feel that ALN’s and collaborative learning are “made for each other.” Then we will show data that tests the hypotheses about the importance of collaborative learning to outcomes in ALN courses, based on four kinds of data: 1. Subjective reports of outcomes by students comparing ALN and face to face sections of courses, via a post-course questionnaire. 2. Grades for students in sections of the courses offered via different modes. 3. Subjective reports of learning related to a specific assignment (ethics scenarios), by students randomly assigned to four communication conditions. 4. Final exam scores for these students.
What we will see is that the conclusion reached about whether or not ALN’s are “better” than traditional modes, and whether or not group assignments are superior to individual assignments, depends upon what kind of dependent variable and measure one looks at. “Objective” measures of learning based on final exams or course grades show no significant differences related to mode of learning. On the other hand, subjective assessments by students do.
2. Methodological Lament One: Why Experiments on ALN’s Aimed at Distance Learners Have Questionable Validity Those trained in experimental psychology tend to feel that only the controlled experiment, with random assignment of subjects to conditions, is an acceptable method for testing hypotheses. In examining this premise, one must remember the difference between “internal validity” and “external validity.” The controlled manipulation of conditions combined with random assignment to these conditions can do the best job of maximizing internal validity by ”ruling out” any explanations for differences in outcomes other than the difference in treatments. External validity, by contrast, refers to the extent to which results of a study can be generalized beyond the circumstances under which it was studied. Asynchronous Learning Networks, as implemented at NJIT, have as one of the primary goals, increasing access to education for busy adults for whom travel to regular oncampus class meetings ranges from inconvenient to impossible, and also improving quality of outcomes for this population in particular. ALNs may also improve quality of educational experiences for some “traditional” on-campus students; this is a separate research question. It is obviously not possible to randomly assign the target population of students to on-campus and distance sections. Given the requirement for availability to be assigned to (mostly daytime) classes for face-to-face sections, and for home or workplace access to state-of-the-art computer equipment for assignment to ALN based sections, self-selection is the only feasible or practical method for determining which students will receive which educational treatment. One can fruitfully explore some interesting issues by randomly assigning students who self-selected into an on-campus face-to-face section, to complete all or some of their work using an ALN, by assigning them to go to computer labs on campus to do
these assignments. However, the generalizability of results of such a study to working adults whose demographic, educational, motivational and computer equipment situations are quite different, is extremely questionable. On the other hand, by choosing a quasiexperimental design, we lose internal validity; we are unable to “control” a myriad of circumstances that differ between the two sets of students. In particular, the students who selfselect into the distance modes are older, work many more hours per week and are more likely to be married and have children. These different life situations may strongly affect both motivation and the amount of time devoted to a course, and thus affect course outcomes as measured by grades and whether or not students withdraw before completion, or take longer than the usual semester to complete a course. In other words, they might be “better” or “worse” students, who spend more or less time on a course than traditional oncampus students participating in a course delivered via faceto-face meetings, and this is likely to be a much stronger determinant of outcomes than the mode of delivery.
3. Methodological Lament Two: Objective vs. Subjective Measures Recently, a number of empirical studies comparing learning outcomes in computer-supported versus face-to-face groups have been carried out (Alavi, 1994; Hiltz, 1994; Leidner and Fuller, 1996; Reinig, 1996, Benbunan-Fich, 1997). Most of these studies measured two types learning, objective and subjective. Objective learning refers to the actual or observed learning. The traditional way in which actual or observed learning is measured is through an exam in which the participants are asked to recall and apply the concepts "learned" in the process of completing an experimental task. Subjective learning is concerned with perceived learning. It is usually measured by a post-test questionnaire, in which participants report their own perceptions about the learning experience Studies in the area of technology supported collaborative learning in synchronous environments using “Group Support Systems” (GSS) have found mixed results in objective and subjective learning. Leidner and Fuller (1996) found that students working collaboratively in groups (discussing cases) perceived themselves to learn more than students who worked alone, but students who worked individually outperformed students who discussed the cases in groups before preparing their individual response to the case. Another study (Reinig, 1996) compared two sections of an introductory MIS course; one used GSS-support for the discussion of eight ethical scenarios in groups, while the other section carried out the discussions face-to-face with no GSS-support. It was observed that students in the GSSsupported class learned more (measured by the scores obtained on the final exam) than students in the traditional class, but no significant differences in self-reported learning were observed. Alavi (1994) reports that GSS-supported collaborative learning leads to higher levels of both
subjective (self-reported) learning and objective learning. Regarding asynchronous networks, the first phase of the Virtual Classroom project at NJIT found no consistent significant differences between traditional and VC supported classes in mastery of the material as measured by grades. Subjectively, however, most students reported that VC is overall a better way of learning than traditional classes (Hiltz, 1994).
4. The Theoretical Model and the Role of Collaborative Learning The “contingency model” theoretical framework used in this study is summarized in Figure 1. The extent to which students engage in collaborative learning online is posited as a key intervening variable which determines whether or not the desired outcomes of improved learning will occur. Passive approaches to learning assume that students "learn" by receiving and assimilating knowledge individually, independent from others (Bouton and Garth, 1983). In contrast, active approaches present learning as a social process which takes place through communication with others. In particular, collaborative or group learning refers to instructional methods that encourage students to work together on academic tasks. It involves interpersonal processes by which a small group of students work together to complete an academic problem-solving task that promotes learning. Collaborative learning is essentially different from the traditional “direct-transfer” or “one-way knowledge transmission” model in which the instructor is the only source of knowledge or skills (Harasim, 1990). In collaborative learning, instruction is learnercentered rather than teacher-centered and knowledge is viewed as a social construct, facilitated by peer interaction, evaluation and cooperation. Therefore, the role of the teacher changes from transferring knowledge to students to being a guide in the students' construction of their own knowledge. Some examples of collaborative learning activities are seminar-style presentations and discussions, debates, group projects, simulation and role-playing exercises, and collaborative composition of essays, stories or research plans (Hiltz and Turoff, 1993). This new conception of learning shifts the focus from teacher-student interaction and to the role of peer relationships in educational success (Johnson, 1981).
5. The Quasi-Experimental Study of Computer and Information Science Courses As part of the project to produce and deliver all of the major courses needed for a B.A. in Information Systems via a combination of Virtual Classroom (VC) plus videotapes, data were collected not only on all students in sections of these experimental courses, but also in sections of the same course taught by the same instructor or set of instructors, using roughly the same syllabus, in three other modes:
traditional face-to-face, “traditional” distance mode of all video, and a combination of face-to-face and VC. Though the mountain of data are still being analyzed, we have the main results for the questionnaire (over 600 responses from students who used the system) and grade data for all three years. In the post-course questionnaire, students were requested to compare their experiences in their course which used VC, to that in other college courses delivered face-toface. Generally, the results of these subjective evaluations were very positive. For example: x Over half of the students in the VC + video experimental sections felt that having this option available enabled them to complete more courses that semester than would have otherwise been possible (and thus make faster progress toward their degree.) x Subjectively, the majority of students feel that the Virtual Classroom improved the convenience of course access (73%), access to their professors (65%), and the quality of learning (58%).
x
Correlation statistics support the theoretical premise that active participation online by both faculty and students, and the use of group or collaborative learning strategies in ALN, are positively related to desirable outcomes.
However, outcomes as measured by grades in the courses show no significant differences between modes (Table 1). For many of the courses, faculty who produced and ALN version did not subsequently offer any purely face-to-face sections, so no comparison is possible. For the 11 courses for which there were a sufficient number of students in FtF and ALN modes, only two showed significant differences, one in each direction. It is hypothesized that when we have the data on SAT scores and Grade Point Averages, that these will account for so much of the variance in final grades, that there is practically no room left for mode of delivery to make a difference. Some students are just “good” at taking examinations, and others are not.
Figure 1: A Causal Model for VC
Technology
Course
Student Characteristics
Equipment access
Level Class size
Motivation
Software Functionality
Ability Type of Subject
Software Interface (Usability)
Instructor Skill and Effort
Skills
Reliability
Organizational Context
Attributes
Amount and Type of Use of VC Perceived Media Richness Active participation? Collaborative learning? ACCESS To Professor Convenience Progress to degree
Quality Course Outcomes VC Satisfaction Grades
Table 1. Grades By Mode: Summary Of Results Course 113 114 231 251 332 365 390 431 432 455 461 totals
No Apparent Diff
VC+ Video Sig. Better
VC+ Video Sig. Worse
VC Appears Better X
VC Appears Worse X X
X X X
X
X X X 2
1
1
X 5
2
Notes: Courses for which there were VC + Video and totally Face-to-Face sections. Significant differences are based on a Chi Square test for which the results were significantly different at the .05 level. “Appears better” means that there were roughly 10 percentage points more A’s and B’s, or 10% or more less Failures, in VC + Video than in FtF, but the differences were not statistically significant at the .05 level. “Appears worse” means that there were 10% or greater more A’s and B’s, or fewer Failures, in FtF than in VC + Video, but the differences were not statistically significant.
6. A Field Experiment on Collaborative Learning Recently completed dissertation research (Benbunan-Fich, 1997) is based on a field experiment that compared groups and individuals solving ethical case scenarios, with and without computer-mediated communication support. A 2x2 factorial design crossing two modes of communication (offline with a task time of two hours vs. asynchronous computer conference with a task period of ten days; these times were established as optimal in pilot studies) and two types of teamwork (individuals working alone vs. individuals collaborating in groups) was designed to assess the separate and joint effects of medium of communication and collaborative vs. individual learning strategies. In the individual offline condition, students solved the case individually, in an in-class exercise like an open-book quiz, and received individual grades based on their own performance. In the individual on line condition, students submitted their individual responses on the computer conference by using the question-response activity software on Virtual Classroom. (This feature allows students to submit their individual responses without seeing what anybody else has written, but after their solutions are posted, they can read the answers of others.) In the group offline condition, team members discussed and solved the case by interacting face-toface and prepared their report. In the group online condition, team members interacted asynchronously using the computer conference as the only means of communication, and submitted a group report.. The subjects were 136 NJIT undergraduate students in the core course Computers and Society, and the ethics
scenario which was the experimental task was one of the assignments in the course. Assignment to experimental conditions was done as close to randomly as possible. Most of the students were in a combination face-to-face plus VC course, but some were in the VC +video condition and could not be assigned to come to campus. Students randomly assigned to a group condition were then randomly assigned to a specific group. Perceived learning was measured immediately after the experiment in the post-test questionnaire, using a seven item scale adapted from Hiltz (1994; Chronbach’s alpha = 92). “Actual” learning was measured in the final exam with two similar ethical scenarios, two weeks after the experiment ended. Table 2. Results of the Controlled Experiment Perceived Learning, by Mode of Learning: Variance Communication Mode FtF (Manual) Online Both
Analysis of
Indiv.
Group
Both
30.47 26.81 28.64
30.15 31.38 30.77
30.31 29.10 29.91
F Anova: Model 2.07 Source Teamwork 3.22 Online Effect 1.04 Interaction 4.23 *= sig. at p
