College of Information Science and Technology has developed ... classes; (2) Students and faculty engage in an active learning ... first years of the effort.
Evaluating an Asynchronous Graduate Degree Program Gregory W. Hislop College of Information Science and Technology Drexel University Philadelphia, PA 19104 Abstract -- Over the past three years, Drexel University’s College of Information Science and Technology has developed asynchronous (any time / any place) learning environments for undergraduate and graduate students. Beginning in fall 1996, Drexel launched an entirely asynchronous Master of Science in Information Systems degree. The hallmarks of the fully asynchronous degree are: (1) Students never gather for traditional face-to-face classes; (2) Students and faculty engage in an active learning experience by network; (3) The degree content and requirements are the same in both traditional and asynchronous offerings; (4) Students learn to work using tools and techniques of distributed work environments. In this paper we provide an overview of the Drexel project and our experience during the first years of this effort. Next, we discuss techniques and instruments we are using to evaluate the project both within the ALN environment and in comparison to traditional degree program delivery. Finally, we present some results from the first years of the effort.
Introduction Asynchronous Learning Networks (ALNs) can be characterized by their support of “anytime, anyplace” education [1]. ALN students do not meet in traditional faceto-face classes. Instead, they become part of a network of learners who may be on campus, nearby, or across the country. These students can also choose when they work, giving them the option to do their work during the day, at night, or on the weekends as fits their schedule. The College of Information Science and Technology at Drexel University began a long-term initiative in early 1994 to develop ALN technology and build on earlier projects in computer mediated communication such as [2]. The Drexel initiative began with a period of limited offerings of ALN courses in information systems in our undergraduate and graduate degree programs. During this phase we developed the support structure and methodology for ALN course delivery [3]. In the current phase of the Drexel initiative, we are offering an entire graduate degree via ALN. The first group of students in this program began their course work in fall, 1996. The degree they are completing is identical to the degree offered on-campus in traditional classes, but these
students will never have to attend a class on-campus. All their work will be done via ALN.
Degree Program Overview The Drexel degree prepares students to apply information technology to solving problems in a wide variety of application domains. The degree approaches information technology from a systems engineering perspective, with a special emphasis on human and organizational elements of systems. Students can focus the elective part of their program on particular technologies, such as database systems. They can also pursue a concentration in software engineering. Classes in the traditional program typically meet once per week for three hours and run for an eleven week quarter. In the ALN program there are no scheduled class times, of course, but classes still follow the quarter schedule for start and end dates. Student work activities and methods of evaluation vary with course content, but some of the common elements include: team projects, seminar discussions, papers and reports, and exams. The ALN classes follow this same pattern, with adjustments as appropriate. Seminar-style activities become online discussions. Project teams work asynchronously, or schedule phone conferences or other gatherings as they feel appropriate. Exams are treated as “take-home” exams. The degree program has a total of about 300 students in it. A high percentage of these students is working full-time and pursuing the degree part-time. The primary audience is people working in information technology jobs who seek a professional degree to advance their careers. A secondary audience is people working in another area (finance, research, etc.) who have become interested in the information technology aspects of their job. Several hundred students have participated in ALN courses over the life of the Drexel project. Over thirty have enrolled in the fully asynchronous degree program begun in fall, 1996.
Evaluation Themes The Drexel ALN project is an opportunity to test the viability of ALN as a mainstream educational delivery vehicle. As such, the project evaluation must consider both
the potential of ALN in its own right and the effectiveness of ALN delivery relative to traditional approaches. With this context, and building on other work in the area, we have identified three themes for the evaluation. These are listed below along with some of the basic questions we are working to address within each theme. Quality of education • • •
Do the students receive an education comparable to that provided by traditional delivery? What are the factors that determine the success or failure of individual students? Are there factors that we could use to predict the success or failure of individual students before they start the degree program?
Cost of education • • • •
Are the costs of an ALN delivered degree comparable to the costs of traditional delivery? How do the costs compare for the university? How do the costs compare for the student? How do the costs compare for an employer paying for a student’s education?
Quality of the experience • •
•
Is the overall satisfaction of students and faculty members in an ALN degree program comparable to those in a traditional degree program? How does the ease of participation in an ALN degree program compare to a traditional degree program? (including time required, schedule, location, ease of communication, ease of learning and teaching) How do the quality and amount of social interaction in an ALN program compare to a traditional degree program?
Evaluation Instruments We have developed a set of measurement instruments to support investigation of the first and third themes outlined above. We are pursuing the second theme using financial analyses. The instruments have been administered to both ALN and traditional degree program students, as appropriate. A brief description of the instruments follows: Background Questionnaire - This questionnaire gathers data in several broad categories. The first is demographic data such as age and gender that will allow us to look for systematic differences among students based on demographic factors. A second group of factual questions addresses possible predictors of success and failure. These include factors such as native language, typing ability, technical background, and prior education. A third group of questions probes perceptions and expectations about the
ALN program. This will allow us to track changes in these factors as students participate in the program. Baseline Knowledge Test - This test samples student knowledge at the beginning of the degree program. The test is a series of short essay questions that are graded by expert evaluators. We adopted this more subjective evaluation approach after careful consideration of a knowledge test based on objective questions. We constructed an objective test but, after reviewing the prototype, rejected that approach. We concluded that the objective test covered a good range of topics relevant to the degree, but that the coverage was limited to narrow, factual elements. To evaluate a graduate program, we felt that essay questions would provide more insight into overall student knowledge and higher level skills including analysis and synthesis. Relationships Matrix - This instrument documents basic information about each student’s social and professional relationship with each other student in the program prior to the start of the program. This will provide a basis for analyzing development and change in social interaction as students proceed through the program. It may also be a factor affecting success or failure of students in the program. Post-course Evaluation - This questionnaire gathers student opinions and student estimates of some factual items at the end of each course. The opinion questions explore the student’s reaction to course content, the instructor, and what the student learned without reference to ALN delivery. It also asks about the ALN aspects of the course. In addition to providing opinions, the students provide estimates of the time they spent on the course, when they did the work, and where they worked. They also provide input on problems they encountered due to the ALN.
Comparing ALN and Traditional Delivery The first cohort of students doing the entire MSIS by ALN took their first class in fall, 1996. We had several groups of students starting the MSIS by traditional delivery in the same term. We took advantage of this situation by collecting data to allow comparison between groups. We administered the background questionnaire, the baseline knowledge test, and the post-course evaluation to the students in the traditional delivery class. The only difference for these students was that they did not answer the questions specific to ALN delivery. In addition to the measurement instruments outlined above, we arranged to keep copies of several work products from students in each of the classes. The assignments are such that they require a subjective evaluation by experts, much like the baseline knowledge test. We are using an evaluation of that type to compare the learning by students in the traditional class versus the ALN class.
Evaluation Control Since we are investigating delivery of an entire graduate degree by ALN, we do not have a controlled experimental environment. However, we have taken a variety of steps to try to improve the validity of our result: Blind Evaluation - Each student has a random number assigned as an identifier for the measurement instrument. Evaluators will not know which students they are evaluating nor whether the students were in a traditional or ALN class. To further protect against evaluator bias, we are not using the course instructors as evaluators for courses that they teach in a given term. Comparability Between Groups - The demographic information will allow us to look for systematic variations that seem to be related to demographic factors. Since we have this data for the ALN and traditional groups, we can control for these factors between groups if those differences appear to be significant. Pilot Testing - As we have developed each measurement instrument we have pilot tested it using small samples of students currently in the program or recently graduated. This has been helpful for refining the instruments and to ensure a useful result on the first application of the instrument in the project environment. Instructor coordination - We do not always have the same instructors teaching the ALN and traditional groups and so we need to be mindful of the potential for bias introduced by instructor differences. In these cases, we are trying to minimize this effect by coordinating the plan for the entire term with both instructors. In addition, the instructors meet during the term to ensure that their approaches and activities remain as similar as possible as the term progresses.
Some Initial Results At the time of this writing, our data analysis is just beginning. However, preliminary analysis shows many positive results for ALN courses. Based on a set of 82 student evaluations, some strengths and weaknesses of ALN appear. Some of the strengths of ALN courses are apparent in issues such as: • Convenience - As expected, students find the “anytime, anyplace” flexibility convenient. 87% considered ALN more convenient. • Instructor access - 95% felt that they had better access to the instructor. 43% felt that they actually communicated with the instructor more. • Collaborative learning - 93% found it useful to see the ideas and assignments of other students On the other hand, possible weaknesses of the ALN approach are indicated by points such as:
•
Face-to-face contact - 51% indicated that they missed the in-class lectures of traditional courses. • Level of effort - 40% felt that they had to work harder in the ALN course. • Level of comfort - 21% felt more inhibited about participating in ALN discussions. We are also seeing interesting differences in student perceptions in ALN classes relative to corresponding traditional classes. Interesting results in the first pair of corresponding classes matching 18 students in the ALN degree with 21 in the traditional degree include: • Course goals - 100% of the ALN students and 73% of traditional students felt that the course goals were clear. • Asking questions - 93% of the ALN students and 100% of the traditional students felt that they had good opportunity to ask questions. • Subject interest - 94% of the ALN students and 82% of the traditional students felt that the course increased their interest in the subject. • Motivation - 88% of the ALN students and 86% of the traditional students felt that they were motivated to do their best work. • Time - 74% of ALN students and 24 % of traditional students reported spending 10 hours or more per week on the class. The overall impression that emerges is that ALN students are as positive or more positive about the quality of education they are receiving than the students in traditional classes. The issue of time is one that we plan to investigate further, including the trade-off between absolute hours spent and flexibility gained in scheduling, and trying to factor-in travel time for traditional students. Travel time is not included in the estimate quoted above. Based on phase one of the project, our general impression is that student performance is comparable in traditional and ALN classes. In phase two we will be able to get some quantification of performance from instructor grading and from work product evaluation by independent evaluators. Preliminary data from the first two quarters of the fully ALN degree do not show a clear trend in performance based on instructor grading. In one term the ALN students did better than the traditional students, in the other term the ALN students did less well. In any case, the numbers are still much too small to establish any trend. As the project continues, we will be able to provide additional detail about students’ perceptions of ALN and their performance. In addition, we will be adding our analysis of methods of delivering ALN courses, cost factors, and results of the course delivery relative to traditional means.
References 1.
Ellis, Clarence, et al. “Groupware: Some Issues and Experiences.” In Baecker, Ronald M., ed. Groupware and Computer-Supported Cooperative Work: Assisting in Human-Human Collaboration. San Mateo, CA. Morgan Kaufmann, 1993
2.
Hiltz, Starr Roxanne, and Murray Turoff. The Network Nation: Human Communication via Computer, 2 ed. Cambridge, MA. The MIT Press. 1993.
3.
Andriole, Stephen J., Richard H. Lytle, and Charlton A. Monsanto. “Asynchronous Learning Networks: Drexel’s Experience.” T.H.E. Journal. 23,3. October, 1995. Pp. 97 - 101.
Acknowledgment We gratefully acknowledge the support of the Alfred P. Sloan Foundation for the Drexel ALN initiative and the intellectual leadership of Frank Mayadas in helping to shape work in this area.
