Session F1D
Validating a Criteria Set for an Online Learning Environment Maxine S. Cohen1and Timothy J. Ellis2
Abstract - This paper presents the continuing effort to gain a better sense of what constitutes quality in an online course. This work was initially presented at FIE 2002, and expanded in a Journal of Engineering Education article accepted as part of the “Best of FIE” edition (April, 2004). Our research started with a broad view that entailed an initial brainstorming activity followed by a series of ranking and rating processes and a factor analysis to create a set of quality indicators from the student perspective. The current research extends the process by examining the validity of the factors. There are different approaches to performing validation activities; we choose the Nominal Group Technique (NGT), which has been used successfully to generate a group consensus for over 30 years. Thirty two students were divided into five groups; one for each of the factors generated from the earlier research efforts. Each group was tasked with developing a detailed definition of the content and construct underlying the assigned factor. Online educators are often unsure of how to structure their courses since few have any experience as students and little experience as teachers in the environment. By following a clear and reproducible methodology, using multiple constituencies, and re-evaluating at different steps in the process, this study provides online educators a meaningful framework upon which to structure course activities. Index Terms - Distance education, evaluation, online learning, quality indicators. INTRODUCTION Distance education has certainly become a main stream offering on today’s college campuses. Students, professors, and administrators all have differing perspectives and varied vested interests on this form of education. Many discussions abound on “doing” online learning effectively and the mechanics of the actual online teaching. Regardless of the perspectives and the various issues that exist, everyone is in agreement that online education is here to stay. Although there is a cadre of research on online learning that is student focused and presents student perspectives, research efforts center on the characteristics of successful online students and techniques to do online teaching. Little research has been found that focuses on qualitatively and quantitatively
evaluating quality in online courses. This study is a start at exploring and defining the area of quality in online education. In order to begin the process of establishing some baseline quality indicators for online courses several research activities have been undertaken. A total of 125 students experienced in on-line education, enrolled in a doctoral program in a graduate school of computer and information sciences participated in a series of three studies. To create a set of quality indicators we conducted a brainstorming activity with 10 students followed by a series of ranking and rating processes by an additional 44 students. This initial work [1] generated a set of 15 quality indicators that appeared to fall into three categories: instructorstudent interaction, student-student interaction, and class organization. Further research [2] incorporating the rankings and ratings of an additional 71 students offered the opportunity to conduct a statistical factor analysis in which five factors were identified. Table I presents the 15 quality indicators initially identified, separated into the five factors derived through the later research. TABLE I Factorial Analysis Factor 1 Question 1 Question 2 Question 8 Factor 2 Question 6 Question 7 Factor 3 Question 3 Question 4 Question 10 Factor 4 Question 5 Question 11 Question 15 Factor 5 Question 12 Question 14 Un-Factored Questions Question 9 Question 13
Community of learners Connection with professor Connection with other students Effective student-to-student communication Instructor accessibility Effective instructor-to-students communication Effective student-to-instructor communication Class organization Learner (student)-centered Expectations clearly articulated Self-paced schedule “Feel” of the class Immediately engages the student Simulates an in class ‘feel’ Incorporation of leading-edge technologies Peer Impact Class Size Peers adequately prepared for online course Anytime, anyplace learning Feedback clear, timely, and meaningful
The goal of this paper is to continue this research by validating the quality indicators identified by the previous research efforts. The previous research activities had identified a set of factors, but the question still remained, do these factors reso-
1 Maxine S. Cohen, Nova Southeastern University, Graduate School of Computer and Information Sciences, 3301 College Ave, Fort Lauderdale, FL 33314,
[email protected] 2 Timothy J. Ellis, Nova Southeastern University, Graduate School of Computer and Information Sciences, College Ave, Fort Lauderdale, FL 33314,
[email protected]
0-7803-8552-7/04/$20.00 © 2004 IEEE October 20 – 23, 2004, Savannah, GA 34th ASEE/IEEE Frontiers in Education Conference F1D-23
Session F1D nate with the end-users (the students that are the benefit of the online education)? Research Questions In this paper, we are trying to answer the following research questions: 1.
2.
How do end-users define the constituents of the quality indicator factors? For this particular study end-users are defined as students. What factors should be added and/or removed in a valid rating instrument to measure quality in online education?
Assumptions and Limitations This research must be viewed in the context of the assumptions and limitations upon which it is based. The students participating in the current study were all graduate students in a Human Computer Interaction course in an Information Systems curriculum; it is assumed that they were representative of graduate students in a technology-intensive curiculum. It is also assumed that the students participated in the validation activity in good faith and responded honestly and accurately. Since the participants in the study were all drawn from a single graduate class from a school of computer and information sciences, the results of this study are delimited by both course content and level of instruction. These results may not be applicable to students in less technology-intensive disciplines or undergraduate students. BACKGROUND Although there is much to say about online education from both the faculty and student perspectives, this brief background section will focus on the five factors identified by the previous research efforts after taking a quick look at the importance of identifying quality indicators for online education. Why Quality With the current academic environment seriously involved in assessment issues, it is possible that a rating system defining online education quality might be on the horizon. Quality based issues are not always on the forefront of decisions about distance education [3]. Lee Alley tried to identify if there was a difference in quality issues for online courses versus traditional courses. He found that when students rate an online course, they look for similar things as found in traditional courses. A knowledgeable professor who interacts with the students is an important criterion, along with features that help create a learning community among the class members [4]. It seems there is a current shift in academe to a more “customer-centered” viewpoint. If academic course work is considered a marketable product, it is possible to see course ratings posted out on the Web similar in style to Amazon and e-bay rating systems. This can lead to outsiders defining what makes for a quality course [2].
Community of Learners Establishing a community of learners seems to resonate in most discussions and research about online education. Psychological isolation can be a problem in online environments. Pictures and creating a social space are important to create a learning community [5]. Without standard class meetings and seeing each other face to face as in a traditional classroom, there is a need to re-create the closeness that the distance seems to cause to disappear. Rovai did an analysis measuring community of two ALNbased university programs. He found that community was stronger in the program that provided learners more and diverse opportunities to interact with each other and that the most important community components in which the groups differed were spirit and trust [6]. The existence of synchronous chats and asynchronous discussion forums are two common tools used to create the online community. With online programs it seems that the online community does not end when the course ends. Often, especially in graduate programs, the community established in the course, continues throughout the program. Community could help in lowering the drop out rate which seems to be higher in online courses. Instructor Accessibility In any course, students expect feedback from faculty, but in the online environment, without good feedback in a timely fashion, students can tend to feel isolated. This is further exacerbated in the online environment, where the 24/7 accessibility seems to be the expectation. Some research [7] has found that the quality of interaction with the instructor is the most significant contributor to a student’s perception of learning. Online learners want prompt feedback, specific feedback, responses from fellow students (without humiliation). They prefer negative comments to be communicated privately via email or with a phone call [8]. The online student population presents many teaching challenges. In a traditional class, students have been “raised” to wait until office hours or the next class meeting to interact with the professor; in the online environment that maxim is ignored. It is not uncommon to have students write multiple email messages to the professor if they do not have a response within a few hours. Time zone boundaries seem to disappear. Students start contacting the professor often before the official start of class. Online students are seen as daring and confrontational regarding their expression of ideas [9]. Although the instructor may be seen as the “guide on the side” students certainly want to have individual access to the professor at all times of the day and/or night. Class Organization Most courses today, online or traditional, use the Web to present the organizational “face” of the course. The course Web page contains the syllabus, reading assignments, and special instructions. With the hypertext organization inherent in the structure of a Web page, interactive links to the course can eas-
0-7803-8552-7/04/$20.00 © 2004 IEEE October 20 – 23, 2004, Savannah, GA 34th ASEE/IEEE Frontiers in Education Conference F1D-24
Session F1D ily be provided. Although many course Web pages are static in nature, they still provide another communication channel for the student. Many online courses are seen as self-paced, but often the instructor does have a time schedule that is followed. Presenting that schedule to the students is part of the class organization. In online courses, there is no natural time clock. “Feel” of the Class
to the exercise and had been thoroughly briefed on the both the process and the subject to be discussed. The NGT entailed a series of six steps, as described below: 1.
2.
Without the face-to-face interaction in an online class, the professor needs to let the students know how the class is going, what is the “flavor” of the class, when class is in session, etc. It takes a lot of creativity and effort to capture this in online courses. Although there is the flexibility of anytime, any place learning in online courses, there still needs to be an establishment of the class culture and expectations. Today students often have had several experiences with online education from work related training programs to formal credit bearing classes and bring these former experiences with them into each online experience. Each of these types of education brings different course expectations. It is up to the professor to set the tone and establish the “feel” of the class. The use of the technology and the computer expectations are also part of the class feel. Engaging the student to feel part of a virtual environment is an important, but difficult task.
3.
4.
5. 6.
Peer Impact As mentioned in the previous discussions online education has many features in common with traditional classes and has quite a few that are different. In a traditional class, it is possible to attend the class and except for feedback from the instructor never interact with another soul during the entire course. That lack of interaction would be almost impossible in an online class. The use of discussion forums and chat tools and therefore interaction with others in the class is an inherent part of most online courses. One of our previous research efforts [1] showed that “connection with other students” and “effective student-tostudent communication” ranked rather low with the students. This result is a surprising finding, since these factors seem to impact the community of learners. A different student population type may have had a different reaction to these factors. Technology-oriented students are typically not very chatty. They also are relatively familiar and comfortable with the technology and therefore may not need to “reach out” to their classmates for help with technological issues. METHODOLOGY The research questions were explored using a Nominal Group Technique (NGT) approach [10]. A total of 32 participants were drawn from a doctoral-level course in Human Computer Interaction. The participants were divided into five groups, corresponding to the five factors identified in Table I. Each of the groups was led by two doctoral students from the Computing Technology in Education program, one served as group facilitator, and the second as group recorder. Each of the group leaders had received two-hours of orientation to the NGT prior
Initial idea generation. Each member in the group silently listed, on paper, everything they thought would be indicative of the quality factor being examined. Round-robin discussion. Each member of the group presented one and only one item from her or his list at a time. Each item was written on the flip chart by the recorder. Brief discussion and clarification of each item. The group leader facilitated discussion of each item in the order in which they appeared on the chart. The discussion was limited to questions, statements of clarification, statements of agreement, and statements of disagreements. Preliminary, silent vote by each member. Each member independently rank-ordered the items listed on the flip chart from most important to least important and rated each item on a five-point scale. Discussion of the preliminary vote. The group leaders facilitated a second discussion of each item. Final silent, independent vote. Each member again ranked each item listed on the flip chart from most important to least important and rated each item on a five-point scale. RESULTS
Tables II through VI present the final versions of the items identified for each of the quality indicator factors. Table II Community of learners Item Quality of faculty to facilitate, remain current, be competent Community tools Online student support Library services Feedback quality Availability of instructors Quality technology Quality group activities Orientation to course Ethics, plagiarism, respect for others Student accessibility to hardware and software Computer skills Instruction on Web design University reputation Meet course demand Seek diversity in faculty and learner Visibility of courses Complete requirements guide Quality of textbooks Comprehensive and fair evaluation General information to build community Length of time for course Consistent with syllabus Common bond
Rank
Ratings
1
4.83
2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24
4.33 4.17 4.6 4.4 4.6 5 3.5 4 3.8 5 3.75 4.25 4 3.67 3.5 3.25 3.5 4 4.25 3.6 3 4 3.2
0-7803-8552-7/04/$20.00 © 2004 IEEE October 20 – 23, 2004, Savannah, GA 34th ASEE/IEEE Frontiers in Education Conference F1D-25
Session F1D Item
Table III Feel of the Class Rank
Table V Class Organization Ratings
Course material
1
5
Positive interaction
2
5
Website usability
3
4.8
Instructor Enthused and energized
4
4.8
5
4.6
Item
There are some interesting findings in the tables that should be noted. Table II, Community of Learners, generated a list of 24 factors, far larger than any other factor. A review of the items included in the factor clearly indicates a wide dispersion of ideas, suggesting that the students were struggling to define the inherent components of the factor. The items identified for the other factors, as listed in Tables III through IV were much more clearly defined and narrowly scoped, suggesting a resonance of the factor among the students. For example, although there were 14 to 15 components identified for the factors peer impact, class organization, and instructor accessibility, the component pieces were tightly grouped and closely aligned with the more commonly accepted definition of the factor. Table IV Peer Impact Item
Rank
Ratings
Rank
Ratings
Clear directions/instructor expectations
1
5
Information on the Website Posted assignments, due dates and requirements in advance Quick response from tech support, instructor immediacy
2
5
3
4.8
4
4.3
Tips, links, instructions, examples
5
3.6
Increase the flow of info access from webpage
6
2.8
Consistency in standards for overall school
7
3.5
Instructor-student interaction
8
3.7
Structure of homepage and links
9
3.2
Choosing the right, reliable online tools
10
3.2
Interaction during term, group work, collaboration Setup the promote sameness with traditional class structure
11
3
12
2.5
Communication with the larger student body
13
2.7
Free software
14
2
Table VI Instructor Accessibility Item
Rank
Ratings
Speed of on time response Clearly defined written instructions on instructor access to include flexible synchronous/asynchronous online office hours Publish schedule (vacations, etc.), update and clarified at start of term
1
5
2
4.67
3
4.5
Knowledge enhancement
1
5
Thought provoking
2
4.83
Multiple contact methods (mail, phone, etc.)
4
4.5
Adaptability
3
4.66
5
5
Instructor feedback
4
4.5
6
4.5
Instructor experience
5
4.3
Course delivery
6
4.3
7
3.83
Diversity
7
4.16
8
4
Interaction of students
8
4.16
Constructive criticism
9
3.66
Clear responses Willingness and basic attitude of the instructor to interact with the class Constant, positive interaction between instructor and students. Reassurance and encouragement of instructor anytime, anywhere Alternative backup plan to access instructor if not contact in reasonable time
9
4.8
10
3.5
Maintain online Website/Resources
10
3.8
Number of students Internal expectations
11
3.16
Availability of online contact tools
11
3.33
Friendship
12
2.83
Reach instructor 24/7
12
3.67
Bios
13
2.5
Good accessibility -- part of the team and class
13
3.5
Not peer impact
14
1.66
Feedback survey to elicit responses from students Student choice of mentor/advisor during orientation
14
2.83
15
2.83
CONCLUSIONS The goal of this study was to answer two questions: 1.
How do end-users define the constituents of the quality indicator factors? For this particular study endusers are defined as students. 2. What factors should be added and/or removed in a valid rating instrument to measure quality in online education? In addressing these questions, five factors that had been identified in previous research [1, 2] were analyzed through the use 0-7803-8552-7/04/$20.00 © 2004 IEEE October 20 – 23, 2004, Savannah, GA 34th ASEE/IEEE Frontiers in Education Conference F1D-26
Session F1D of a Nominal Group Technique. Four of the factors – instructor accessibility, class organization, feel of the class, and peer impact – appeared to match well with student expectations and seemed to accurately reflect the student perspective of the constituents of quality in a course delivered via an online modality. The one remaining factor – community of learners – however, did not appear to resonate with the students participating in the study. Although the literature strongly supports the importance of community in an online environment [5, 6], and faculty view that aspect of the learning environment as very important [2], the very great range of constituents discussed as elements of this factor suggests that, to the student, the sense of community is simply not as important nor clearly defined. Ongoing research into the constituents of quality in online learning is certainly indicated. This study, as detailed in the assumptions and limitations section, focused on a narrow band of students – graduate students in a technology-intensive program. It would be very interesting to extend the methodology applied in this study to students at the undergraduate level and to a broader range of disciplines. It would also be very interesting to apply the methodology to the other stakeholders in higher education: the faculty and administration. Identifying the factors indicating quality in online education from the faculty perspective and the administrative perspective, and correlating those perspectives with the indicators identified from the student perspective, could also be quite interesting.
[9]
Kubala, T. “Addressing student needs: Teaching on the Internet”. T.H.E. Journal, 25(8), March, 1998, pp. 71-75.
[10]
Delbecq. A. L., Van de Ven, A. H. & Gustafson, D. Group techniques for program planning: A guide to nominal group and Delphi process. Glenview, IL: Scott Foresman, 1975.
REFERENCES [1]
Cohen, M.S. and Ellis, T. J. “Developing a criteria set for an online learning environment”. Proceedings of the 32nd ASEE/IEEE Frontiers in Education Conference, 2002, pp. T3E8-T3E12.
[2]
Cohen, M.S. and Ellis, T.J. “Developing a criteria set for an online learning environment: from the student and faculty perspectives”. Journal of Engineering Education. In press.
[3]
Sherry, A. “Quality and its measurement in distance education”. Handbook of Distance Education. Ed. M.G. Moore and W.G. Anderson. Mahwah, NJ: Lawrence Erlbaum Associates, 2003. pp.435-459.
[4]
Carnevale, D. “What makes an online course succeed? Not everyone agrees, a study finds”. The Chronicle of Higher Education, October 27, 2000.
[5]
Bernard, R., Rubalcava, B. and St-Pierre, D. “Collaborative online distance learning: issues for future practice and research”. Distance Education, 21 (2), 2000, pp. 260-277.
[6]
Rovai, A. “Sense of community, perceived cognitive learning, and persistence in asynchronous learning networks”. The Internet and Higher Education, 5(4), 2002, pp. 319-332.
[7]
Fredericksen, E., Pickett, A., Shea, P., Pelz, W., and Swan, K. “Student satisfaction and perceived learning with on-line courses: Principles and examples from the SUNY learning network”. Journal of Asynchronous Learning Networks, 4(2), 2000. http://www.aln.org/publications/jaln/v4n2/v4n2_fredericksen.asp
[8]
Rossman, M. “Successful Online Teaching Using an Asychronous Learner Discussion Forum”. Journal of Asynchronous Learning Networks, 3 (2), 1999.
0-7803-8552-7/04/$20.00 © 2004 IEEE October 20 – 23, 2004, Savannah, GA 34th ASEE/IEEE Frontiers in Education Conference F1D-27
