Session T3E
Building the Better Asynchronous Computer Mediated Communication System for Use in Distributed Education Timothy J. Ellis1, Maxine S. Cohen2 Abstract - The viability of distributed educational services delivered via asynchronous learning networks (ALN) has been well established both in practice and through the literature. Despite wide and enthusiastic support for ALN based distributed education, the learning environment is still quite immature. The software constituting the environment has largely been developed in an ad hoc fashion. To great extent, the primary design goal appears to have been to have the online environment simulate the classroom as closely as possible. This paper addresses the design weaknesses of distributed education systems by developing a requirements specification for an effective asynchronous computer mediated communication system for use in a learning environment. The specification was developed through a Nominal Group Technique process involving eight computing technology in education graduate students with significant experience as students in an online environment. The specification was validated through a Delphi process involving a different set of 20 experienced graduate students. Index Terms – asynchronous communications, threaded discussion forums, distance education, online learning environments, online pedagogy. INTRODUCTION The viability of distributed educational services delivered via asynchronous learning networks (ALN) has been well established both in practice and through the literature. Over 1,300 colleges and universities offer degree programs via an online modality. Several thousand articles comparing online to faceto-face instruction and analyzing the strengths and weaknesses of various pedagogical strategies have been published. Despite the wide and enthusiastic support for ALN based distributed education, the learning environment is still quite immature. The software constituting the environment has been largely developed in an ad hoc fashion. To a great extent, the primary design goal appears to have been to have the online environment simulate the traditional classroom as closely as possible. Design was largely based on the limitations inherent in the technology rather than end user needs, and changes have been made incrementally more in response to improvements in
data communications capabilities than to address the needs of the learners. In order for process and design improvements to occur and develop into a more mature, stable environment, a standardized, systematic, and reproducible procedure needs to be followed. Much of the present work in design is either ad hoc or being driven by the available technology. Typically, each new system contains everything of its predecessor and more, creating bloated software that can overwhelm the user with features without directly addressing the actual needs of the user. A clear understanding of the needs and expectations of the users of the ALN is prerequisite to developing a mature, stable environment. The users of ALNs are quite sophisticated and often have clear expectations of computerized systems. Many users have experience with several different computer mediated communication systems (i.e. newsgroups, email, instant messages, bulletin boards, etc.) and know what capabilities they want in a system. Involving these informed users in the early stages of the design process is essential. The goal of this paper is to initiate the desired design process by developing a list of features, functions, or characteristics necessary for an effective asynchronous computer mediated communication system for use in a learning environment, as envisioned by students – one of the two major stakeholders in the environment. Two functional definitions are important within this goal statement: 1. “Communication” entails at least instructor-student, student-instructor, and student-student discourse. Other discourse paths might well be possible and are not eliminated. 2. “Asynchronous computer mediated communication system” refers to only to the hardware, software, policies, and procedures that promote discourse within the learning environment and should not be confused with a comprehensive course management system. In addressing this goal, the study attempted to answer the following research questions: 1. Why is it important to use asynchronous tools? 2. What is inadequate with asynchronous communication tools today?
1
Timothy J. Ellis, Associate Professor, Nova Southeastern University, Graduate School of Computer and Information Sciences, Fort Lauderdale, FL 33314,
[email protected] 2 Maxine S. Cohen, Professor, Nova Southeastern University, Graduate School of Computer and Information Sciences, Fort Lauderdale, FL 33314,
[email protected]
0-7803-9077-6/05/$20.00 © 2005 IEEE October 19 – 22, 2005, Indianapolis, IN 35th ASEE/IEEE Frontiers in Education Conference T3E-15
Session T3E 3.
What features should an asynchronous computer mediated communication system have? BACKGROUND
Teaching online requires a great deal of preparation both in terms of preparing the actual materials and becoming familiar with the technology and its inherent problems. Online contexts do not provide the rich additional features that are found in a face-to-face teaching environment, so online environments need to be carefully designed to maximize the learning opportunities [8]. Online teaching can certainly provide additional features that are missing and not possible in the face-to-face environment, but these features need to be planned and used appropriately. Hicks, Reid, and Rigmor [8] stated that online learning environments have the following eight characteristics: computer-mediation, the potential for accessing large amount of dynamic information through the WWW, the use of hypertext and working with materials in a non-linear way, access to realworld contexts via the internet, the capacity to communicate via email and other electronic technologies with lecturers and other students, new methods for the administration of learning such as submitting assignments and getting results, networking, and internationalizing the curriculum. The common thread in all these systems is the coursework uses the internet and a computer to mediate the course of study. [1] The debate is not whether online teaching should exist. There are many discussions in the research literature already establishing the viability of that instructional delivery modality, including the What’s the Difference Report [18], Russell’s No Significant Difference [14] research, Innovations in Online Learning: Moving Beyond No Significant Difference [16] and The Effectiveness Debate. [12]. Much research has also been directed at how to measure effective online learning. The interested reader may want to review Fraser’s work [5] for his development of the Distance and Open Learning Environment Scales (DOLES) instrument and Walker’s development [17] of the Distance Education Learning Environments Survey (DELES). Interesting research has also been directed at examining how to assess online work, specifically in the discussion forums arena. [4] Despite this attention to online learning environments, research examining the common themes involved in online interaction and what support the environment should provide is, unfortunately, still rather limited. [12] The descriptive literature is practice-based and more of a “how to” do distance learning. [15] Furthermore, a lack of theoretical grounding of web-based communication tools has been noted in the literature. [9,10] Research into learning environments is nothing new. Lewin [11] created a function to describe human behavior. His function can be stated as B=f(P,E), where B, the behavior, is a function of two inputs, P, the person and E, the environment. Although environments have changed greatly since 1936, his basic function remains valid today. Typically, changes do not occur very rapidly with people. Today, people’s exposure to
technology and the mass of information contained in their head based on prior experiences is quite extensive and often carries high expectations of the technology. Designers need to be aware of the technological capabilities of existing computer mediated communication systems, but not lose sight of creating environments that are educationally sound. With the move to a more learner-centered environment, students’ needs and expectations must be included as part of the design parameters. Just a few years ago, students’ expectations of online learning environments were relatively rudimentary and rather unsophisticated. The fact something even worked was considered an accomplishment. Today, students are more sophisticated and have higher expectations. There is little tolerance for barriers caused by unfriendly technology and poor interaction. Hannifan [7] discusses how the designer needs to be aware of the framework and how the frameworks influence the learning environment. Using grounded design principles, the features and requirements can be adequately represented. The importance of participatory and collaborative design methods need to be considered as well.[6] These new environments have changed in focus and are more learner-centered. Designers need to pay attention to learners’ needs and their interaction with the environments. [1] Although cost and time to market are important factors, this research looks at what criteria are needed from a requirements gathering perspective. These systems need to be flexible and usable in different learning contexts. Learning styles and adaptive environments need to be considered in the design mix. The large players in the online course delivery systems (WebCT, BlackBoard, Moodle) all keep adding enhancements and new features to remain competitive. The developers realize they need more than the best features, but an underlying cohesive structure and framework. Moodle [13] has a three page description of their philosophy that they refer to as “social constructionist pedagogy”. Furthermore, they end their philosophical essay with “further improvements in pedagogical support will be a major direction for Moodle development”. It is obvious that building a better environment is not a trivial undertaking. METHODOLOGY Addressing the research questions underlying this study entailed a two-step process. An inclusive set of criteria candidates was developed by an Expert Panel of eight graduate students majoring in computing technology in education. The expert status of the panel was based upon the following criteria: an average of 7.8 predominately asynchronous courses completed; an average of 6.7 courses featuring discussion forum assignments completed; and familiarity of an average of 3.7 different discussion forum systems. The panel utilized a Nominal Group Technique (NGT) approach [3] to develop a consensus on the features and characteristics necessary for a successful educational asynchronous communication environment. The NGT entailed a series of six steps, as described below:
0-7803-9077-6/05/$20.00 © 2005 IEEE October 19 – 22, 2005, Indianapolis, IN 35th ASEE/IEEE Frontiers in Education Conference T3E-16
Session T3E 1.
2.
3.
4.
5.
6.
Initial idea generation. Each member in the group silently listed, on paper, everything they thought would be of importance in an asynchronous communication environment for use in education. Round-robin discussion. Each member of the group presented one and only one item from her or his list at a time. Each item was recorded on a spreadsheet and displayed on an LCD panel. Brief discussion and clarification of each item. The group leader facilitated discussion of each item in the order in which they appeared on the screen. The discussion was limited to questions, statements of clarification, statements of agreement, and statements of disagreements. Preliminary, silent vote by each member. Each member independently rated the items listed on the display on a 5-point scale on which 1 = “Of no importance” and 5 = “Of critical importance”. Discussion of the preliminary vote. The group leaders provided each participant with a printout of the ratings each item received in the preliminary vote and facilitated a second discussion of each item. Final silent, independent vote. Each member again rated each item, using a three point scale: retain, delete, or duplicate of another item.
1.
Initial anonymous vote. Each member was given a link to a Web-form that listed the 48 criteria candidates developed by the Expert Panel. The Validation Panel member was to rate the importance of each item for a well-designed asynchronous communication environment on a five point scale. The members were also allowed to make comments on each item and suggestions for additional items. Figure 1 presents a sample of an item from this first round.
Figure 1: Round 1 Questionnaire 2.
The NGT process described above produced a list of criteria candidates for the ideal asynchronous communication network for the learning environment. Even though the list reflected the informed opinions of experienced stakeholders, it was developed in a very compressed timeframe of only three hours. Validation of the list by a second, more reflective review was indicated. Toward that end, a Validation Panel of 20 experienced online students drawn from the same pool as the Expert Panel was convened to assess the criteria set by means of an online Delphi Method. The Delphi Method [2] is a well-established process for determining group consensus on a topic by a series of questionnaires on which there are anonymous votes followed by structured feedback on the results of the previous round of voting. Typically, the process continues until a document that all find, at the least, acceptable is generated. The online Delphi Method implemented in this study entailed the following steps:
Second anonymous vote. Each member was given a link to a second Web-form that contained item-byitem results from the first round. The goal for this round was to identify items that should be removed from the criteria list.The members were asked to categorize each item using the following scale: • No! = I would not "buy into" using a product with this feature • Prefer not = I would rather not see the feature in the product, but would accept it if present • Don't care = I don't care if the feature is present or not • Nice to have = I would rather see the feature in the product, but would accept it if absent • Vital! = I would not "buy into" using a product without this feature Figure 2 presents an example of a Round 2 item.
Figure 2: Round 2 Feedback and Questionnaire 3.
Third anonymous vote. Each member was given a link to a third Web-form that contained only those items that received both at least one “No! = I would not "buy into" using a product with this feature” and one “Vital! = I would not "buy into" using a product without this feature” vote. In this round the members were asked to resolve “irreconcilable differences” by 0-7803-9077-6/05/$20.00 © 2005 IEEE October 19 – 22, 2005, Indianapolis, IN 35th ASEE/IEEE Frontiers in Education Conference T3E-17
Session T3E categorizing each item using the three-point scale illustrated in Figure 3.
TABLE II CRITERIA CANDIDATES PRODUCING IRRECONCIABLE DIFFERENCES Item: Pro:
Figure 3: Round 3 Feedback and Questionnaire 4.
Since irreconcilable differences still existed after the third round, a fourth round was deemed appropriate. The goal of this round was to add definition to the irreconcilable differences by soliciting explanatory comments for votes of either “No!” or “Vital”. Figure 4 presents an example of an item from this round.
Figure 4: Round 4 Feedback and Questionnaire
Con:
Item: Pro: Con:
intelligent auto-notification of activity on a thread including both postings and viewings of postings Having some method for notifying users of new content sure does cut down on searching for it. This is very distracting--similar to the signal on the email notification--I turn off---auto-notification does not serve a functional purpose for the threads in my point of view---if working asynchronously why do I care who is on etc---I don't and this sort of feature would annoy me if cannot turn off [sic].
Item:
personal rating and marking of contributions by the student
Pro:
Regardless of what students say, I believe they value the opinions of their peers. We live in a world where constructive criticism from one's peers can enable us to become less sensitive and more willing to accept and to handle criticism in situations where the stakes are higher (i.e. instructor-to-student and bossto-employee). It can be a very positive lesson taught in a nonthreatening arena. Knowing that students can rate your contributions and have them recorded could hinder the very interaction an instructor may want.
Con:
RESULTS This study produced two results of interest. Table I lists the criteria for an effective asynchronous communication network for use in an educational setting validated by the students participating in the study. The first column of the table lists the 48 items developed by the Expert Panel. The next five columns present the distribution of ratings for each item from the second round of review by the Validation Panel. Table II explores the areas of irreconcilable difference developed in the second round by listing the criteria on which members of the Validation Panel could not reach agreement. This listing is amplified by a representative sample of supporting rationales for votes of either “No!” or “Vital!” from the fourth round of the validation process.
private student to student and student to instructor communication threads I feel this is important because sometimes the students might want to discuss the course, curriculum or school with other students or the prof without fear of retribution from others, such as bad feelings, downgrading. I would not like to have private threads. Any private communications should be handled in email. Private threads would not support group cohesion.
Item: Pro:
Con:
Item: Pro:
Con:
email functionality that is separate and distinct by course and not mixed with other email (personal/business) Most people using a course management system would also have other email accounts for personal and business use. Keeping the instructional email separate is the best way to insure that other email and spam don't clutter up the instructional mailbox and interfere with learning. While perhaps a personal email account may be desireable [sic], email functionality that is separate and distinct by course could be overwhelming to the student. Too many locations to check for communications. Better to have a filter feature. a persistent instant messenger functionality in which instant messenger communication is saved in the discussion thread This is vital for increasing references used by students. IM communication that can be saved will contribute not only to increasing interaction but also towards increasing the overall knowledge base. No---I do not like pop-up instant messenger--personally I do not like IM---this is not a feature that serves any functional purpose to the course, only to students who like this and want to save these type of communication....
0-7803-9077-6/05/$20.00 © 2005 IEEE October 19 – 22, 2005, Indianapolis, IN 35th ASEE/IEEE Frontiers in Education Conference T3E-18
Session T3E TABLE I CRITERIA FOR AN EFFECTIVE EDUCATIONAL ASYNCHRONOUS COMMUNICATION NETWORK
The ideal Asynchronous Learning Network communication environment should offer the student the capability of … ensuring security and privacy adding attachments to posts obvious indication and easy access to new postings/material user selectable level of detail - collapsible threads an easy to use/user customizable interface some sort of standard interface across courses without inhibiting academic freedom editing and deleting posts by both author and the course professor searching and sorting postings by multiple criteria printing the entire thread easily ADA compliance navigation aides so the student can pinpoint location in the environment at all times scrolling through a whole conversation creating threads interactive help features to assist users with the functionality of the system control over the logical structure of how the postings are listed/viewed viewing a variety of media types to satisfy different learning styles document sharing capability for collaboration a single point of entry into an asynchronous portal an html editor for creating enhanced postings that include tables, fonts, and images, without having to code spell check/grammar check facilities saving a draft of a post for later posting cross platform capabilities (SCORM) a priotorizable scheme for listing postings "banking" links from the threads, creating a consolidated list of links from all postings in a thread a calendar of planned availability/unavailability for students and professor additional course content that can augment initial course information immediate access to library databases via single sign in saving the discussion area on her or his computer private student to student and student to instructor communication threads1 intelligent auto-notification of activity on a thread including both postings and viewings of postings1 multiple window capability having voice, video, and typewritten chat and chat rooms viewing the work of others when assignment is complete allowing video and voice over IP to facilitate interview and group study creating and posting a multimedia presentation for critique and/or review alternative graphic displays of the discussion area i.e. semantic web a student directory with pictures and bio that is accessible while reading or creating posting seeing and hearing in class discussions that were prerecorded during class sessions a counter feature accessible to student and faculty to facilitate counting postings by thread/topic remotely viewing instructor in the classroom - live or prerecorded querying who's online currently taking electronic notes while viewing the online lecture and have them synchronize with lectures email functionality that is separate and distinct by course and not mixed with other email (personal/business) 1 a personal reflection tool like a blog zooming the display in and out a persistent instant messenger functionality in which instant messenger communication is saved in the discussion thread1 peer and self assessment built-in2 personal rating and marking of contributions by the student1
1 0% 0% 0% 0% 0% 0% 0% 0% 0% 0% 0% 0% 0% 0% 0% 0% 0% 0% 0% 0% 0% 0% 0% 0% 0% 0% 0% 0% 6% 6% 0% 0% 0% 0% 0% 0% 0% 0% 0% 0% 0% 0% 19% 0% 0% 6% 6% 13%
Rating Distribution 2 3 4 0% 0% 0% 0% 0% 6% 0% 0% 6% 0% 0% 13% 0% 0% 13% 0% 0% 25% 0% 0% 31% 0% 6% 19% 0% 6% 19% 0% 13% 6% 0% 6% 25% 0% 6% 31% 0% 6% 31% 0% 0% 50% 0% 6% 44% 0% 6% 44% 0% 19% 38% 0% 19% 38% 6% 0% 63% 0% 13% 56% 0% 19% 44% 0% 25% 31% 0% 13% 63% 0% 13% 63% 0% 25% 38% 0% 6% 75% 0% 13% 69% 0% 19% 56% 6% 6% 44% 6% 6% 44% 6% 19% 44% 0% 13% 81% 0% 31% 50% 0% 25% 69% 6% 19% 69% 0% 31% 63% 6% 31% 50% 13% 25% 44% 0% 50% 44% 0% 56% 38% 6% 56% 25% 0% 56% 44% 13% 6% 31% 13% 38% 44% 6% 50% 44% 19% 31% 38% 13% 44% 38% 25% 25% 31%
5 100% 94% 94% 88% 88% 75% 69% 75% 75% 81% 69% 63% 63% 50% 50% 50% 44% 44% 31% 31% 38% 44% 25% 25% 38% 19% 19% 25% 38% 38% 31% 6% 19% 6% 6% 6% 13% 19% 6% 6% 13% 0% 31% 6% 0% 6% 0% 6%
1 = No! 2 = Prefer not 3 = Don't care 4 = Nice to have 5 = Vital! 1
Items with “irreconcilable differences”, requiring votes on Rounds 3 and 4
2
Item removed after the second round
CONCLUSIONS The asynchronous communication facilities currently available in online learning networks are clearly inadequate. The students who participated in this study had no difficulty identifying a number of features, functionalities, and characteristics that could greatly enhance the learning environment.
This list should, of course, be viewed as an initial set of requirements for the next generation asynchronous learning environment. Further research in two areas is definitely indicated: replicating the process executed in this study using a wider range of students in terms of both discipline and academic level; and replicating the process using experienced faculty to gain the perspective of the second major stakeholder in the learning relationship.
0-7803-9077-6/05/$20.00 © 2005 IEEE October 19 – 22, 2005, Indianapolis, IN 35th ASEE/IEEE Frontiers in Education Conference T3E-19
Session T3E REFERENCES [1] [2] [3] [4] [5] [6] [7]
[8] [9]
[10] [11] [12] [13] [14] [15]
[16]
[17]
[18]
Abdelraheem, A. “Computerized learning environments: problems, design challenges and future promises”. The Journal of Interactive Online Learning, 2, (2), 2003. http://www.ncolr.org Adler, M. and E. Ziglio. Gazing into the Oracle: The Delphi Method and Its Application to Social Policy and Public Health. London, Jessica Kingsley Publishers, 1996. Delbecq. A. L., Van de Ven, A. H. & Gustafson, D. Group techniques for program planning: A guide to nominal group and Delphi process. Glenview, IL: Scott Foresman, 1975. Dringus, L. P., & Ellis, T. “Using data mining as a strategy for assessing asynchronous discussion forums”. Computers & Education, 45, 2005, 141-160. Fraser, B. J. “Classroom environment instruments: development, validity and application”. Learning Environments Research, 1, 1998, 7-33. Hakkinen, P. “Challenges for design of computer-based learning environments. British Journal of Educational Technology”, 33 (4), 2002, 461-469. Hannafin, M. J. “Resource-based teaching and learning: A new generation of Web-based enhanced learning environments”. Proceedings of the Educational Technology/Emerging Technologies in Higher Education Symposium, Muscat, Oman: Sultan Qaboos University, 2001, 61-77. Hicks, M, Reid, I.C. & Rigmor, G. “Designing responsive online learning environments: approaches to supporting students”. AARE conference. 1999. Jarvela, S., & Hakkinen, P. The levels of web-based discussions: using perspective-taking theory as an analytical tool. In H. v. Oostendorp (Ed.), Cognition in a digital world, (pp. 7795). Mahwah, NJ: Lawrence Erlbaum Associates, 2003. Koschmann, T. D. “Toward a theory of computer support for collaborative learning”. Journal of Learning Sciences, 3(3), 1994, 219-225. Lewin, K. Principles of topological psychology (F. Heider & G.M. Heider, Trans.). New York: McGraw-Hill, 1936. Merisotis, J. P., & Olsen, J. K. “The ‘effectiveness’ debate: what we know about the quality of distance learning in the U.S.” TechKnowLogia, 2 (1), 2000, 42-44. Moodle [online] Available from World Wide Web http://www.moodle.org Russell, T. L. “The no significant difference phenomenon”. Raleigh: North Carolina State University, Office of Instructional Telecommunications, 1999. Sunal, D. W., Sunal, C.S., Odell, M.R.& Sundberg, C.A. “Research-supported best practices for developing online learning”. The journal of interactive online learning, 2 (1), summer, 2003. http://www.ncolr.org Twigg, C. A. “Innovations in online learning: moving beyond no significant difference”. Pew Learning and Technology Program, Center for Academic Transformation, Rensselaer Polytechnic Institute, 2001. http://www.center.rpi.edu Walker, S. “Evaluation, description & effects of distance education learning environments in higher education”. 9th Annual International Distance Education Conference, Center for Distance Learning Research, January 22-25, 2002. What’s the difference report? [online]. Available from World Wide Web http://www.ihep.com
0-7803-9077-6/05/$20.00 © 2005 IEEE October 19 – 22, 2005, Indianapolis, IN 35th ASEE/IEEE Frontiers in Education Conference T3E-20
