Web-based learning: effects on learning process ... - Semantic Scholar

350

IEEE TRANSACTIONS ON EDUCATION, VOL. 45, NO. 4, NOVEMBER 2002

Web-Based Learning: Effects on Learning Process and Outcome Mohamed Khalifa and Rinky Lam

Abstract—Educators have witnessed lately a proliferation of Web-based learning applications. These Web-learning environments have made learning much more convenient by stretching the spatial and temporal barriers. Their effectiveness, however, remains to be examined. In this research, the authors study the relative effectiveness of two different types of Web-learning environments: distributed passive learning (DPL) versus distributed interactive learning (DIL) environments. In the DPL environment, the Web is only used to deliver linear learning material, such as Word files and PowerPoint slides. In the DIL environment, however, the learning material is in hypertext format, providing the learner with more exploration and interactivity capabilities. The results of an empirical study show that the DIL environment is superior to the DPL environment in terms of both the learning process and the learning outcome. Index Terms—Constructivist learning, distributed learning, hypertext, interactive learning, Web-based learning.

I. INTRODUCTION

I

N ADDITION to enhancing traditional educational methods, information technology (IT) can also enable new ways of education delivery and innovative pedagogic strategies. Teaching is no longer confined to a time and a place. With computer and communication technologies, the teacher and the students become spatially and temporally dispersed learners. The time and physical boundaries of the traditional classroom are stretched to a learning space. A growing number of universities worldwide are now offering virtual education programs. Several companies are also providing online training for their employees. A simple search on the Web will result in hundreds of sites offering virtual courses or resources for developing and delivering such courses. Distance education, virtual classes, cyber courses, and interactive learning are just a few of the multitude of terms that are used to describe different implementations of this technology-enabled learning space. Multiple learning methods, e.g., constructivist, collaborative, and experiential, are being supported. Web-based technologies are certainly revolutionizing education. An examination of cyber courses reveals several different possible applications of Web technologies to the creation of different types of learning environments. Although some Web sites are merely delivering traditional learning material (e.g., Word documents, PowerPoint presentations), others are taking a better advantage of hypertext and collaborative Manuscript received November 26, 2001; revised April 1, 2002. This work was substantially supported by CityU Strategic Research Grant 7000900. The authors are with the Department of Information Systems, City University of Hong Kong, Kowloon, Hong Kong S.A.R. of the P.R.C. Digital Object Identifier 10.1109/TE.2002.804395

technologies. These different environments are obviously not equally effective. What are the effects of different Web-based learning environments on learning and performance? What are the main guidelines for designing effective Web-based learning environments? These are important questions for researchers and practitioners alike. Indeed, several researchers, e.g., Alavi and Leidner [1], call for greater depth and breadth of research on such topics. To determine the relative effectiveness of different environments, educators need to examine the instructional strategies supported by these environments. After all, IT is just an enabler. What makes the difference is not the technology itself, but rather, the learning methods enabled and supported by the technology. In this paper, the authors empirically study the relative effectiveness of two Web-based learning environments, i.e., distributed passive learning (DPL) versus distributed interactive learning (DIL). More specifically, a learning environment in which the Web is used to deliver traditional learning material is compared to an environment in which the Web is used to support hypertext-based exploration in terms of the learning process and outcome. In other words, the authors try to determine whether the effort required to transform traditional learning material, such as Word files and PowerPoint presentations, into a hypertext format is justified by an improvement in the process of learning and the level of knowledge acquired by the learners. The paper is organized as follows. First, the research framework is presented and the hypotheses discussed. The description of an empirical study designed to verify the hypotheses follows. Finally, the results and their implications are discussed. II. RESEARCH FRAMEWORK AND HYPOTHESES The effectiveness of a learning environment is primarily determined by the learning method or methods supported. Leidner and Jarvenpaa [2] did a comprehensive review of the main learning methods. These included objectivism, constructivism, collaborativism, cognitive information processing, and socioculturism. Among these, only objectivism and constructivism are radically different, because the others (i.e., collaborativism, cognitive information processing, and socioculturism) are extensions or derivations of constructivism. Objectivism is a teacher-centered learning method whose goal is to represent and transfer objective realities from the instructors to the learners. Instructors control the pace of learning and determine what knowledge should be presented and/or delivered to the learners. In an objectivist environment, the learners become passive recipients of instruction, acquiring and assimilating common understanding from instructors or

0018-9359/02$17.00 © 2002 IEEE

KHALIFA AND LAM: WEB-BASED LEARNING: EFFECTS ON LEARNING PROCESS AND OUTCOME

experts, rather than creating their own knowledge. Web-based environments that support the objectivist learning method are mainly used to deliver learning material without enabling any interaction with the material or with other learners. In contrast with objectivism, the existence of a common external reality is denied in constructivism, which stipulates that each individual can explore and produce unique concepts based on their experience and biases. Constructivism is learner-centered, meaning that the role of the instructor is to assist the learners in constructing their own knowledge. Proponents of constructivism criticize the objectivism method on the grounds that it transmits knowledge to students with little concern for whether the students understand or assimilate the material into their cognitive schema [3]. Web-based environments that support the constructivist learning method encourage free exploration of the learning material and enable a greater level of interactivity. Based on the learning methods discussed previously, the authors identify two fundamentally different Web-based learning environments: the DPL environments and the DIL environments. In the DPL environment, the Web is used to deliver learning material that the learners can access at their own convenience. The learning method, however, remains objectivist. The information representation is mainly linear (e.g., text file, presentation slides, and videos) and is not interactive. The learner is passively receiving information from the instructor. The DIL, on the other hand, is used to describe a constructivist-learning environment that encourages exploration and enables interactivity. The Web is used not only as a delivery medium but also to foster free exploration of information and to allow the learner to interact with the material and other learners. The differences between the DPL Web sites and the DIL Web sites can be examined from multiple, interrelated perspectives: information organization, learning process, and learning outcome (see Fig. 1). The information organization affects the learning process, and, in turn, both the information organization and the learning process influence the learning outcome. In the DPL environment, the Web is mainly used to deliver traditional learning material anywhere and anytime. The main characteristic of traditional learning material is its linearity. It usually consists of text files (e.g., syllabus, guidelines, lecture notes, and cases) and/or presentation slides (a copy of the visual aids used to deliver the lectures, e.g., PowerPoint files). Sometimes, it also includes linear multimedia material (e.g., audio and video RealPlayer files). These materials can be downloaded through the Web site to be used by the learners at their convenience. In the DIL environment, on the other hand, the Web is used to enable the learners 1) to explore the learning material freely at their own pace and 2) to interact with the instructor and other learners. The first capability (i.e., exploration) is the main requirement for constructivism in its basic form. The second capability (i.e., interaction), on the other hand, enables the extension of constructivism to collaborative learning. In this study, the authors compare the basic form of constructivism (i.e., exploration) to objectivism (i.e., delivery). Several Web sites fall into the DPL category since it requires minimal authorship effort. The instructors can simply upload

351

Fig. 1. Research framework.

the same teaching material that they have been using for years to their Web site. They do not need to redesign the learning material into the nonlinear structure that is enabled by hypertext technology. Hypertext design is usually difficult and time consuming, especially when performed for the first time. Its difficulty does not stem from the technology, because simple hypertext structures can be created with word processors, but rather, from a radical change in the presentation of information. The instructors must learn how to present the learning material in a meaningful and effective nonlinear format. The organization of the information is mainly linear in the DPL environment and is hypertext based in the DIL environment. Hypertext is believed to have important advantages from two standpoints: knowledge representation and interface modality [4]. As a user interface modality, hypertext enables the accessibility of domain knowledge from different perspectives (hyperlinks originating from different nodes), allowing exploration at different levels of detail and granularity. The ability of the students to explore at different levels is the characteristic that underlies the hypothesized superiority of hypertext over linear text in terms of effect on the learning process. Hypothesis 1: The “interactive distributed learning” Web sites (i.e., hypertext-based learning material) will enable a more active and explorative learning process than will the “passive distributed learning” Web sites (mainly linear material). Hypertext is more appropriate than linear text for open learning applications in which the learner is allowed freedom of action and encouraged to take the initiative [5]. The learner can freely navigate through the hypertext structure, deciding on the nodes to access and the sequence of accessing them. With a linear structure, on the other hand, the learner is constrained by a sequential access structure that is designed according to the mental model of the author (e.g., instructor). Although the hypertext navigational structure is also designed according to

352

IEEE TRANSACTIONS ON EDUCATION, VOL. 45, NO. 4, NOVEMBER 2002

the author’s mental model, the navigation of such a structure is sufficiently flexible to accommodate the mental model of the learner. Hypertext navigation allows for different levels of prior knowledge and facilitates exploration [6]. Explorative learning, as argued by several researchers [5]–[8], is more effective than instruction-based learning. From the knowledge representation perspective, hypertext supports the associative nature of the human mind [5], [6], [9]–[11]. According to Jonassen [12], knowledge represented with hypertext preserves the rich associations among concepts and corresponds to the knowledge schema of an expert (i.e., the author). This concept underlies the hypothesized superiority of hypertext over linear text in terms of the effect on the learning outcome (knowledge acquisition). Hypothesis 2: The “interactive distributed learning” Web sites (hypertext-based learning material) will facilitate the acquisition of a higher level of understanding by the learners than will the “passive distributed learning” Web sites (mainly linear material). In contrast to learning through passive exposure to information, interactivity is especially valued for its ability to engage learners in the material [13], [14]. Hypertext enables contextualized access to knowledge, shown by Mao and Benbasat [15] to enhance understanding, to reduce the motivational cost of learning and to be highly effective for resolving comprehension difficulties. The associations provided by links between nodes representing related concepts facilitate the processes of remembering, concept formation, and understanding [16], [17]. However, not all hypertext structures are effective. The hypertext navigational structure should map an explicit representation of an expert’s schema [9] to have a significant effect on the knowledge structure constructed by the learner [10], [12]. Designed in such away, the hypertext representation of the learning material has the potential of bridging the gap between the schema of the expert and that of the novice.

III. EMPIRICAL STUDY To test the hypotheses presented in the previous section, a field experiment was conducted. The experiment involved 32 undergraduate students enrolled in an introductory Management Information Systems course. All students were familiar with two browsers (Microsoft Internet Explorer and Netscape), and they knew not only how to navigate the Web, but also how to upload and download files using a number of tools. The students were randomly divided into two groups that used two different Web sites in an alternate fashion to prepare for class. One of the Web sites represented the DPL environment; the second one represented the DIL environment. Both groups had access to both sites alternatively to prepare for different lectures. Group 1 used the DPL site to prepare for the odd lectures and the DIL site for the even lectures. Group 2, on the other hand, used the DIL site to prepare for the odd lectures and the DPL site to prepare for the even lectures. This experimental design allowed us to perform both within-group and between-group comparisons. The students were given passwords that determined which particular Web site they could access at any given time. The learning ma-

terial for any given lecture was made available to the students one week before the lecture. To measure the effect of these Web sites on learning, the students had to take a weekly quiz designed to test their level of understanding of the learning material before each lecture. A total of five quizzes covering five different topics were given. For the first quiz, the students did not use any Web material, but rather, a hard copy of the lecture notes and the presentation slides. The first quiz was used as a pretreatment test of possible group differences. A -test at the 5% level did not show any significant differences between the two groups in terms of performance (grades for the quiz). Group 1 then used the DIL site to prepare for quizzes 2 and 4 and the DPL site to prepare for quizzes 3 and 5. Group 2, on the other hand, used the DPL site for quizzes 2 and 4 and the DIL site for quizzes 3 and 5. To encourage the students to perform well, they were told that the best two out of five scores obtained would count toward their final grade. After the completion of the experiments, the students were given a survey designed to assess their perceptions regarding the effects of the two Web environments on the learning process. A. Design of the Web Sites The DPL Web site was designed to deliver only linear learning material, such as Word files and PowerPoint presentations. As illustrated in Fig. 2, the page design consisted of two frames, with the upper frame controlling the lower one. The upper frame contained two buttons: a Word file icon and a PowerPoint file icon. The first button loaded a Word file containing the lecture notes into the second frame; the second button loaded a PowerPoint presentation to be used as a visual aid for the lecture. This design is typical of DPL Web sites, which by definition serve only for distributing traditional learning material. The DIL Web site integrated into a hypertext navigational structure the lecture notes and presentation slides used in the DPL site. As illustrated in Fig. 3, the page design consisted of two frames with the left frame controlling the right one. The left frame contained hyperlinks representing the table of contents. It enabled a direct access to the different topics covered by the lecture notes and provided a visual representation of the hierarchical structure of these topics. The right frame contained the hypertext representation of the material covering the topic selected from the left frame. The hypertext navigational structure was designed according to a concept map produced by the instructor. A concept map is a graphical representation of concepts and meaningful relationships among them [18]. It consists of a graph in which the nodes represent concepts at different levels of abstraction, and the links and crosslinks represent propositions depicting meaningful relationships between concepts. Concept maps are explicit representations of integrated knowledge networks [19]. They have been used as a basis for the elicitation of knowledge structures and for the mapping of these structures into computer-based instructional systems [10]. Concept mapping is, in fact, one of the most frequently used knowledge elicitation techniques [18], [20]. The translation of the concept map into a hypertext navigational structure was simply achieved by representing each concept with a separate screen and each link/crosslink with a hyperlink labeled to

KHALIFA AND LAM: WEB-BASED LEARNING: EFFECTS ON LEARNING PROCESS AND OUTCOME

Fig. 2.

Sample screen of DPL Web site.

Fig. 3.

Sample screen of DIL Web site.

convey the meaning of the relationship. Designed in this way, the hypertext navigational structure provided a more explicit representation of the instructor’s structural knowledge than did the linear material delivered by the DPL site.

353

B. Measurements To assess the effects of the two Web learning environments on the learning process, the authors relied on student percep-

354

IEEE TRANSACTIONS ON EDUCATION, VOL. 45, NO. 4, NOVEMBER 2002

TABLE I PERCEIVED EFFECTS OF WEB SITES ON LEARNING PROCESS

tions. The students were asked to evaluate the two environments in terms of process effectiveness, support of exploration, degree of interactivity, and enjoyment. More specifically, for each environment they had to rate on a 5-point Likert scale (from strongly disagree to strongly agree) their level of agreement with the following statements: • The Web site was effective in supporting my learning method. • The Web site enabled me to explore the learning material freely. • The Web site provided me with the appropriate level of interactivity with the learning material. • I enjoyed using the Web site to learn. To study the effect of the two Web learning environments on the learning outcome, the authors assessed student performance on the quizzes. To measure student performance, the structure of the observed learning outcome (SOLO) taxonomy [21] was used. This taxonomy is a research-based measure that is extensively used in educational research. Imrie [22] showed that SOLO is a widely applicable framework for judging the structure of essays, answers to technical questions, medical diagnoses, and open-ended problems. In the SOLO taxonomy, there are five structural levels of observed learning outcome. 1) Prestructural: The task is started, but the learner is distracted or misled. The use of irrelevant information provides no correct knowledge about the question. There is no meaningful response. 2) Unistructural: The learner focuses on the relevant domain and concentrates on one aspect. The answer contains one correct feature or item of information. 3) Multistructural: The learner picks up more and more relevant or correct information but does not integrate it. The answer is list-like, containing a number of unconnected items. 4) Relational: The learner integrates the parts with each other so that the whole has a coherent structure and meaning. The answer relates components to make a case or logical whole. 5) Extended Abstract: The learner generalizes the structure to take in new and more abstract features, representing a new and higher mode of operation. The answer not only relates components to make a logical whole, but also contains a connection to a related area of knowledge beyond the demand of the question.

Fig. 4.

Performance Results.

The quizzes were marked by two different teachers who taught the course previously and who were familiar with the SOLO-scoring scheme. For cases in which the scores did not match, the evaluators were asked to check their scoring without being told of each other’s score. If, after the second round, the two scores still did not match, the case was discarded. Less than 5% of the cases were dismissed. IV. RESULTS AND DISCUSSION The data confirmed both hypotheses. The DIL Web site was found to be superior to the DPL Web site in terms of both the learning process and the learning outcome. The process evaluation results (see Table I) offer a clear indication that the students perceived the DIL environment as providing a more active and explorative learning process than the DPL environment did, as postulated in Hypothesis 1. Furthermore, in addition to (or maybe because of) being more interactive and more explorative, the DIL environment was also perceived by the students to be more effective in supporting their learning methods and more enjoyable to use. The between-subjects and within-subjects comparisons of the performance scores also indicate that the users of the DIL Web sites achieved a higher level of learning, as stipulated in Hypothesis 2. The mean SOLO scores obtained by the two groups for the five different quizzes (including the pretreatment quiz) are presented in Fig. 4.

KHALIFA AND LAM: WEB-BASED LEARNING: EFFECTS ON LEARNING PROCESS AND OUTCOME

355

TABLE II BETWEEN-GROUP COMPARISONS OF SOLO SCORES

TABLE III WITHIN-GROUP COMPARISONS OF SOLO SCORES

The between-group comparison indicates that the performance of the group that used the DIL Web site to prepare for the quiz always achieved a significantly higher performance (see Table II). For the pretreatment evaluation (quiz 1), both groups were given hard copies of the lecture notes and the presentation slides. The SOLO scores obtained by the two groups for quiz 1 were not significantly different, as indicated by a -test at the 5% significance level. Hence, differences in the results of the subsequent quizzes could not be attributed to differences between the individuals forming the groups. For the treatment quizzes, whatever group used the DIL environment obtained a significantly higher SOLO score than the other group. These results represent a strong confirmation of Hypothesis 2. The within-group comparisons are consistent with the between-group comparisons (see Table III). The average SOLO scores for the quizzes associated with the DIL site are significantly higher than those associated with the DPL site for Group 1, Group 2, and all students combined. It is important to notice that the overall average SOLO score for the DIL users is 3.76, implying that the users have achieved a learning outcome that is multistructural (score of 3) and almost relational (score of 4). The authors interpret these results as an indication that the explicit representation of relationships between concepts with the hyperlinks has helped the learners to integrate separate concepts into a coherent whole. The overall average SOLO score for the DPL users, on the other hand, is only 2.95, indicating that their answers were list-like without connections between the different items. The results of the empirical study provided strong evidence of the superiority of the DIL environment over the DPL environment in terms of both the learning process and the learning outcome. The verification of the hypotheses with the relatively small sample size is indicative of the strong statistical significance of the results. The small sample size remains, however, a limitation in terms of the generalization of the results. It is also important to emphasize that in this study, the authors only considered the basic form of the DIL environment, which supported

constructivist learning but not collaborative learning (no groupware capabilities). The authors expect the DIL environments that enable the learners to interact not only with the learning material but also with the instructor and other learners to be even more effective. Instructors who wonder whether it is worthwhile to spend the effort and time to convert linear material into a hypertext structure have some answers in the results of this study. In future research, the authors will study the causal relationship between the learning process and the learning outcome. They will also examine the effects of other features, e.g., collaboration and live broadcasting, and compare the remaining categories of the proposed Web learning taxonomy. REFERENCES [1] M. Alavi and D. E. Leidner, “Research commentary: Technology-mediated learning—A call for greater depth and breadth of research,” Inf. Syst. Res., vol. 12, no. 1, pp. 1–10, 2001. [2] D. E. Leidner and S. L. Jarvenpaa, “The use of information technology to enhance management school education: A theoretical view,” MIS Quart., Sept. 1995. [3] D. E. Leidner and M. Fuller, “Improving student learning of conceptual information: GSS supported collaborative learning vs. individual constructive learning,” Decision Support Syst., vol. 20, no. 2, pp. 149–163, June 1997. [4] J. Conklin, “Hypertext: A survey and introduction,” IEEE Computer, vol. 20, no. 9, 1987. [5] M. Khalifa and R. Kwok, “Remote learning technologies: Effectiveness of hypertext and GSS,” Decision Support Syst., vol. 26, no. 3, 1999. [6] C. Chou, “Developing hypertext-based learning courseware for computer networks: The macro and micro stages,” IEEE Trans. Educ., vol. 42, pp. 39–44, Feb. 1999. [7] V. Wulf, “Exploration environments: Supporting users to learn groupware functions,” Interacting Comput., vol. 13, no. 2, pp. 265–299, Dec. 2000. [8] K. H. Lim and I. Benbasat, “An empirical study of computer system learning: Comparison of co-discovery and self-discovery methods,” Inf. Syst. Res., vol. 8, no. 3, pp. 254–272, Sept. 1997. [9] D. H. Jonassen, “Semantic networking as cognitive tools,” in Cognitive Tools for Learning, P. Kommers, D. H. Jonassen, and J. T. Mayes, Eds, Berlin: Springer-Verlag, 1992, pp. 19–21. [10] R. McAleese, “Concepts as hypertext nodes: The ability to learn while navigating through hypertext nets,” in Designing Hypermedia for Learning, D. H. Jonassen and H. Mandl, Eds, Berlin: Springer-Verlag, 1990, pp. 97–115.

356

[11] J. E. McDonald, K. R. Paap, and D. R. McDonald, “Hypertext perspectives: Using pathfinder to build hypertext systems,” in Studies in Knowledge Organization, R. W. Schvanenveldt, Ed. Norwood, NJ: Pathfinder Associative Networks, 1990. [12] D. H. Jonassen, “Semantic network elicitation: Tools for structuring hypertext,” in Hypertext: State of the Art, R. McAleese and C. Green, Eds, Oxford, U.K.: Intellect, 1990, pp. 142–152. [13] S. S. Liaw and H. M. Huang, “Enhancing interactivity in Web-based instruction: A review of the literature,” Educ. Technol., pp. 41–45, May–June 2000. [14] M. Tremayne and S. Dunwoody, “Interactivity, information processing, and learning on the World Wide Web,” Sci. Commun., vol. 23, no. 2, pp. 111–134, Dec. 2001. [15] J. Y. Mao and I. Benbasat, “Contextualized access to knowledge: Theoretical perspectives and a process-tracing study,” Inf. Syst. J., vol. 8, 1998. [16] G. Kearsley, “Authoring considerations for hypertext,” Educ. Technol., Nov. 1988. [17] R. J. Spiro, P. J. Feltovich, M. J. Jacobson, and R. L. Coulson, “Cognitive flexibility, constructivism, and hypertext: Random access instruction for advanced knowledge acquisition in ill-structured domains,” Educ. Technol., May 1991. [18] J. Turns, C. J. Atman, and R. Adams, “Concept maps for engineering education: A cognitively motivated tool supporting varied assessment functions,” IEEE Trans. Educ., vol. 43, pp. 164–173, May 2000. [19] D. H. Jonassen, “Effects of semantically structured hypertext knowledge bases on users’ knowledge structures,” in Hypertext, A Psychology Perspective, C. McKnight, A. Dillon, and J. Richardson, Eds, Chichester, U.K.: Ellis Horwood, 1993, pp. 153–168.

IEEE TRANSACTIONS ON EDUCATION, VOL. 45, NO. 4, NOVEMBER 2002

[20] J. Wilson, “Concept maps about chemical equilibrium and students’ achievement scores,” Res. Sci. Educ., vol. 26, no. 2, 1996. [21] J. B. Biggs and K. F. Collis, Evaluating the Quality of Learning: The SOLO Taxonomy. White Plains, NY: Longman, 1982. [22] B. W. Imrie, “Assessment for learning: Quality and taxonomies,” Assessment Evaluation Higher Educ., vol. 20, no. 2, 1995.

Mohamed Khalifa received the M.A. degree in decision sciences and the Ph.D. degree in information systems from the Wharton Business School, University of Pennsylvania, Philadelphia. Currently, he is an Associate Professor and Director of the Asia Center for Electronic Business at the Information Systems Department, City University of Hong Kong, Kowloon, Hong Kong, Special Administrative Region (SAR) of the P.R.C. He has published several books and more than 50 refereed articles in journals, such as Communications of the ACM, Decision Support Systems, Data Base, and Information and Management, as well as the IEEE TRANSACTIONS ON ENGINEERING MANAGEMENT and the IEEE TRANSACTIONS ON SYSTEMS, MAN, and CYBERNETICS.

Rinky Lam is currently pursuing the M.Phil. degree in information systems at the City University of Hong Kong, Kowloon, Hong Kong, Special Administrative Region (SAR) of the P.R.C. Her research interests include knowledge management and e-learning.