Chu, H.-C. (2014). Potential Negative Effects of Mobile Learning on Students’ Learning Achievement and Cognitive Load—A Format Assessment Perspective. Educational Technology & Society, 17 (1), 332–344.
Potential Negative Effects of Mobile Learning on Students’ Learning Achievement and Cognitive Load—A Format Assessment Perspective Hui-Chun Chu Department of Computer Science and Information Management, Soochow University, 56, Sec. 1, Kui-Yang St., Taipei 100, Taiwan //
[email protected] (Submitted June 12, 2012; Revised March 27, 2013; Accepted May 11, 2013) ABSTRACT Owing to the advancement of mobile and wireless communication technologies, an increasing number of mobile learning studies have been conducted in recent years. In a mobile learning environment, students are able to learn indoors and outdoors with access to online resources at any time. However, whether or not new learning scenarios that combine both real-world contexts and digital-world resources are beneficial to the students has been questioned. Moreover, it is also interesting to probe whether the existing e-learning strategies are effective when situated in those mobile learning scenarios. In this study, an in-field activity on an indigenous culture course of an elementary school with a formative assessment-based learning strategy was conducted to investigate the possible negative effects of mobile learning by analyzing the students’ cognitive load and learning achievement. It is interesting to find that, without proper treatment, the performance of students using those existing online learning strategies, known to be “effective,” might be disappointing or may even negatively affect the students’ learning achievements. Furthermore, the negative effects could be due to the heavy cognitive load caused by an improper learning design. Such findings offer good references for those who intend to design and conduct mobile learning activities.
Keywords Mobile learning, Ubiquitous learning, Formative assessment, Cognitive load, Technology-enhanced learning
Background and objectives In the past decade, the advancement of mobile technology has offered an opportunity to provide learning supports at anytime and anywhere. Various studies have reported the benefits of applying mobile technologies (e.g., personal digital assistants, smart phones, or portable computers) to the learning activities of various courses, including science, social science, and language courses (Hoppe, Joiner, Milrad, & Sharples, 2003; Liu, Wang, Liang, Chan, Ko, & Yang, 2003; Hsu, Hwang, & Chang, 2010; Sharples, Taylor, & Vavoula, 2007; Wong & Looi, 2011; Wong, Chin, Tan, & Liu, 2010; Zhang et al., 2010). For example, Chen, Kao, and Sheu (2003), based on scaffolding theory, developed a bird-watching mobile learning system that enables learners to observe birds outdoors, while gaining information about the birds via mobile devices. Moreover, teachers can send questions to students by means of these devices. Later, Chen, Chang, and Wang (2008) also employed the scaffolding theory to conduct mobile learning activities. In the meantime, Chu, Hwang, Huang, and Wu (2008) conducted several outdoor learning activities in a butterfly ecology garden by integrating mobile learning environments with electronic library facilities. Recently, Hwang, Wu and Ke (2011) proposed an interactive concept-mapping approach for conducting mobile learning activities for natural science courses. During the process of mobile learning, students interact with authentic contexts through words, pictures, sounds, animations, and images, all provided by the mobile devices. Compared with traditional instruction or conventional web-based learning, mobile or ubiquitous learning scenarios could be much more complex for learners since they need to simultaneously deal with learning materials in both the real world and the digital world. In the past decade, many studies have shown that proper mobile learning strategies or tools need to be considered to help the students acquire the expected learning achievements in real-world environments (Chu, Hwang, & Tsai, 2010; Chen, Chang, & Wang, 2008; Chen, Kao, & Sheu, 2003; Liu, Lin, Tsai, & Paas, 2012; Looi, Zhang, So, Chen & Wong, 2010). Most of those studies have focused on whether the students’ learning performance can be improved by the mobile learning system or by using proper traditional instruction strategies (Hwang, Wu, Tseng, & Huang, 2011; Wu, Hwang, Su, & Huang, 2012). Only a few studies have been conducted to investigate the impacts of mobile learning on students’ cognitive load (e.g., Liu et al., 2012), which might be an important indicator to evaluate the effectiveness of applying well-recognized web-based learning strategies to mobile learning activities (Ozcinar, 2009).
ISSN 1436-4522 (online) and 1176-3647 (print). © International Forum of Educational Technology & Society (IFETS). The authors and the forum jointly retain the copyright of the articles. Permission to make digital or hard copies of part or all of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear the full citation on the first page. Copyrights for components of this work owned by others than IFETS must be honoured. Abstracting with credit is permitted. To copy otherwise, to republish, to post on servers, or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from the editors at
[email protected].
332
Cognitive load theory was proposed in the 1980s. It is concerned with the way in which humans’ cognitive architecture deals with learning objects during the learning process or when performing a particular task. Human cognitive architecture is composed of working short-term and long-term memory, in which all conscious cognitive processing occurs; moreover, cognitive load has been recognized as being closely related to the demand of working memory resources during the learning process (Paas, Renkl, & Sweller, 2003). Baddeley and Hitch (1974) found that human working memory can handle only a very limited number of novel interacting elements, possibly no more than two or three; that is, improper design of learning content and instructional strategies is likely to increase the cognitive load of students owing to overloading their working memory. In this study, the students’ cognitive loads as well as their learning outcomes of an in-field mobile learning activity are investigated. In-field mobile learning refers to the learning activities conducted in the field using mobile devices, such as learning activities in museums, ecology parks or historical buildings. The students can benefit as a result of the activities promoting their interest in and motivation regarding the learning subjects when they learn in the physical field rather than learning in the classroom. In an in-field mobile learning activity, students need to observe and interact with the real-world learning targets while at the same time paying attention to the learning guidance or supplementary content provided by the learning system via the mobile device and wireless communications. Consequently, it is likely that the learning burden of individual students will exceed what they are able to cope with, implying that measuring their cognitive load as well as their learning achievement is necessary. In a mobile or ubiquitous learning activity, the learning materials from both the real-world and the digital-world environments could be a noticeable source of mental load, since the amount and complexity of the materials are likely to increase interactions among the task, the students, and the learning materials (Paas & van Merriënboer, 1994; van Merriënboer & Sweller, 2005; Wirth, Kunsting, & Leutner, 2009); moreover, the use of improper learning strategies could increase students’ mental effort as well. Therefore, it is worth investigating both the effectiveness and the cognitive load when conducting learning activities with those well-known learning strategies. Consequently, it has become an important and challenging issue to re-examine whether those effective web-based tutoring or assessment strategies can benefit mobile or ubiquitous learning, in particular, the formative assessment strategy that has been recognized as being one of the most effective methods for web-based learning (Chen & Chen, 2009; Heinrich, Milne, & Moore, 2009; Hung, Lin, & Hwang, 2010; Wang, 2008, 2011). When learning with the formative assessment strategy in the physical mobile environment, students were asked to observe real-world learning targets to answer a set of questions. The characteristic of this approach is that the learning system does not provide correct answers to the students when they fail to correctly answer some questions; instead, only some clues are provided to help them find the answers by themselves. That is, for those students who are inexperienced in learning with such an approach or whose learning progress is not as good as expected, their cognitive loads may be overloaded since they need to look for answers in the physical world and answer the questions repeatedly until the learning system confirms that they give the correct answer to each question three times. Paas and van Merriënboer (1994) have indicated that learners’ cognitive load may increase owing to the characteristics of the task, the subject performing the task, or the interactions between both. Kalyuga (2009) has further indicated that learning strategies aiming to increase germane cognitive load (the cognitive load that helps students learn in a more effective way) might lead to intrinsic cognitive load (the cognitive load that might seriously affect the students’ learning performance) owing to the complexity and difficulty of the learning materials and the knowledge level of the learners. In the in-field mobile learning environment with the formative assessment approach, students’ cognitive load may be high owing to the pressure caused by the limited learning time, the number of questions to be answered, students’ lack of experience in learning with mobile devices, and the need to repeatedly answer the answers until all of the criteria of the learning task are fulfilled. That is, it is quite possible that students have high cognitive load owing to the nature of the formative assessment strategy and the complexity of the in-field mobile learning environment that combines the physical learning materials and the digital learning resources. This is especially so for novices. Therefore, it is important to investigate the potential negative effects of such an in-field mobile learning approach. This study aims to investigate the effect of applying web-based learning strategies that are known to be effective in mobile learning activities. To achieve this objective, a formative assessment-based learning system was developed for use in a mobile learning activity. Such learning strategies have been reported as being effective in web-based learning environments (Wang et al., 2004; Wang, 2008, 2011). Moreover, the learning achievements and cognitive load of the students who participated in the learning activity were measured to evaluate the effects of this approach. 333
Literature review Cognitive load Cognitive load theory is a multidimensional construct that represents the different loads that performing a particular task imposes on the cognitive system (Paas, Tuovinen, Tabbers, & van Gerven, 2003; Sweller, van Merriënboer, & Paas, 1998). Scholars have reported that both the background of the learner and the environmental context could be sources of cognitive load (Paas & van Merriënboer, 1994); moreover, learning difficulty is highly relevant to the increase in cognitive load (Sweller, 1988; Kalyuga, 2009). Researchers have pointed out the existence of three types of cognitive load (Sweller, van Merriënboer, & Paas, 1998): ˙ “Intrinsic cognitive load” refers to the inherent structure and complexity of the instructional materials. It increases when the element of interactivity is added. The element of interactivity depends on the number of different types of information that learners must integrate and absorb at the same time. On the other hand, “intrinsic cognitive load” is relevant to how much information the working memory needs to deal with at the same time. ˙ “Extraneous cognitive load” or “ineffective load” refers to the degree to which a task influences learning. It is related to the poor design of the instructional strategy, which does not take the necessary instructional variables into account. The worse the instructional design, the more difficult the learning task will become. ˙ “Germane cognitive load” or “effective load” refers to an instructional design that facilitates the learning process by properly taking the necessary instructional variables into consideration. Sweller et al. (1998, p. 34) indicated that the germane load is to “separate useful, learning-relevant demands on working memory from irrelevant and wasteful forms of cognitive processing.” Therefore, germane load is relevant to the learning activities designed to improve students’ learning performance or motivation by reducing their extraneous cognitive load. Although such activities increase total cognitive load, they are helpful to learning if the total cognitive load does not exceed the students’ working memory capacity. Therefore, germane load needs to be carefully applied in designing learning activities (Kalyuga, 2009). Consequently, learning can be effective if the cognitive loads imposed by the learning materials and by the way those materials are presented are well managed. Paas, Merriënboer, and Adam (1994) have further indicated that those three types of cognitive load can be summed up to “mental load” and “mental effort” since “extraneous cognitive load” and “germane cognitive load” are in fact two extremes that refer to the design of learning strategies. Mental load refers to the interaction among the task, subject characteristics and learning materials; it can be considered to be the load which is imposed by the task or environmental demands. Therefore, this task-centered dimension represents “intrinsic cognitive load.” Mental effort is a human-centered dimension that is concerned with the cognitive resources (i.e., the amount of resources) actually allocated to a task. Mental effort reflects the amount of learning targets and materials with which the learner is engaged, implying that it is likely to be affected by the design of the learning strategies; that is, it represents the extraneous and the germane cognitive loads (Paas, van Merriënboer, & Adam, 1994; Zheng, 2009). In past decades, the cognitive load theory has been applied to many educational applications, including traditional instruction and technology-enhanced learning activities (Chang, Hsu, & Yu, 2011; Hollender, Hofmann, Deneke, & Schmitz, 2010; Hung et al., 2010; Liu et al., 2012). In recent years, owing to the popularity of mobile devices, mobile learning has become an important technology-enhanced learning approach; therefore, it is worth investigating the cognitive load issue in mobile learning settings. In this study, an experiment is performed by conducting an infield activity to compare the effectiveness of mobile learning and traditional instruction approaches from the aspects of cognitive load.
Formative assessment Formative assessment has been recognized by educators and researchers as an important element in conducting learning activities for improving student learning effectiveness (Bell & Cowie, 2001; Hargreaves, 2008; Hwang & Chang, 2011). It is a form of assessment integrated into the interaction between teachers and students for offering feedback (Oosterhof, Conrad, & Ely, 2008; Vonderwell, Liang, & Alderman, 2007). Researchers have indicated that 334
the provision of feedback can benefit students in improving their learning achievement and motivation (Johnson, Perry, & Shamir, 2010; Panjaburee, Hwang, Triampo, & Shih, 2010). Owing to the advancement and popularity of computer and communication technologies, many researchers have been devoted to the development of web-based learning and assessment systems (Hwang, Tseng, & Hwang, 2008; Ho, Yin, Hwang, Shyu, & Yean, 2009). Meanwhile, formative assessments have played an important role in such web-based environments. Many researchers have indicated that the learning effectiveness of students can be improved by including the formative assessment strategy in the design of the web-learning environments (Gardner, Sheridan, & White, 2002; Gikandi, Morrow, & Davis, 2011; Wang, 2008, 2011; Wilson, Boyd, Chen, & Jamal, 2011). That is, formative assessment has been recognized as being beneficial to student learning in both traditional and web-based learning environments. Therefore, in this study, formative assessment was chosen as a representative strategy of technology-enhanced learning as opposed to conventional instructional strategies, which adopt summative assessment.
Research design The in-field learning activity is related to the “indigenous culture” unit of the social studies course of an elementary school, and is part of the existing curriculum of the target school. The objective of the social studies course is to introduce the indigenous language, culture, and history of Taiwan. The eighteen-week course consists of in-class instruction and a field trip. Most weeks, the students receive in-class instruction from the teacher. In the middle of the semester, they visit the target ancient building for field studies. In this study, the supplementary materials and the learning tasks followed the original learning design of the course.
Participants The participants in this experiment were two classes of fifth-grade students of an elementary school in Taiwan. One class (thirty-three students) was assigned to be the experimental group, and the other (thirty-one students) was the control group. The students were assigned to these two classes by S-shaped distribution at the beginning of the academic year. From this perspective, these two classes were viewed as having similar distributions of students in terms of their academic backgrounds.
Experiment procedure As has been mentioned above, cognitive load theory (CLT; Paas & van Merriënboer, 1994; Sweller, van Merriënboer, & Paas, 1998) is related to the development of instructional methods to stimulate learners’ ability of applying acquired knowledge or skills to new contexts via efficiently using their limited cognitive processing capacity. Mayer and Moreno (2003) indicated that, to foster students’ ability of applying learned knowledge or skills to new situations, it is important to consider several aspects of learning design, including the presentation of learning materials, the structure of the materials, and linkage of the new knowledge or skills to the existing knowledge. In this study, a well-known e-learning strategy that has been shown to be effective by several previous studies (i.e., the formative assessment-based learning strategy) was employed for conducting mobile learning activities, in which the students were asked to observe a set of learning targets in the real-world environment based on the questions raised by the learning system. When the students failed to correctly answer some questions, the learning system did not provide the correct answers to them; instead, some clues were given to help them find the answers by themselves. We aimed to examine whether the students’ cognitive processing capacity was overloaded when they needed to face both the learning targets in the real world and the questions and clues provided by the learning systems from two aspects, mental load and mental effort. The former is related to the load imposed by the task or environmental demands, while the latter is related to the cognitive resources (i.e., the amount of resources) actually allocated to a task. In this study, the two groups of students were situated in the same learning environment and learning content with different learning modes, that is, the conventional mobile learning mode that engaged the students in observing the learning targets and finding the answers to the learning sheet by themselves, and the traditional instructional mode in 335
which the teachers guided the students to observe the learning targets and presented the relevant materials of individual learning targets to them. Accordingly, we compared the loads imposed by the learning tasks and the learning achievements of the two groups. Figure 1 shows the experimental procedure of this study. Before the field trip, an orientation was given to introduce the learning environment and the learning tasks. During the in-field learning process, the students in the experimental group were guided by a mobile device using the formative assessment-based mobile learning (FAML) strategy after receiving some training in the use of the mobile learning system. On the other hand, the students in the control group learned with the traditional in-field instruction mode; that is, they were instructed by the teacher when observing the learning targets in the field. Moreover, they received advice from the teacher when encountering problems completing the learning tasks. During the learning process, the treatments of the experimental and control groups, including the learning sheet, learning content, instructor, and learning time, were equivalent.
Figure 1. Experiment procedure Before and after participating in the experiment, the learning achievements of the two groups of students in the social studies course were compared based on their pre-test and post-test scores. In this study, the results of the most recent mid-term examination of the social studies course were taken as the pre-test results. After participating in the experiment, the two groups of students took a post-test on the topic learned in this tour-based learning activity. Moreover, the cognitive loads of the two groups of students were analyzed and compared as well.
Measuring tools The measuring tools in this study included learning achievement test sheets and questionnaires for evaluation of the learning situation. A pre-test and a post-test were developed to evaluate the learning effectiveness of the students. The pre-test was developed based on what the students had learned in the social studies course before participating in the experiment; that is, it was designed for testing the students’ prior knowledge by excluding the part of the content learned during the experiment process. It was composed of 30 multiple-choice items (60%), 10 true-or-false (20%) questions, and 3 fill-in-the-blank items (20%), with a full score of 100. On the other hand, the post-test was developed based on what they had learned in the experiment; that is, it was designed for evaluating their learning outcomes of the instructional process in this study. The post-test consisted of 40 multiple-choice items with a full score of 100. Both the pre-test and post-test were designed by the teacher who taught the indigenous culture course to the two groups of students and were evaluated by two social-science educators for expert validity. 336
In addition, the cognitive load survey proposed by Sweller et al. (1998) was adopted for measuring the cognitive load of individual students. The cognitive load survey consists of four items on a seven-point scale: two items for mental load (intrinsic load) and two for mental effort (extraneous and germane load). Mental load is relevant to the number of interacting information elements and the extent to which these elements interact, while mental effort is caused when learning activities and/or materials encourage higher-order thinking and challenge the learner at an appropriate level within what Vygotsky (1978) called their zone of proximal development. In this study, the Cronbach’s α value of the cognitive load form was 0.92. For the mental effort and the mental load dimensions, the Cronbach’s α values were 0.86 and 0.85, respectively. These values show high reliability.
Learning environment The learning environment of this study is Chin-An temple in Tainan County of Taiwan, as shown in Figure 2. The teaching content consisted of five main parts, including the temple square, front hall, main hall, and side temples. This learning activity was designed to include twelve main units, such as the Holland wells, the stone-carved lions at the main gate and the ridge of the roof, each of which has its own architectural characteristics and historical story.
Figure 2. Scenario of the mobile learning activity A mobile learning system using the FAML strategy was developed to conduct the local cultural learning activity with wireless networks and mobile devices, as shown in Figure 3. This system is implemented with MSSQL, ASP.NET, and IIS. There are several management functions developed for teachers, including user profile management, subject materials management, item bank management, and learning portfolio management.
Figure 3. Framework of the Mobile Learning System 337
Formative assessment-based mobile learning system The FAML strategy originated from the idea of the web-based assessment and test analysis system (WATA), which is an evaluative strategy with three characteristics, including repeated answering, non-answer provision, and immediate feedback (Wang, Wang, Wang, Huang, & Chen, 2004). These characteristics allow students to repeatedly participate in the “practicing, reflecting, and revising” process. That is, students will practise and observe more, gain feedback immediately (reflect), and revise their answers (Wang et al., 2004). From the previous applications in web-based learning environments, WATA has been recognized as being an effective formative assessment strategy. It assists learners in identifying their learning flaws to trigger their motivation for active learning, enabling them to become familiar with the learning content (Wang, 2008, 2010; Wang, Wang, & Huang, 2008). Therefore, it is interesting and appropriate to examine the effect of applying such a strategy on the learning achievements and cognitive loads of students in a mobile learning environment.
Figure 4. The formative assessment-based mobile learning strategy As shown in Figure 4, when students approach target positions, the mobile learning system will guide them to observe the target learning objects and interact with them based on the FAML mechanism. Once the students submit their answers, the system provides them with hints or supplementary materials, but the validity of the answers is not confirmed. That is, the students need to find out whether their answers are correct or not by themselves. For each 338
learning target, there are four or five test items to be answered. The students need to pass all of the items to complete the learning task of the target. The interface of the mobile learning system is shown in Figure 5, and includes a material-browsing interface (the left snapshot) and an assessment interface (the right snapshot). After the students browse the learning materials, the learning system randomly selects test items from the item pool of that unit. Whether the answer submitted by the students is correct or not, no feedback concerning the correctness of the answer is given; instead, only hints or supplemental materials are presented. Once an item has been successively and correctly answered three times, it is removed from the item pool; however, once an incorrect answer is given, the accumulative total of correctness is reset. When the item pool of a unit is empty (i.e., all of the questions in that pool have been successively and correctly answered three times), the student is considered to have passed that unit. In this study, each student needed to pass twelve units of the indigenous culture course.
A question about the ancient well
Description of the background stories of an ancient well
Figure 5. Interface of the mobile learning system for the indigenous culture course
Results and analysis Learning achievement Before the experiment, the most recent mid-term test scores of the two groups in the social studies course were taken as the result of the pre-test, to ensure that the two groups of students had equal abilities in this subject before the learning activity. The mean and standard deviation of the pre-test for the experimental group (M = 88.45, SD = 16.56) and for the control group (M = 89.48, SD = 16.01) revealed that there was no significant difference between the two groups with t(64) = 0.257 and p > .05. That is, the two groups of students had statistically equivalent abilities before learning the subject unit. After participating in the learning activity, the two groups of students took a post-test. The t-test result, t(64) = 3.34 and p < .01, showed that the learning achievement of the control group (M = 64.60, SD = 13.37) was better than that of the experimental group (M = 52.88, SD = 14.61), as shown in Table 1. The Levene’s test for equality of variances showed that the groups were homogenous with F(1, 64) = 0.882 and p = .351. Moreover, the Cohen’s d value of 0.84 indicates a large effect size (Cohen, 1988). That is, a significant difference was found in the learning achievements of the two groups of students. Such a finding (i.e., the control group had significantly better learning achievements than the experimental group) was completely different from what the researchers had expected. This surprising and interesting result indicated that the formative-assessment learning strategy for mobile learning had an unfavorable effect on the learning 339
achievements of the students in comparison with that for the traditional approach. Moreover, the finding is quite different from those reported by other researchers who have applied the formative assessment strategy to web-based learning (Costa, Mullan, Kothe, & Butow, 2010; Crisp & Ward, 2008; Pachler, Daly, Mor, & Mellar, 2010). Table 1. The t-test results for the post-test scores of the two groups of students Group N Mean SD t Control group 31 64.60 13.37 3.34*** Experimental group 33 52.88 14.61 *** p < .01
Cohen’s d 0.84
Cognitive load To analyze the factors that might affect the students’ learning achievements, the cognitive loads of the two groups of students were measured by employing the cognitive load survey developed by Sweller et al. (1998) using an independent t-test. The result is shown in Table 2. The overall evaluation form of cognitive load and the level of mental effort (extraneous cognitive load and germane cognitive load) showed no significant difference, but the level of mental load (intrinsic load) did show a significant difference, with the degree of mental load of the experimental group being significantly higher than that of the control group. Cognitive-load theory (Kirschner, 2002; Paas & van Merriënboer, 1994; Sweller et al., 1998) assumes that the working memory architecture (Baddeley & Logie, 1999) can be viewed as a system of limited capacity; therefore, only a limited number of elements can be handled at the same time. If the cognitive load of a task (or various tasks) exceeds the limits of working memory, then performance is seriously affected (DeStefano & LeFevre, 2007).We found no significant difference between the mental effort of the experimental group (M = 4.48, SD = 2.10) and that of the control group (M = 4.64, SD = 1.60). This implies that the instructional variables had been well considered in this formative assessment-based mobile learning strategy, which revealed a good balance between controlling the germane cognitive load and the extraneous cognitive load of the students. Therefore, the use of the formative assessment-based method (i.e., WATA) for mobile learning is an acceptable instructional design. On the other hand, the mental load of the experimental group (M = 2.62, SD = 1.60) is significantly higher than that of the control group (M = 1.58, SD = 1.11) with t(64) = −3.08 and p < .01. That is, the average intrinsic cognitive load (task-centered dimension) for the students in the experimental group was higher than expected, which might have led to negative effects on their learning achievements. Therefore, the researchers re-examined the students’ learning portfolios and found that the experimental group students needed to answer at least twelve questions in each subject unit. Even for those students who always gave correct answers, they need to correctly answer each question three times for passing the exam. During the field trip, the students need to answer forty-eight questions and might receive three hints for guiding them to find the answer of each question via observing the real-world objects. This implies that they might be asked to observe the real-world learning objects and search for the digital-world resources at least one hundred forty-four times. Because it takes much more time to seek the relevant learning materials in a mobile learning environment that combines both real-world and digital-world resources than in a pure web-based learning environment, the excessive amount of learning content (questions or test items for guiding the students to observe and learn) could be the main reason for the excessive mental load and this unexpected result. Table 2. The t-test results for the cognitive load of the two groups of students Cognitive Load Group N Mean SD Mental effort Control 31 4.64 2.10 (Extraneous load and germane load) Experimental 33 4.48 1.60 Mental load Control 31 1.58 1.11 (Intrinsic load) Experimental 33 2.62 1.60 ** p < .01
t 0.33 −3.08**
340
Conclusions and discussion This study explores the effect of online learning strategies on a mobile learning environment that combines digital learning resources and real-world learning contexts. A mobile learning activity was conducted with a formative assessment-based learning strategy; moreover, we investigated the cognitive loads and learning achievements of the students to examine the effect of the learning strategy on such an innovative and complex learning scenario. The results showed that the students in the experimental group did not achieve as much as those in the traditional group. This finding is quite different from what has been previously reported (Chu, Hwang, & Tsai, 2010; Chen et al., 2010; Hwang & Chang, 2011; Hwang et al., 2009; Wang, 2008). Further analyses revealed that one of the reasons why students who learned in the real-world scenario with mobile devices obtained unfavorable learning achievements was their high cognitive loads. From the t-test result of the cognitive load, it was found that high mental load was the main factor causing the negative effect on the learning achievements in the experimental group. This implies that the use of the formative assessment strategy was not the main reason for this unexpected result, since mental load refers to the overloading of the working memory of students. Because the students were repeatedly asked to find learning materials from both the real-world and the digital world environments in a limited period of time and there were many learning tasks to be completed, students’ working memories were likely to be overloaded. From the observations of the students’ in-field learning behaviors, it was found that most were eager to complete the learning tasks. Because time was limited, students tended to answer the questions in a hurry and therefore were likely to give incorrect answers. When they did, they were asked to make additional observations and answer the questions again. On the other hand, the feedback from the learning system was instant and frequent, which could make the students become anxious since they needed to read the hints or guidance repeatedly and, most of the time, the feedback might relate to an additional observation to be made in the present learning task. Adding to their pressure, some of their peers might have already proceeded to the next learning task. Therefore, most of the students hurried to repeatedly observe the learning targets and answer the questions. In that case, they did not make deep reflections or thorough observations before going on to the next learning task, which not only caused repeated failures in the tasks, but also led to high cognitive loads. It is also interesting to compare the result of this study with those of previous studies (e.g., Costa et al., 2010; Crisp & Ward, 2008; Pachler et al., 2010; Wang, 2008, 2010; Wang, Wang, & Huang, 2008) that reported the effectiveness of the formative assessment-based learning strategy in web-based learning environments. From this case, we can understand that using mobile devices to learn in an authentic learning environment is not always successful, even when adopting those learning strategies that have been reported as being effective in the well-recognized web-based learning environments. Therefore, it is important to investigate whether the learning strategies that have been claimed to be effective in traditional computer systems or web-based learning environments can work well for mobile or ubiquitous learning. Moreover, it is also important to develop new strategies or modify existing strategies for learning design by taking the special features of mobile and ubiquitous learning (i.e., interacting with real-world contexts with personalized supports or hints from the digital system) into account. This conclusion conforms to what has been reported by Chu, Hwang, Tsai, and Tseng (2010), who claim that developing proper learning guidance procedures or tools will help students improve their learning achievements in a mobile and ubiquitous learning environment. That is, with a proper learning design, the effect of mobile and ubiquitous learning can be much better than that of the traditional approach. These findings also show that decreasing students’ cognitive load could be another important issue for mobile learning. Most web-based learning studies have devoted their efforts to building virtual reality tools or learning materials to help the students construct knowledge (Verhoeven, Schnotz, & Paas, 2009; Wang et al., 2004; Wang et al., 2008). Nevertheless, in mobile learning environments, the students are already situated in a more complex learning situation that combines real-world and digital-world resources; therefore, it is a challenging issue to propose new learning strategies or rethink the existing approaches such that the students can learn with reasonable cognitive loads. To sum up, the findings of this study are helpful to those researchers who plan to develop mobile learning systems or conduct mobile learning activities in the future. For those researchers who are interested in this study, the authors would be happy to provide the experimental data upon request. In the meantime, since this study focuses mainly on a fifth-grade social studies course in an elementary school, it is worth investigating the effects of similar approaches on 341
other subjects and different grades of students. Currently, we are working on extending this study by conducting various mobile learning activities for some natural science courses in several elementary schools.
Acknowledgements The author would like to thank Ms. Yen-Ru Shih and Ms. Hsun-Fang Chang for their assistance in developing the learning materials and conducting the experiment. This study was supported in part by the National Science Council of the Republic of China under contract number NSC 102-2511-S-031-001 and NSC 101-2628-S-031-001-MY2.
References Baddeley, A. D., & Hitch, G. J. (1974). Working memory. In G. A. Bower (Ed.), Recent advances in learning and motivation (Vol. 8, pp. 47–90). New York, NY: Academic Press. Baddeley, A. D., & Logie, R. H. (1999). Working memory: The multiple-component model. In A. Miyake & P. Shah (Eds.), Models of working memory: Mechanisms of active maintenance and executive control (pp. 28–61). Cambridge, UK: Cambridge University Press. Bell, B., & Cowie, B. (2001). The characteristics of formative assessment in science education. Science Education, 85, 536–553. Chang, T.-W., Hsu, J.-M., & Yu, P.-T. (2011). A comparison of single- and dual-screen environment in programming language: Cognitive loads and learning effects. Educational Technology & Society, 14(2), 188–200. Chen, C. M., & Chen, M. C. (2009). Mobile formative assessment tool based on data mining techniques for supporting web-based learning. Computers & Education, 52(1), 256–273. Chen, G. D., Chang, C. K., & Wang, C. Y. (2008). Ubiquitous learning website: Scaffold learners by mobile devices with information-aware techniques. Computers & Education, 50, 77–90. Chen, Y. S., Kao, T. C., & Sheu, J. P. (2003). A mobile learning system for scaffolding bird watching learning. Journal of Computer Assisted Learning, 19(3), 347–359. Chu, H. C., Hwang, G. J., & Tsai, C. C. (2010). A knowledge engineering approach to developing mindtools for context-aware ubiquitous learning. Computers & Education, 54(1), 289–297. Chu, H. C., Hwang, G. J., Huang, S. X., & Wu, T. T. (2008). A knowledge engineering approach to developing e-libraries for mobile learning. Electronic Library, 26(3), 303–317. Chu, H. C., Hwang, G. J., Tsai, C. C., & Tseng, & Judy C. R. (2010). A two-tier test approach to developing location-aware mobile learning systems for natural science courses. Computers & Education, 55(4), 1618–1627. Cohen, J., (1988). Statistical power analysis for the behavioral sciences (2nd ed.). Hillsdale, NJ: Lawrence Erlbaum Associates. Costa, Daniel S. J., Mullan, B. A., Kothe, E. J., & Butow, P. (2010). A web-based formative assessment tool for Masters students: A pilot study. Computers & Education, 54(4), 1248–1253. Crisp, V., & Ward, C. (2008). The development of a formative scenario-based computer assisted assessment tool in psychology for teachers: The PePCAA project. Computers & Education, 50(4), 1509–1526. DeStefano, D. & LeFevre, J-A. (2007). Cognitive load in hypertext reading: A review. Computers in Human Behavior, 23(3), 1616–1641. Gardner, L., Sheridan, D., & White, D. (2002). A web-based learning and assessment system to support flexible education. Journal of Computer Assisted Learning, 18(2), 125–136. Gikandi, J. W., Morrow, D., & Davis, N. E. (2011). Online formative assessment in higher education: A review of the literature. Computers & Education, 57(4), 2333–2351. Hargreaves, E. (2008). Assessment. In G. McCulloch, & D. Crook (Eds.), The Routledge international encyclopedia of education (pp. 37–38). New York: Routledge. Heinrich, E., Milne, J., & Moore, M. (2009). An investigation into e-tool use for formative assignment assessment: Status and recommendations. Educational Technology & Society, 12(4), 176–192. 342
Ho, T. F., Yin, P. Y., Hwang, G. J., Shyu, S. J., & Yean, Y. N. (2009). Multi-objective parallel test-sheet composition using enhanced particle swarm optimization. Educational Technology & Society, 12(4), 193–206. Hollender, N., Hofmann, C., Deneke, M., & Schmitz, B. (2010). Integrating cognitive load theory and concepts of human– computer interaction. Computers in Human Behavior, 26(6), 1278–1288. Hoppe, H. U., Joiner, R., Milrad, M., & Sharples, M. (2003). Guest editorial: Wireless and mobile technologies in education. Journal of Computer Assisted Learning, 19(3), 255–259. Hsu, C. K., Hwang, G. J., & Chang, C. K. (2010). Development of a reading material recommendation system based on a knowledge engineering approach. Computers & Education, 55(1), 76–83. Hung, P. H., Lin, Y. F., & Hwang, G. J. (2010). Formative assessment design for PDA integrated ecology observation. Educational Technology & Society, 13(3), 33–42. Hwang, G. J., & Chang, H. F. (2011). A formative assessment-based mobile learning approach to improving the learning attitudes and achievements of students. Computers & Education, 56(4), 1023–1031. Hwang, G. J., Tseng, Judy. C. R., & Hwang, G. H. (2008). Diagnosing student learning problems based on historical assessment records. Innovations in Education and Teaching International, 45(1), 77–89. Hwang, G. J., Wu, P. H., & Ke, H. R. (2011). An interactive concept map approach to supporting mobile learning activities for natural science courses. Computers & Education, 57(4), 2272–2280. Hwang, G. J., Wu, C. H., Tseng, Judy C. R., & Huang, I. W. (2011). Development of a ubiquitous learning platform based on a real-time help-seeking mechanism. British Journal of Educational Technology, 42(6), 992–1002. Johnson, E. P., Perry, J., & Shamir, H. (2010). Variability in reading ability gains as a function of computer-assisted instruction method of presentation. Computers & Education, 55(1), 209–217. Kalyuga, S. (2009). Cognitive load theory. In S. Kalyuga (Ed.), Managing Cognitive Load in Adaptive Multimedia Learning (pp. 34–57). Hershey, PA: Information Science Reference. doi: 10.4018/ 978-1-60566-048-6 Kirschner, P. (2002). Cognitive load theory: Implications of cognitive load theory on the design of learning. Learning & Instruction, 12, 1–10. Liu, T. C., Lin, Y. C., Tsai, M. J., & Paas, F., (2012). Split-attention and redundancy effects on mobile learning. Computers & Education, 58(1), 172–180. Liu, T. C., Wang, H. Y., Liang, T., Chan, T. K., Ko, H. W., & Yang, J. C. (2003). Wireless and mobile technologies to enhance teaching and learning. Journal of Computer Assisted Learning, 19(3), 371–382. Looi, C.-K., Seow, P., Zhang, B, So, H.-J., Chen W., & Wong, L.-H. (2010). Leveraging mobile technology for sustainable seamless learning: A research agenda. British Journal of Educational Technology, 41(2), 154–169. Mayer, R. E. & Moreno, R. (2003). Nine ways to reduce cognitive load in multimedia learning. Educational Psychologist, 38(1), 43–52. Oosterhof, A., Conrad, R. M., & Ely, D. P. (2008). Assessing learners online. Upper Saddle River, NJ: Pearson. Ozcinar, Z. (2009). The topic of instructional design in research journals: A citation analysis for the years 1980–2008. Australasian Journal of Educational Technology, 25(4), 559–580. Paas, F. & Van Merriënboer, J. J. G. (1994). Instructional control of cognitive load in the training of complex cognitive tasks. Educational Psychology Review, 6, 51–71. Paas, F. Renkl. A., & Sweller, J. (2003), Cognitive load theory and instructional design: Recent developments. Educational Psychologist, 38(1), 1–4. Paas, F., Tuovinen, J. E., Tabbers, H. K., & van Gerven, P. W. M. (2003). Cognitive load measurement as a means to advance cognitive load theory. Educational Psychologist, 38 (1), 63–71. Paas, F., Van Merriënboer, J. J. G., & Adam, J. J. (1994). Measurement of cognitive load in instructional research. Perceptualand Mofor Skills, 79, 419–430. Pachler, N., Daly, C., Mor, Y., & Mellar, H. (2010). Formative e-assessment: Practitioner cases. Computers & Education, 54(3), 715–721. Panjaburee, P., Hwang, G. J., Triampo, W., & Shih, B. Y. (2010). A multi-expert approach for developing testing and diagnostic 343
systems based on the concept effect model. Computers & Education, 55, 527–540. Sharples, M., Taylor, J., & Vavoula, G. (2007). A theory of learning for the mobile age. In R. Andrews and C. Haythornthwaite (eds.), The Sage Handbook of Elearning Research, pp.221–247, London: Sage. Shih, J. L., Chu, H. C., Hwang, G. J., & Kinshuk (2011). An investigation of attitudes of students and teachers about participating in a context-aware ubiquitous learning activity. British Journal of Educational Technology, 42(3), 373–394. Sweller, J. (1988). Cognitive load during problem solving: Effects on learning. Cognitive Science, 12(2), 257–285. Sweller, J., Van Merriënboer, J. J. G., & Paas, F. G. W. C. (1998). Cognitive architecture and instructional design. Educational Psychology Review, 10(3), 251–297. van Merriënboer, J. J. G., & Sweller, J. (2005). Cognitive load theory and complex learning: Recent developments and future directions. Educational Psychology Review, 17(2), 147–177. Verhoeven, L., Schnotz, W., & Paas, F. (2009). Cognitive load in interactive knowledge construction. Learning and Instruction, 19(5), 369–375. Vonderwell, S., Liang, X., & Alderman, K. (2007). Asynchronous discussions and assessment in online learning. Journal of Research on Technology in Education, 39(3), 309–328. Vygotsky, L. (1978). Mind in society: The development of higher psychological process. Cambridge, MA: Harvard University Press. Wang, T. H. (2008). Web-based quiz-game-like formative assessment: Development and evaluation. Computers & Education, 51, 1247–1263. Wang, T. H. (2010). Web-based dynamic assessment: Taking assessment as teaching and learning strategy for improving students’ e-Learning effectiveness. Computers & Education, 54(4), 1157–1166. Wang, T. H. (2011). Developing web-based assessment strategies for facilitating junior high school students to perform selfregulated learning in an e-Learning environment. Computers & Education, 57(2), 1801–1812. Wang, T. H., Wang, K. H., & Huang, S. C. (2008). Designing a web-based assessment environment for improving pre-service teacher assessment literacy. Computers & Education, 51(1), 448–462. Wang, T. H., Wang, K. H., Wang, W. L., Huang, S. C., & Chen, S. Y. (2004). Web-based assessment and test analyses (WATA) system: Development and evaluation. Journal of Computer Assisted Learning, 20(1), 59–71. Wilson, K., Boyd, C., Chen, L., & Jamal, S. (2011). Improving student performance in a first-year geography course: Examining the importance of computer-assisted formative assessment. Computers & Education, 57(2), 1493–1500. Wirth, J., Künsting, J. & Leutner, D. (2009). The impact of goal specificity and goal type on learning outcome and cognitive load. Computers in Human Behavior, 25(2), 299–305. Wong, L. H, & Looi, C. K. (2011). What seams do we remove in mobile-assisted seamless learning? A critical review of the literature. Computers & Education, 57(4), 2364–2381. Wong, L. H, Chin, C. K., Tan, C. L., & Liu, M. (2010). Students’ personal and social meaning making in a Chinese idiom mobile learning environment. Educational Technology & Society, 13(4), 15–26. Wu, P.-H., Hwang, G.-J., Su, L.-H., & Huang, Y.-M. (2012). A context-aware mobile learning system for supporting cognitive apprenticeships in nursing skills training. Educational Technology & Society, 15(1), 223–236. Zhang, B.-H, Looi, C.-K., Wong, L.-H., Seow, P., Chia, G., Chen, W.-L.,…Norris, C. (2010). Deconstructing and reconstructing: Transforming primary science learning via a mobilized curriculum. Computers & Education, 55(4), 1504–1523. Zheng, R. (2009). Cognitive effects of multimedia learning (Premier reference source). Hershey, NY: Information Science Reference.
344
