Journal of Information Technology and Applications Vol. 2, No. 2, pp. 69-79, 2007
The Effect of Computer Simulation Instruction on Student Learning: A Meta-analysis of Studies in Taiwan Yuen-kuang Liao* Department of Education, National Taiwan Normal University
[email protected] Yu-wen Chen Teacher, Ching Shin Elementary School, Taipei
[email protected]
Abstract A meta-analysis was performed to synthesize existing research comparing the effects of computer simulation instruction (CSI) versus traditional instruction (TI) on students’ achievement in Taiwan. Twenty-nine studies were located from four sources, and their quantitative data was transformed into Effect Size (ES). The overall grand mean of the study-weighted ES for all 29 studies was 0.54. The results suggest that CSI is more effective than TI in Taiwan. However, only 1 (reliability of measure) of the 17 variables had significant main effect on mean ES. The results of this study suggest that CSI clearly has a more positive effect on students’ learning than TI. The results also shed light on the debate between Clark and Kozma regarding learning from the media. Keywords: computer simulation, virtual reality, CAI, CAL, achievement, meta-analysis
1.
involvement in the learning process, and facilitated their practice and mastery of concepts and principles; clearly computer simulation helped students to meet their learning objectives or goals. Michael [7] pointed out that simulation programs such as Electronic Workbench, LegoCAD, and Car Builder are helping students learn about events, processes, and activities that either replicate or mimic the real world. According to Michael[7], computer simulation can afford learners numerous advantages. For example, computer simulations can (1) provide the students with the opportunity to engage in activities that may otherwise be unattainable, (2) enhance academic performance and the learning achievement levels of students, and (3) be equally as effective as real-life hands-on laboratory experiences. Chou [8], and Serpell[9] also noted the significantly greater effectiveness of computer simulation instruction as compared to traditional instruction. Slack & Stewartv[10], Johnson & Stewart [11], and Collins & Morrison [12] reported that by using genetics construction kits as part of a strategic computer simulation, undergraduate and high school students learned to “solve” genetics programs and to build accurate and rich mental models of genetic knowledge. However, Parker [13] and Tannehill [14] have found no significant differences between computer simulation instruction and traditional instruction. Hopkins [15], Hummel & Batty [16], and Tylinski [17] even reported an opposite finding: the significantly greater effectiveness of traditional instruction. Two review studies regarding the effectiveness of
Introduction
Computer technology has been widely used in education for more than forty years. More specifically, computer simulation as an instructional technology has been commonly used in education [1], [2]. A study by Heinich, Molenda & Russell [3] reported that computer simulations were extensively used for job training in 95% of the Colleges of Management in the USA. In the colleges using CSI, 1/6 of the faculty and 1/4 of the total instructional time were given to computer-simulation-related activities. Faria [4] also found that more than 1700 business schools in the USA used computer simulation software for instruction; more than 200 different types of software were used for this purpose. Many potential benefits have been claimed for the use of computer simulation in teaching. For instance, Huppert, Yaakobi, & Lazarovvitz [5] have noted that “In computer simulations, students have opportunities to receive supplemental contact with the variables tested in real experiences or dangerous ones. Students can be active during the simulated experiments by identifying the study problem, writing in their notebooks their hypotheses, planning and performing the simulated experiments, gathering results, collecting data in their notebooks, plotting these data back in the computer, and using the data for drawing tables and graphs.” (p.232) Rivers and Vockel [6] also found in their study that computer simulations enhanced students’ active * Corresponding author.
69
Journal of Information Technology and Applications Vol. 2, No. 2, pp. 69-79, 2007 simulation (CS) in Taiwan began with system/tool design. Two earliest studies (i.e., [24], [25]) with regard to CS were published in 1980s. However, most of these studies were published after 1996. The total number of studies counted for this category is 41, and the subject areas of these CS systems are diverse, including Physics, Chemistry, Mathematics, Erath Science, Geography, Statistics, Computer Science, Health Education, Physical Education, Architectures, etc. The earliest empirical study of CS was published in 1993 [26], while more than 80% of these studies were published after 2000. The grade levels of students participated in these studies ranged from elementary to graduate, and the subject areas studied are also wide-ranging.
computer simulation have also been analyzed. A review conducted by Deckkers & Donatti [18] concluded that although simulations were more effective in the development of attitudes than lectures, it appeared that the claims of improved cognitive development and learning retention were not readily supported. Lee’s [19] review used a meta-analytic approach: it collected 19 studies and yielded 51 effect sizes. The study found that the overall mean effect size for academic achievement was .41, meaning about 66% of the students in computer simulation classes outperformed the average students from the control groups. On the other hand, the overall effect size for attitude was -.04, meaning control groups performed slightly better than computer simulation classes. Obviously, the results of these two reviews were inconsistent. It is assumed that these conflicting results may be due to the different sources used by the two review studies or their different methodologies. Computer simulations have been defined in different ways by different researchers. According to Alessi & Trollip [20], simulation is just one type among many of computer assisted instruction (CAI). Lee [19] defined simulation in a broad sense as a computer program which temporarily creates a set of images (items, objects) and connects them through cause-and-effect relations. Thomas & Hooper [21] defined computer-based instructional simulation as a computer program containing a manipulatable model of a real theoretical system. The program then enables the student to change the model from a given state to a specified goal-state by directing it through a number of intermediate states. Thus, the program accepts commands from the user, alters the state of the model, and when appropriate displays the new state (p.498). Virtual reality (VR) is a computer technology which combines computer graphics, computer simulation, and human-computer interfaces [22]. In one sense, VR shares some characteristics of computer simulation, such as the mimicking of real life and user-driven control. However, it is not the intention of this study to discuss the different definitions of computer simulation. For the purposes of the present meta-analysis, studies employing computer simulations or VRs as delivery systems for instruction were considered to be types of computer simulation given a broader definition of the term, and were thus included in the group of studies analyzed.
1.2 Purposes of this Study In spite of the many claims for the potential benefits of using computer simulation in education, the results of research comparing the effects of computer simulation instruction and those of traditional instruction in Taiwan are conflicting. For example, Chao[27], Chuang [28], Huang [29], Lin [30], Nein [31], and Su [32] all reported significant gains for CSI as compared with traditional instruction. On the other side, Chen [33], Jao [26], Tseng [34], and Yu [35] found no significant differences in the effectiveness of using CSI and traditional methods of instruction. Owing to the contradictory evidence provided by existing research in the area, and the fact that very little, if any, thorough quantitative synthesis of computer simulation instruction in Taiwan has been done, the present author thought it important to conduct a meta-analysis in order to clarify the above-mentioned research findings. Moreover, since 2 meta-analyses of CSI have been published in the USA [18],[19], and since this is the first meta-analysis of CSI to be conducted in Taiwan, the synthesis of previous research undertaken here not only examines the accumulated research-based effects of computer simulation on students' learning efficiency in Taiwan, but also provides a comparative view of meta-analyses of CSI in Taiwan and the USA.
2. Procedures The research method used in this study is a meta-analytic approach similar to that suggested by Kulik, Kulik, & Bangert-Drowns[36]. Their approach requires a reviewer to (a) locate studies through objective and replicable searches; (b) code the studies for salient features; (c) describe outcomes on a common scale; and (d) use statistical methods to relate study features to outcomes [36]. Their method differs from that of Glass, McGaw, & Smith [37]: in Kulik et al’s approach each individual study, defined as the set of results from a single publication, was weighted equally with all the other studies, so that the problem
1.1 The Development of Computer Simulation in Taiwan The development of CAI in Taiwan has moved from the development of traditional courseware for mainframe computers to Windows-based CAI, then to multimedia CAI, and finally to web-based CAI (see details in [23]). As part of the process of development of CAI in Taiwan, the development of computer 70
Journal of Information Technology and Applications Vol. 2, No. 2, pp. 69-79, 2007 analysis, outcomes from a variety of different studies using a variety of different instruments had to be expressed on a common scale. The transformation used for this purpose was the one recommended by Glass et al [37] and modified by others (e.g., Hunter, Schmidt, and Jackson [38]). To reduce measurements to a common scale each outcome was coded as an Effect Size (ES), which is defined as the difference between the mean scores of two groups divided by the standard deviation of the control group. For those studies that did not report means and standard deviations, F values or t values were used to estimate the ES; these formulas are presented in Table 1. Also, with studies that employed a one-group pretest-posttest design, in which there was no control group, an alternative approach suggested by Andrews, Guitar, and Howie [39] was used. In their approach the ES is estimated by comparing the post-treatment mean with the pre-treatment mean, and then dividing by the pre-treatment standard deviation.
of aggregate multiple effect sizes from a single study could be avoided. The purpose of the present study was then to synthesize and analyze the existing research on the effects of these two instructional approaches. It was necessary to define these approaches precisely in order to ensure the proper selection of appropriate studies. “Computer Simulation Instruction (CSI)” was taken to refer to classes using computer simulation as a replacement for or supplement to traditional instruction in order to teach students. “Traditional Instruction (TI)” was taken to refer to classes using traditional methods of instruction, that is, non-computer-based methods, to teach students.
2.1 Data Sources The studies considered for use in this meta-analysis came from four sources. One large group of studies came from computer searches of the Chinese Periodical Index. A second group of studies came from the Dissertation and Thesis Abstract System of Taiwan. A third group of studies was retrieved from the Government Research Bulletin (GRB) of Taiwan. The last group of studies was retrieved via the bibliographies of the documents located. Twenty-nine studies were located through these search procedures: 25 came from the Dissertation and Thesis Abstract System, 2 from National Science Council (NSC) research projects, and the other 2 were retrieved from published journals and proceedings. Several criteria were established for inclusion of studies in the present analysis. 1. Studies had to compare the effects of CSI versus TI on students’ achievement. 2. Studies had to provide quantitative results from both CSI and TI classes. 3. Studies had to be retrievable from university or college libraries by interlibrary loan, from the GRB, or from Taiwan’s Dissertation and Thesis Abstract System. 4. Studies had to use Taiwan’s students as subjects. There were also several criteria for eliminating studies or reports cited by other reviews: (a) studies did not report sufficient quantitative data in order to estimate Effect Sizes; (b) studies reported only correlation coefficients - the r value or Chi-square value; (c) studies could not be obtained through interlibrary loans or from standard clearinghouses.
Table 1: Formula Used in Calculating Effect Size ___________________________________________ Mean and SD ES =（Mx－Mc）/ SDc t - value ES = t × 1 / N1 + 1 / N 2 F - Value ES = F × 1 / N1 + 1 / N 2 Note. ES = Effect size. Mx = mean for the experimental group. Mc = mean for the control group. SDc = standard deviation of the control group. N1 = number of subjects in the experimental group. N2 = number of subjects in the control group.
In most cases, the application of the formula given by Glass and his colleagues was quite straightforward. But in some cases, more than one value was available for use in the ES formula. Thus, when some studies reported differences in both posttest measures and pre-post gains, and some studies reported both raw-score differences and covariance-adjusted differences between groups, the pre-post gains and covariance-adjusted differences were selected for estimating ES. In addition, several subscales and subgroups were used in measuring a single outcome: for example, those that reported separate data by gender or grade. In such cases, each comparison was weighted in inverse proportion to the number of comparisons within the study (i.e. 1/n, where n = the number of comparisons in the study) so that the overweighting of the ES in a given study could be avoided (see, for example, [40], p. 230).
2.3 Variables Studied
2.2 Outcome Measures
Seventeen variables were coded for each study in the present synthesis. These variables are listed in Table 2. Each of these variables was placed in one of the following sets of characteristics: (a) study
The instructional outcome measured most often in the 29 studies was student learning, as indicated on standard or researcher-developed achievement tests at the end of the instructional program. For statistical 71
Journal of Information Technology and Applications Vol. 2, No. 2, pp. 69-79, 2007 characteristics, (b) methodological characteristics, and (c) design characteristics. The first 3 variables in the set of study characteristics were coded so that potentially different effects for subjects with different backgrounds could be detected. The other 2 variables (i.e., type of publication and year of publication) in the set of study characteristics were coded because it is important to know how effects are related to sources of information over time. The 6 variables placed in the set of methodological characteristics were coded so that effects related to the characteristics of research procedures could be detected. The last 6 variables in the set of design characteristics were coded because it is critical to know how effects are related to the nature and design of the primary research. Each variable was employed as a factor in an analysis of variance (ANOVA), used to investigate whether there were significant differences within each variable on the ES.
3. Results The number of comparisons and the study-weighted ESs (defined as the overall ES for a single study) are reported in Table 3. Of the 29 studies included in the present synthesis, 27 (93%) of the study-weighted ESs were positive and favored the CSI group, while 2 (7%) of them were negative and favored the TI group. The range of the study-weighted ESs was from -0.197 to 2.67. The overall grand mean for all 29 study-weighted ESs was 0.537. When this mean ES was converted to percentiles, the percentiles indicating student achievement were 70 for the CSI group and 50 for the TI group. The overall grand median for all 29 study-weighted ESs was 0.373, suggesting that percentiles indicating student achievement were 64 for the CSI group and 50 for the TI group. The standard deviation of 0.573 reflects the mild variability in ESs across studies.
Table 2: The Assignments of Studied Variables in Each Characteristic ___________________________________________ Characteristics Variables ___________________________________________ Study Characteristics Grade Level Location of School Subject Area Type of Publication Year of Publication Methodological Characteristics Instructor Bias Instrumentation Reliability of Measure Sample Size Selection Bias Type of Research Design Design Characteristics Comparison group Duration of Treatment Implementation of Innovation Type of Instruction for Treatment Type of Innovation Visual Presentation .
Table 3: Number of Comparisons and Study-weighted Effect Sizes ________________________________________ Author(s) Year N of ES Comparison Chang, H. P. [41] 2001 2 0.273 Chao, J. H. [27] 2001 1 0.645 Chao, J. T. [42] 1999 10 0.322 Chen, C. H. [33] 2002 3 -0.083 Chen, C. [43] 2003 1 0.262 Chen, H. [44] 2005 2 0.207 Chen, T. [45] 1998 1 0.373 Chen, Y. S. [46] 2002 2 0.238 Chuang, C. F. [28] 2000 1 1.964 Guan, H. D. [47] 1999 1 0.428 Hsu, Y. S. et al [48] 2001 1 0.302 Hsu, Y. S. [49] 2002 1 0.370 Huang, C. K. [50] 2002 3 0.433 Huang, J. C. [29] 2002 1 1.165 Jao, Y. H. [26] 1993 2 0.144 Lai, Y. J. [51] 2002 1 -0.197 Li, M. [52] 2005 2 0.837 Lin, C. [53]] 2004 1 0.410 Lin, U. C. [30] 2002 1 2.670 Lin, Y. [54] 2005 1 0.849 Nein, J. S. [31] 2002 4 0.531 Shen, S. [55] 2005 1 0.650 Su, C. Y. [56] 2000 1 0.335 Su, J. S. [32] 2002 15 0.511 Tseng, C. [34] 2002 2 0.084 Wang, C. [57] 2005 1 0.716 Wang, K. K. [58] 1994 3 0.290 Wang, T. [59] 2005 1 0.543 Yu, J. [35] 2002 1 0.144 Overall grand mean 0.537 Overall grand SD 0.573 Overall grand median 0.373 95% Confidence interval 0.319~0.755
2.4 Coder Reliability To obtain more reliable outcomes from coding, the author of this study and 2 research assistants coded the studies. Each of the 2 research assistants coded half of the studies on each of the independent variables. To check for accuracy, the author coded each of the studies independently. The inter-coder agreement for the studies coded by coders was 85%. In addition, the different codings (i.e. inter-coder differences) in studies handled by two coders were discussed. Final agreement had to be reached after discussion.
72
Journal of Information Technology and Applications Vol. 2, No. 2, pp. 69-79, 2007 Methodological Characteristics Instructor bias 3,25 Instrumentation 2,26 Reliability of measure 2,26 Selection bias 2,26 Sample size 2,26 Type of research design 3,25
Range -0.197~2.670 Positive ES (%) 27(93%) Negative ES (%) 2(7%) Note. Total N = 2729. Total N of studies = 29. Total N of comparisons = 67 Among the 67 ESs included in the present synthesis, 59 (88%) were positive and favored the CSI group, while 8 (12%) were negative and favored the TI group. The range of ESs was from -0.691 to 2.67. The ESs for the 67 comparisons are displayed in a scatter diagram in Figure 1. This diagram shows that despite several large effects, most of the ESs were small to moderate in magnitude. About 66% of the ESs lie between -0.5 and 0.5, while less than 31% of the ESs were greater than 0.5. Table 4 lists the F values for the 17 variables for all study-weighted ESs in the study. Descriptive statistics for the 17 variables are presented in Table 5. Among the 17 variables, only 1 variable (reliability of measure), showed statistically significant impact. The post hoc test (Scheffe), F(2,26)=3.88, p
