Science Education Department, Florida Institute of Technology, Melbourne, ... recently reported is a 12-year longitudinal study of science concept learning in.
An investigation of the Effectiveness of Concept Mapping as an instructional Tool PHILLIP B. HORTON Professor, Science Education Department, Florida Institute of Technology ANDREW A. McCONNEY, MICHAEL GALLO, AMANDA L. WOODS, GARY J. SENN, AND DENIS HAMELIN Science Education Department, Florida Institute of Technology, Melbourne, FL 32901
INTRODUCTION
The study of concept mapping as a research topic evolved from work conducted at Cornell University under the auspices of Novak (Novak and Gowin, 1984). Most recently reported is a 12-year longitudinal study of science concept learning in which Novak and his colleagues developed concept maps as a tool to represent knowledge structures (Novak and Musonda, 1991). Predicated on Ausubel’s assimilation theory of cognitive learning, these maps depicted the hierarchy and relationships among concepts. Data gathered from clinical interviews given before and after instruction were transformed from their raw, propositional form to concept maps. These “before and after” maps, which represented specific pre- and postinstruction concept meanings held by students, were then analyzed for changes in students’ cognitive structure. In summarizing the results of this study, Novak and Musonda reported that experimental students showed “many more valid conceptions and many fewer invalid conceptions” (p. 148) when compared to a similar sample of students who received no formal instruction in basic science concepts. In the research done since Novak developed this tool, concept mapping has become a viable educational medium. For example, there is evidence that concept maps can help teachers become more effective (Beyerbach and Smith, 1990; Hoz et al., 1990), and can serve as a heuristic for curriculum development (Starr and Krajcik, 1990). Perhaps most importantly, concept maps have been reported to be a potent instructional tool for promoting what Ausubel has described as meaningful learning. Meaningful learning refers to anchoring new ideas or concepts with previously acquired knowledge in a nonarbitrary way (Novak, 1977). It is this latter role of concept maps that is the focus of this study. Science Education 77(1): 95-1 11 (1993) 0 1993 John Wiley & Sons, Inc.
CCC 0036-8326193/ 010095-17
96
HORTON ET AL.
RESEARCH QUESTIONS
Concept mapping’s efficacy in promoting meaningful learning has been reported in previous studies. For example, concept maps have been reported t o advance meaningful learning in: (a) the classification and problem-solving skills of seventhgrade students (Novak et al., 1983); (b) earth science concepts (Ault, 1985); and (c) biology concepts (Lehman et al., 1985; Okebukola, 1990; Stewart et al., 1979). Collectively, these reports (and other related studies) led to the following research questions: 1. What is the effectiveness of concept mapping as an instructional tool for improving students’ achievement? 2. What is the effectiveness of concept mapping as a strategy for improving students’ attitudes? In addition to concept mapping’s impact on achievement and attitudes, there have also been studies that addressed certain peripheral issues. For example, it has been reported that a distinction should be made between using teacher-prepared maps and those created by student concept-mapping activities as the basis of an instructional program (Cliburn, 1990). Novak (1979) observed that using teachermade maps in college genetics instruction was not especially beneficial and had the potential to confuse students. On the other hand, Cliburn (1990) stated that he frequently uses his own concept maps as instructional tools in an expository setting, and that “work with teacher-prepared maps in the classroom over the past seven years has convinced me that, when thoughtfully applied in a systematic fashion, this strategy facilitates both content-specific learning and integrative understanding by students, and has other advantages as well” (p. 212). This distinction between teacher-prepared maps and student-prepared maps generated the third research question:
3. Is there a difference in the effectiveness of teacher-prepared versus studentprepared concept maps in improving student achievement and/or attitudes? Gender was also considered an important factor in some studies. In Novak’s longitudinal study it was observed that female students had a tendency to create less integrated and less complex concept maps (Novak and Musonda, 1991). Novak and Musonda also reported that in later grades males had better conceptual understanding than females. In another study, Jegede et al. (1989) reported that males who used concept mapping demonstrated greater achievement gains than similar females. These studies led to the fourth and final research question:
4. Is there a gender effect when concept mapping is used as an instructional tool? PROBLEM STATEMENT
This study’s purpose was to integrate the outcomes of the available research on the effects on students of using concept mapping as an instructional tool. Metaan-
CONCEPT MAPPING
97
alysis, originally developed by Gene Glass (1981), was used to synthesize the research findings by expressing the size of treatment effects on a common scale (effect sizes). In its simplest form, an effect size (ES) is determined by subtracting the mean score on the dependent variable of the control group from the mean score of the experimental group and dividing this difference by the standard deviation of the control group. The metaanalysis technique was selected since it is “currently the best available method for cumulating and integrating the results of research” (Borg and Gall, 1989, p. 173). The specific metaanalytic procedures followed were those outlined by Kulik and Kulik (1989), and Hedges et al. (1989). METHOD Data Sources and Procedures for Locating Studies In collecting the studies used in this metaanalysis, appropriate data sources that would provide a large number of studies related to concept mapping were first determined. These sources were then searched for reports that would be applicable to the study. (The term “reports” used in this context refers to dissertations and any published or unpublished articles appearing in a journal, periodical, book, or collection of papers.) Copies of potentially applicable studies were then obtained. Finally, a secondary list of possible studies for the metaanalysis was generated from the bibliographies that appeared in the reports obtained. The data sources used were the computerized data bases of (a) the Educational Resources Information Center ( E R I C ) ; (b) Dissertation Abstracts International; and (c) Psychological Abstracts. A search of these data bases was conducted using the key words: “concept mapping,” “semantic mapping,” and “concept maps”; in searching Dissertation Abstracts International the key word “education” was also included. Adopting the approach of Bangert-Drowns et al. (1985), a research report was determined to be acceptable if it satisfied three criteria. First, the studies had to occur in actual classrooms. Second, only studies that utilized the concept mapping technique as an instructional tool were considered; that is, only those studies that compared quantitatively measured outcomes for treatment classes using concept mapping, with outcomes for classes using some alternative instructional method as a control were included. Third, the report had to provide sufficient data for the calculation of an effect size. The decision to consider a report for the metaanalysis was based on the information contained in its title and/or abstract. Attempts were then made to acquire copies of the identified reports by searching the campus library and submitting requests for interlibrary loans. For dissertations that were not available locally or through interlibrary loans, attempts were made to obtain copies from ERIC, University Microfilms International, the appropriate academic department at the university where a dissertation originated, or the authors themselves. A similar procedure was followed for those titles that were on the secondary list of studies. The bibliographic searches yielded 133 titles warranting consideration. A minimum of two authors independently read each report and jointly made the decision to accept or reject the report. In the final analysis, only 18 of the research reports
98
HORTON ET AL.
considered met the a priori criteria for inclusion in the metaanalysis. These 18 reports provided the data used in this study. (Of the studies that were not included in this analysis, nearly all failed to meet the second criterion mentioned above, i.e., they did not compare quantitatively measured outcomes for treatment classes using concept mapping, with outcomes for classes using some alternative instructional method as a control.) Coding the Studies and Quantifying the Results
Based on guidelines provided by Hedges et al. (1989), a metaanalysis coding form (see Appendix) was designed to facilitate the collection of key characteristics and data from the 18 concept mapping reports. The types of outcomes most frequently reported in the coded studies were achievement and attitude but other measures were also reported (e.g., retention, anxiety). Glass’s formula was used to calculate ESs for studies that reported means and standard deviations for both treatment and control groups (Glass et al., 1981). For those studies that did not fully report this information, ESs were calculated using procedures detailed in either Hedges et al. (1989) or Kulik and Kulik (1989). Where multiple comparisons were reported in a study, Kulik and Kulik’s (1989) conservative approach was adopted: only one ES per outcome area was coded. This procedure avoided the problem of nonindependence and insured that undue weight was not given to studies with multiple scales or measures. The most important or relevant ES for each outcome area was selected for the metaanalysis based on the following guidelines provided by Bangert-Drowns et al. (1985):
1. When results were provided from both true experimental and quasi-experimental comparisons, the results of the true experiment were chosen. 2. When results from both a long and a short concept mapping implementation were available, the results of the longer implementation were selected. 3. When results of both the direct effects in the area of instruction, and the transfer effects of concept mapping to other subject areas were available, direct effects were chosen. 4. In all other cases, total score and total group results in preference to subscore and subgroup results were used. These guidelines can be summarized by the single rule: “use the ES from what would ordinarily be considered the most methodologically sound comparison when comparisons differed in methodological adequacy” (Bangert-Drowns et al., 1985, p. 63). Raters
The coding team was comprised of one science education faculty member (the senior author) and five students participating in a doctoral seminar centered on research design. The senior author has taught graduate courses in statistics and research design for 13 years. As a prerequisite to participation in this seminar, the
CONCEPT MAPPING
99
five graduate students had all successfully completed graduate courses in statistics and educational research design. At a minimum, two members of the team independently coded the characteristics of each study. The forms were then jointly reviewed, any coding disagreements discussed, and a final coding form produced. To determine the reliability of coding, Krippendorff‘s (1980) suggestion to use Scott’s pi as a measure of the reproducibility of raters’ coding was followed. This measure takes into account the percentage of coded items which would be expected to agree merely by chance. Reliability estimates ranging from 0.79 to 0.82 were obtained for three pair-wise comparisons when the team members rated a sample study. These measures indicate that the coding agreement was, on average, 81% higher than what one would expect due to chance. This level of coding reliability is in accordance with standards offered by Krippendorff (1980).
R ESULTS The search yielded 18 research reports that met the criteria for analysis. Fourteen of these gave outcome measures in either achievement or attitude, three gave outcomes for both student attitude and achievement, and one study (Spaulding, 1989) examined concept mapping in two content areas for independent groups of students. Consequently, for Spaulding’s research, a separate ES was listed for each content area in accordance with the guidelines followed (Kulik and Kulik, 1989). This resulted in a total of 19 studies listed. The study results were presented in two ways. In Table 1 the major features of each study were tabulated. Three additional tabulations organized the results based on sample characteristics, treatment and control group characteristics, and experimental design characteristics. As shown in Table 1, student-prepared concept maps were most common in the studies examined. In 15 of the 19 studies (79%) analyzed, students prepared the maps, while in three studies teachers prepared the maps. In one study, concept maps were prepared by both teachers and students. Two studies involved elementary grade students, two involved middle school students, nine involved high school students, and the remaining six had college students as subjects. Nearly all the studies examined involved science content; only two used nonscience subject areas. One of these explored the effect of concept mapping on students’ reading comprehension while the second investigated the effect of concept mapping on social studies achievement. Of the other 17 studies, biological science was the content focus of nine studies, while three studies used a chemistry content area. One study examined concept mapping in both biology and chemistry for independent groups of students (Spaulding, 1989). The three remaining studies used earth or physical science content. Studies also varied in terms of sample size and duration. The average sample size of the 19 studies was 95 students. Eight studies had sample sizes greater than 100, three studies had sample sizes between 50 and 100, and eight studies had sample sizes of less than 50. The average length of the 19 studies was slightly less than six weeks. One study lasted 22 weeks, while two studies lasted one week or less.
Earth Science Chemistry Marine Science Biology Biology Chemistry Biology Biology Social Studies Biology Biology Ecology Physics Reading Biology Chemistry Biology Chemistry Physical Science
Abayomi, 1988 D Basili, 1988 D Bodolus, 1986 D Cliburn, 1985 D Heinze-Fry & Novak, 1990 Huang, 1991 D Jegede et al., 1989 Lehrnan et al., 1985 Loncaric, 1986 D Martin & Lucy, 1992 Okebukola & Jegede, 1988 Okebukola & Jegede, 1989 Pankratius, 1987 D Prater & Terry, 1988 Schmid & Telaro, 1990 Spaulding, 1989 D Spaulding, 1989 D Stensvold & Wilson, 1990 Willerman & Mac Harg, 1991
U, unknown; D, dissertation.
Content
Study
9 8
9+ 11 10
9 5 9+ 13 13 12 5
10
8 13 9 14 13 13
Grade
3 4 8 6 4 3 3 3 0.2
U
3 4 4 6 21 1
U
4 22
Length (weeks)
156 49 244 70 37 129 51 237 41 31 190 138 28 30 43 44 107 104 82
Sample size
TABLE 1 Major Study Features and Effect Sizes in 19 Concept Mapping Studies
Students Students Students Teacher Students Students Students Students Students Both Students Students Students Teachers Students Students Students Students Teachers
Maps Made by
4.88
1.01
0.05 0.32
Attitude ES
0.13 0.70 0.11 - 0.31 - 0.13 0.35 0.39
0.15 0.12 0.06 0.68 0.52 0.21 2.02 0.04 0.97 0.48 1.63
Achievement ES
r
B
-I
m
B
i
0 0
-L
CONCEPT MAPPING 101 As can also be seen in Table 1, the ESs calculated showed considerable scatter for both achievement and attitude outcome measures. For student achievement, ESs ranged from a low of -0.31 to a high of 2.02 standard deviation units, while for attitude measures ESs showed an even wider range (0.05-4.88 standard deviation units). Also, the studies listed were evenly split between dissertations (nine studies) and other published or presented research (ten studies). Tables 2-4 offer breakdowns of the studies reviewed according to their sample, treatment, control group, and experimental design characteristics. In these three tables, ES medians (a measure of central tendency more resistant to skew than the mean), and standard errors (SE) are also provided. Standard error is the appropriate measure of variability when comparing sample means to a population mean (Gravetter and Wallnau, 1985), and allows readers to assess the precision of study outcomes summarized in this metaanalysis.
Sample Characteristics Eleven variables (see Appendix) were used to describe the sample characteristics of each study. As Table 2 shows, only two of these (grade level and school location) were reported consistently enough to warrant tabling here. Of the 18 concept mapping studies which assessed achievement, two were conducted at the elementary level (grade 5), two were at the middle school level (grade 8), and nine studies were conducted at the high school level (five at grade 9, two at grade 10, and one each at grades 11 and 12). Finally, five of the 18 studies took place in college settings (four at the freshman level and one at the sophomore level). Four studies assessed students’ attitudes toward the subject area. Of these, two were conducted at the high school level (one each for grades 9 and 10) and the other two studies were at the college level (one each for freshmen and sophomores). Four of the 18 achievement studies took place outside of the United States and Canada. Three originated in Nigeria, and one in Taiwan. The remaining 14 studies
TABLE 2 Effect Sizes for 18 Concept Mapping Studies Classified by Sample Characteristics Achievement Median
Mean
SE
n
Median
Mean
SE
2 2 9 5
0.84 0.27 0.11 0.52
0.84 0.27 0.31 0.63
0.14 0.12 0.23 0.27
0 0 2 2
0.53 2.60
0.53 2.60
0.48 2.28
14 4
0.24 0.92
0.29 1.00
0.09 0.48
2 2
0.19 2.95
0.19 2.95
0.14 1.94
n Grade level Elementary Middle High school ColIege School location U S . or Canada Other countries
SE, standard error.
Attitude
0.59 0.48 0.42 0.66 0.88 0.05 0.08 0.87 0.09
0.74 0.33 - 0.04 0.60 0.27 - 0.08
0.68 0.48 0.14 0.35 0.88 0.06
0.13 0.52 0.09 0.42 0.33 0.04 0.48 0.27 0.13
5
7 4 8 2 5
13 2 3
Mean
3 1 14 9 2 5
Median
SE, standard error. =One study used concept maps prepared by both individuals and groups. bTeachersand students were the sources of key terms in one study.
Treatment characteristics Map preparation By teacher By both By studentsa Students individually Students in groups Unknown Source of key termsb Teacher Students Unknown Map preparation in class Outside of class Unknown Control group characteristics Conventional No treatment Placebo
n
Achievement
0.17 0.12 0.15
0.26 0.20 0.08
0.14 0.27 0.02
1.57
0.05
0.05 0.67
2.95
2.95
2.95 0.05
1.12
na
1.94
na
1.94
na
0.05 0.05 2.95 0.05
1.47 1.94 1.98 2.95
na
1.01 2.95
na
SE
0.18 0.24 0.76 0.13
Mean
0.32
Median
0.32
n
Attitude
0.10
SE
-
TABLE 3 Effect Sizes for 19 Concept Mapping Studies Classified by Treatment and Control Group Characteristics
r
D
4
m
d z
I
0 u
N
0
A
Design Characteristics
~~
3
9 9
9 6
0.19 0.21
0.50 0.33 0.57
0.25 0.15
0.39
0.45 0.10 0.25
1.16 0.18
0.16
0.97 0.13
0.28 0.28 0.12 0.21
0.15
0.25 0.09 0.14
SE
0.52 0.55
0.14 0.41 0.13 0.17
8 2 3 10 1 2
0.46
0.35
0.84
0.m 0.1 5
Mean
0.60 0.41 0.24 0.39 0.52 0.55
0.35 0.21
0.15 0.84
0.48
Median
1 17
2
9 7
n
Achievement
aSome studies used more than one method to control preexisting differences.
SE,standard error.
Content area Biology Physical science Nonscience Unit of analysis Class lndividuai Control for preexisting differencesa Pretest as a covariate Other covariates Random assignment of subjects Random assignment of classes Matching of subjects No control Mortality Zero Some Unknown Treatment fidelity Medium High
~
n
TABLE 4 Effect Sizes for 19 Concept Mapping Studies Classified by Design Characteristics
2.60 0.53
0.53
2.60
0.53
0.53
2.28 0.48
0.48
1.12 na
1.57 0.32 0.67 0.32
1.1
SE
1.12
1.57
Mean
1.57 0.67
0.67
Median
Attitude
0 0
A
Z c)
2
p
$
104 HORTON ET AL.
were conducted either in the U.S. or Canada (one Canadian study). Of the four studies which measured attitude, two were conducted in the U.S. and two in Nigeria. Treatment Characteristics
Table 3 presents the 19 concept mapping studies according to their treatment and control group characteristics. Of the six variables (see Appendix) that were used to describe the treatment characteristics of each study, four are reported in Table 3. Treatment length was not reported for many studies, and this variable is not included in Table 3. The overall length of studies is reported in Table 1. Concept maps were prepared by students in 14 of the 18 achievement studies, while teachers prepared the maps in three studies. One study (Martin and Lucy, 1992) used maps prepared by both teachers and students. Concept maps were prepared by students in three of the four studies which measured students’ attitudes, and teachers prepared the maps in the other attitude study. For the 15 achievement studies in which students (or students and teachers) prepared the maps, eight studies used individually prepared maps only. Okebukola and Jegede (1988) reported that students prepared concept maps both in groups and as individuals, while in Basili’s (1988) research, maps were prepared in groups only. The five remaining studies did not provide this information. Similarly, this information was not provided in one of the three attitude studies in which students prepared the concept maps, but in the other two attitude studies students prepared maps individually. The teacher was the source of key concept-mapping terms for five of the 15 achievement studies in which students prepared concept maps. Individual students were the source of key terms in another seven of these 15 achievement studies, and in Martin and Lucy’s study (1992) key concept-mapping terms were provided by both students and teachers. The source of key terms in the four remaining studies was not indicated. Students were the source of key concept-mapping terms for two of the three attitude studies; this information was not provided in the other attitude study. Concept maps were prepared in class for eight of the 15 achievement studies in which students prepared maps, while maps were prepared outside of class in two studies. The remaining five studies did not include this information. Concept maps were prepared in class for two of the three attitude studies in which students prepared maps; the other study did not indicate where students prepared the maps. Control Group Characteristics
Four variables (see Appendix) were used to describe the control group characteristics of each study. As Table 3 shows, only three of these were reported as no study was found to use a waiting list control group. The majority (13) of achievement outcomes analyzed were measured in comparison to control groups which received a conventional or traditional didactic type of instruction. Two other achievement studies employed control groups which
CONCEPT MAPPING 105 received no treatment, while three studies made use of a placebo treatment for their comparison groups. The four attitude outcomes were all measured in comparison to groups which received conventional instruction. Design Characteristics
Five variables (see Appendix) were used to describe the design characteristics of each study, all of which are included in Table 4. Biological science was the content area in nine of the 18 achievement studies examined. Physical or chemical science formed the subject area in another seven studies, while the two studies conducted with elementary students used nonscience content areas. Biological science was also the content area used in all four of the attitude studies coded. Individual students formed the experimental unit of analysis in all but one of the 18 achievement studies. Likewise, all four studies which examined student attitudes used the individual student as the unit of analysis. In attempting to equalize samples through randomization, matching, or covariate techniques, most studies (12) used either a pretest as covariate or random assignment of intact classes to treatment and control groups. Three studies (Bodolus, 1986; Jegede et al., 1989; Pankratius, 1987) used both of these techniques to control for possible preexisting differences. All four attitude studies used a pretest as covariate, while two of these also employed random assignment of classes. Two achievement studies failed to provide any control for possible preexisting differences in their subjects. Of the 18 achievement studies, 50% experienced some mortality of subjects, while in a further six studies the existence of mortality could not be determined, and three studies reported zero mortality. Three attitude studies also reported zero mortality, while in one study the presence or absence of mortality could not be determined. Study designs were further characterized according to the degree of faithfulness treatments exhibited to stated theoretical objectives. For achievement outcomes, nine studies were determined to have a high degree of treatment fidelity, while the other nine studies were coded as having medium treatment fidelity. Similarly, for the four attitude studies, two exhibited high treatment fidelity, and two exhibited medium treatment fidelity. DISCUSSION
In answering the first two questions posed in this study, metaanalysis results showed that concept mapping has generally positive effects on both student achievement and attitudes in the 19 studies examined. Concept mapping raised individual student achievement in the average study by 0.46 standard deviations, or from the 50th to the 68th percentile. Concept mapping also strongly improved student attitudes. These positive overall effects are generally supportive of what previously had been reported in the literature and were therefore somewhat expected. However, the examination of these effects using the cross-tabulations presented in Tables 2-4 reveal some differences which may warrant closer inspection, or raise new questions for future research. For instance, quite different average ESs were
106 HORTON ET AL.
found depending on the subject area under examination. In the nine studies with biology as the content focus, students exposed to concept mapping achieved on average at the 72nd percentile, while experimental students of the typical study examining chemical or physical science achieved at only the 56th percentile. Also noted with interest was a considerable difference in the average size of both achievement and attitude effects depending on the location of the study. The 14 concept mapping studies conducted in the U.S. and Canada raised average achievement by 0.29 standard deviations (from the 50th to the 61st percentile), while in stark contrast. the four achievement studies conducted in Nigeria (three studies) and Taiwan (one study) raised average achievement by fully 1.00 standard deviation (from the 50th to the 84th percentile). The same contrast was even more evident in concept mapping’s effect on student attitudes. For the two U.S. studies, concept mapping improved the average student’s attitude by 0.19 standard deviations whereas in the two Nigerian studies, students’ attitudes improved on average by 2.95 standard deviations (from the 50th to the 57th and 99th percentiles, respectively). From these notable differences depending on study location, it may be that novelty and Hawthorne effects were perhaps much stronger for Nigerian students than for students in North America who may be accustomed to experiencing instructional and technological innovations. Thus, in combination with the effects of concept mapping, these confounding variables may have acted to produce the large observed differences. It was also interesting to note the differences in achievement effects when comparing those studies which employed a conventional or traditional control group, with those which employed a placebo treatment for their control groups. When compared to normally instructed students, average concept mapping students achieved at the 72nd percentile. On the other hand, when compared to students receiving an alternative treatment, concept mapping students achieved at only the 47th percentile. This may be a somewhat artificial difference however, as two of the three studies using a placebo exhibited negative ESs (Spaulding, 1989). These two were calculated using means and standard deviations reported by Spaulding, but may not provide an accurate reflection of the treatment’s effect. Control and experimental students in his study were significantly different before the start of the concept mapping treatment, and ended the experiment in the same condition. A caution is therefore issued for readers examining this comparison. The metaanalysis also provided evidence that there is little difference in the effectiveness of teacher-preparerd versus student-prepared concept maps in improving students’ achievement. In the four studies which used teacher-prepared maps as instructional tools, the average student’s achievement improved from the 50th to the 71st percentile. Similarly, in studies which used student-prepared maps, average achievement improved from the 50th to the 66th percentile. It was therefore somewhat surprising that when students did prepare their own concept maps, they showed much stronger achievement gains if they themselves were required to supply the key terms necessary to construct the maps (mean ES = 0.87). For the five studies in which teachers supplied the key terms for students who then constructed maps, minimal average gain in achievement was evident (mean ES = 0.08).
CONCEPT MAPPING 107 A comparison between the effectiveness of teacher-prepared and student-prepared concept maps in improving student attitudes was not made since only one of the four studies employed teacher-prepared maps. Also, since only one study (Jegedel et al., 1989) included information about whether concept mapping as an instructional tool had different effects for male and female students, the fourth research question remained unanswered.
RECOMMENDATIONS As readers will have noted, only two of the 11 variables describing the students and settings under examination were reported often enough t o allow their inclusion in this metaanalysis. Many reports also failed to include important information such as the length of treatment to which students were exposed, or treatment details such as who prepared the concept maps (teachers or students or both) and who supplied the concept-mapping terms. In the interests of reporting research which is consistently meaningful and reproducible, the following recommendation is offered: authors, researchers, and reviewers should pay greater attention to clearly defined and fully described samples and procedures as these are necessary and useful report components. When clearly reported, these components allow interested readers to judge the adequacy or applicability of any findings or conclusions, and also facilitate replication of the research.
LIMITATIONS Initially, it may seem surprising that the literature search for this metaanalysis yielded only 19 studies from over a decade of research on concept mapping as an instructional tool. However, in conducting their seminal metaanalysis on the effectiveness of computer-based education, Bangert-Drowns et al. (1985) were able to include only 42 studies (about 8%) of the nearly 500 titles originally generated by their literature search. This metaanalysis of concept mapping’s effectiveness as an instructional tool is therefore in line with the above exclusion rate as we found 19 out of a possible 133 studies (about 14%) met the necessary criteria. Despite this, some of the comparisons provided in this summary are based on a limited number of studies, and readers are cautioned to bear this in mind. Readers should also be reminded of the caution offered by Bangert-Drowns et al. (1985): “studies located for most meta-analyses represent very poorly the total universe of studies in which a treatment might have been studied” (p. 66). While the view of these authors that metaanalytic conclusions must necessarily be limited by past research is shared and applicable to the findings of this study, no limitations on the possibilities for future instructional innovators are implied or intended. CONCLUSIONS The results of this metaanalysis of 19 studies showed that the top-down instructional strategy of concept mapping, pioneered by Novak and based on Ausubelian learning theory, has had generally medium positive effects on students’ achieve-
108 HORTON ET AL. ment, and large positive effects on student’s attitudes. In these 19 studies, achievement and attitude gains were most strongly pronounced in studies conducted in Nigeria. Improved achievement was more evident in studies which used biology as their content focus, and was also considerably stronger in studies which used conventional instruction rather than a placebo for their control groups. No evidence was found to show that student-prepared maps were more effective than teacherprepared maps. Note: An earlier version of this article was presented at the 65th annual meeting of the National Association for Research in Science Teaching, Boston, MA.
APPENDIX METAANALYSIS CODING FORM Study Characteristics Reference Study code & subgroup Coder ID (initials) Sample characteristics Sample size, control Sample size, treatment Grade level Age Socioeconomic status (low, medium, high, mixed, unknown) Ability (low, medium, high, mixed, unknown) Homogeneous ability levels Gender (YOfemale) Ethnic (% group with largest proportion, approximately) Homogeneous ability levels School size (#, small, medium, large) Type of school (public, private, U.S., etc.) School location (rural, suburban, urban, inner city, mixed) Treatment Teacher prepared maps Students prepared maps Maps prepared: individually or in groups Source of key terms: teachers or students Maps prepared: in class or out of class Length in weeks Hours of treatment (exact if given, or # periods*.75) Control group Conventional No treatment Placebo Waiting list
CONCEPT MAPPING 109 Teachers Years experience (average) Education (BS, MS, PhD, mixed, unknown) Trained for study (low, medium, high, mixed, unknown) Gender (O/O female) Ethnicity (YO) Assignment (in or out of field) Design Content area (biology, chemistry, etc.) Unit of analysis (n = class or individual) Control for preexisting differences: Pretest as a covariate Other covariates (yes or no) Random assignment of subjects Random assignment of classes Matching of classes Matching of subjects No control No information Mortality (zero, #, unknown) Treatment fidelity (low, medium, high) Outcomes Type of outcome (achievement, attitude, etc.) Congruence of outcome/treatment (low, medium, high) Reactivity of outcome measure (low, medium, high) Format of outcome measure (multiple choice, essay, etc.) Test &/or scale name, or unpublished Effect size (ES) information Control group pretest mean Treatment group pretest mean Control group posttest mean Treatment group posttest mean Control group standard deviation Treatment group standard deviation Pooled standard deviation Calculated ES Direction of effect ( + , 0, or - ) Reported as statistically significant (yes or no) Source of standard deviation (control group or pooled) Source of effect size data for approximate calculations F or t-test statistics ANCOVA sums of squares ANCOVA adjusted means Gainscore or pre-post correlations Value of F , t , or r used
110 HORTON ET AL.
REFERENCES Abayomi, B. I. (1988). The effects of concept mapping and cognitive style on science achievement. Doctoral Dissertation, Georgia State University College of Education. Dissertation Abstracts International, 49, 1420A. Ault, C. (1985). Concept mapping as a strategy in earth science. Journal of College Science Teaching, 15, 38-44. Bangert-Drowns, R . , Kulik, J., & Kulik, C. (1985). Effectiveness of computer-based education in secondary schools. Journal of Computer-Based Instruction, 12, 59-68. Basili, P. A. (1988). Conceptual change strategies within cooperative groups of community college chemistry students: An experiment. Doctoral Dissertation, University of Maryland. Dissertation Abstracts International, 49, 1752A. Beyerbach, B., & Smith, J. (1990). Using a computerized concept mapping program to assess preservice teachers’ thinking about effective teaching. Journal of Research in Science Teaching, 27, 961-972. Bodolus, J. E. (1986). The use of concept mapping strategy to facilitate meaningful learning for ninth grade students in science. Doctoral Dissertation, Temple University. Dissertation Abstracts International, 47, 3387A Borg, W. R., & Gall, M. D. (1989). Educational research: An introduction, 5th Ed. New York, Longman. Cliburn, J. W. Jr. (1985). An Ausubelian approach to instruction: The use of concept maps as advance organizers in a junior college anatomy and physiology class. Doctoral Dissertation, University of Southern Mississippi. Dissertation Abstracts International, 47, 852A. Cliburn, J. W. Jr. (1990). Concept maps to promote meaningful learning. Journal of College Science Teaching, 19, 212-217. Glass, G . V., McGaw, B., & Smith, M . L. (1981). Meta-analysis in social research. Beverly Hills, CA: Sage Publications. Gravetter, F. J., & Wallnau, L. B. (1985). Statistics for the behavioral sciences, 2nd Ed. St. Paul, MN: West Publishing Company. Hedges, L. V., Shymansky, J. A . , & Woodworth, G. (1989). Modern methods of metaanalysis: A practical guide. Washington, DC: National Science Teachers Association. Heinze-Fry, J. A., & Novak, J. D. (1990). Concept mapping brings long-term movement toward meaningful learning. Science Education, 74, 461-472. Hoz, R., Tomer, Y., & Tamir, P. (1990). The relations between disciplinary and pedagogical knowledge and the length of teaching experience of biology and geography teachers. Journal of Research in Science Teaching, 27, 973-988. Huang, W. (1991). Concept mapping and chemistry achievement, integrated science process skills, logical thinking abilities, and gender at teachers’ colleges in Taiwan. Unpublished doctoral dissertation, Florida Institute of Technology, Melbourne, FL. Jegede, 0. J., Alaiyemola, F. F., & Okebukola, P. A. (1989). The effect of a metacognitive strategy on students’ anxiety and achievement in biology. ERIC Document Reproduction Service No. ED 313 219. Jegede, 0. J., Alaiyemola, F. F., & Okebukola, P. A. (1990). The effect of concept mapping on students’ anxiety and achievement in biology. Journal of Research in Science Teaching, 27, 951-960. Kulik, J. A., & Kulik, C. C. (1989). The concept of meta-analysis. International Journal of Educational Research, 13, 227-234. Krippendorff, K. (1980). Content analysis: An introduction to its methodology. Beverly Hills, CA: Sage Publications. Lehman, J. D., Carter, C., & Kahle, J. B. (1985). Concept mapping, vee mapping, and
CONCEPT MAPPING 111 achievement: Results of a field study with black high school students. Journal of Research in Science Teaching, 22, 663-673. Loncaric, L. (1986). The effect of a concept mapping strategy program upon the acquisition of social studies concepts. Doctoral Dissertation, University of Pittsburgh. Dissertation Abstracts International, 47, 2006A. Martin, D. J., & Lucy, E. C. (1992, March). The effect of concept mapping on biology achievement of field dependent students. Paper presented at the annual meeting of the National Association for Research in Science Teaching, Boston, MA. Novak, J. D. (1977). A theory of education. Ithaca, NY: Cornell University Press. Novak, J. D. (1979). Meaningful reception learning as a basis for rational thinking. In A. E. Lawson (Ed.), The psychology of teaching for thinking and creativity. 1980 AETS Handbook. Columbus, OH: ERICISMEAC. Novak, J. D., & Gowin, D. B. (1984). Learning how to learn. New York: Cambridge University Press. Novak, J. D., & Musonda, D. (1991). A twelve-year longitudinal study of science concept learning. American Educational Research Journal, 28, 117-153. Novak, J. D., Gowin, D. B., & Johansen, G. T. (1983). The use of concept mapping and knowledge vee mapping with junior high school science students. Science Education, 67, 625-645. Okebukola, P. A. (1990). Attaining meaningful learning of concepts in genetics and ecology: An examination of the potency of the concept-mapping technique. Journal of Research in Science Teaching, 27, 493-504. Okebukola, P. A,, & Jegede, 0. J. (1988). Cognitive preference and learning mode as determinants of meaningful learning through concept mapping. Science Education, 72, 489-500. Okebukola, P. A., & Jegede, 0. J. (1989). Students’ anxiety towards and perception of difficulty of some biological concepts under the concept-mapping heuristic. Research in Science and Technological Education, 7, 85-92. Pankratius, W. J. (1987). Building an organized knowledge base: Concept mapping and achievement in secondary school physics. Doctoral Dissertation, Georgia State University College of Education. Dissertation Abstracts International, 49, 474A. Prater, D. L., & Terry, C. A. (1988). Effects of mapping strategies on reading comprehension and writing performance. Reading Psychology, 9, 101-120. Schmid, R. F., & Telaro, G. (1990). Concept mapping as an instructional strategy for high school biology. Journal of Educational Research, 84, 78-85. Spaulding, D. T. (1989). Concept mapping and achievement in high school biology and chemistry. Doctoral Dissertation, Florida Institute of Technology. Dissertation Abstracts International, 50, 1619A. Starr, M., & Krajcik, J. (1990). Concept maps as a heuristic for science curriculum development: Toward improvement in process and product. Journal of Research in Science Teaching, 27, 987-1000. Stensvold, M. S . , & Wilson, J. T. (1990). The interaction of verbal ability with concept mapping in learning from a chemistry laboratory activity. Science Education, 74, 473480. Stewart, J., Van Kirk, J., & Rowell, R. (1979). Concept maps: A tool for use in biology teaching. American Biology Teacher, 41 (3), 171-175. Willerman, M., & Mac Harg, R. A. (1991). The concept map as an advance organizer. Journal of Research in Science Teaching, 28, 705-711.
Accepted for publication 14 September 1992
