Theory Into Practice Designing and Validating ...

This article was downloaded by: [University North Carolina - Chapel Hill] On: 22 July 2015, At: 07:52 Publisher: Routledge Informa Ltd Registered in England and Wales Registered Number: 1072954 Registered office: 5 Howick Place, London, SW1P 1WG

Theory Into Practice Publication details, including instructions for authors and subscription information: http://www.tandfonline.com/loi/htip20

Designing and Validating Assessments of Complex Thinking in Science Kihyun Ryoo & Marcia C. Linn Accepted author version posted online: 05 May 2015.

Click for updates To cite this article: Kihyun Ryoo & Marcia C. Linn (2015) Designing and Validating Assessments of Complex Thinking in Science, Theory Into Practice, 54:3, 238-254, DOI: 10.1080/00405841.2015.1044374 To link to this article: http://dx.doi.org/10.1080/00405841.2015.1044374

PLEASE SCROLL DOWN FOR ARTICLE Taylor & Francis makes every effort to ensure the accuracy of all the information (the “Content”) contained in the publications on our platform. However, Taylor & Francis, our agents, and our licensors make no representations or warranties whatsoever as to the accuracy, completeness, or suitability for any purpose of the Content. Any opinions and views expressed in this publication are the opinions and views of the authors, and are not the views of or endorsed by Taylor & Francis. The accuracy of the Content should not be relied upon and should be independently verified with primary sources of information. Taylor and Francis shall not be liable for any losses, actions, claims, proceedings, demands, costs, expenses, damages, and other liabilities whatsoever or howsoever caused arising directly or indirectly in connection with, in relation to or arising out of the use of the Content. This article may be used for research, teaching, and private study purposes. Any substantial or systematic reproduction, redistribution, reselling, loan, sub-licensing, systematic supply, or distribution in any form to anyone is expressly forbidden. Terms &

Downloaded by [University North Carolina - Chapel Hill] at 07:52 22 July 2015

Conditions of access and use can be found at http://www.tandfonline.com/page/termsand-conditions

Theory Into Practice, 54:238–254, 2015 Copyright q The College of Education and Human Ecology, The Ohio State University ISSN: 0040-5841 print/1543-0421 online DOI: 10.1080/00405841.2015.1044374

Downloaded by [University North Carolina - Chapel Hill] at 07:52 22 July 2015

Kihyun Ryoo Marcia C. Linn

Designing and Validating Assessments of Complex Thinking in Science Typical assessment systems often measure isolated ideas rather than the coherent understanding valued in current science classrooms. Such assessments may motivate students to memorize, rather than to use new ideas to solve complex problems. To meet the requirements of the Next Generation Science Standards, instruction needs to emphasize sustained investigations, and assessments need to create a detailed picture of students’ conceptual understanding and reasoning processes. This article describes the design process and potential for automated scoring of 2 forms of

Kihyun Ryoo is an Assistant Professor in the School of Education, University of North Carolina, Chapel Hill and Marcia C. Linn is a Professor of Education in Mathematics, Science, and Technology at the University of California, Berkeley. Correspondence should be addressed to Professor Kihyun Ryoo, School of Education, University of North Carolina at Chapel Hill, 309E Peabody Hall, CB#3500, Chapel Hill, NC 27599-3500. E-mail: [email protected]. Color versions of one or more of the figures in the article can be found online at www.tandfonline.com/ htip.

238

inquiry assessment: Energy Stories and MySystem. To design these assessments, we formed a partnership of teachers, discipline experts, researchers, technologists, and psychometricians to align curriculum, assessments, and rubrics. We illustrate how these items document middle school students’ reasoning about energy flow in life science. We used evidence from review by science teachers and experts in the discipline; classroom experiments; and psychometric analysis to validate the assessments, rubrics, and automated scoring.

C

standards call for assessments that align with innovative inquiry instruction and measure the progress of students as they develop the ability to reason about complex everyday problems. We describe the design process for two important types of inquiry assessments, Energy Stories and MySystem, which we developed as part of a research program aimed at promoting cumulative understanding: Cumulative Learning using Embedded ONTEMPORARY SCIENCE

Downloaded by [University North Carolina - Chapel Hill] at 07:52 22 July 2015

Ryoo and Linn

Assessment Results (CLEAR). We conceptualize cumulative learning as a process of knowledge integration where students develop coherent ideas that they can readily use to solve new problems (Linn & Eylon, 2011). We describe the design of Energy Stories and MySystem for measuring middle school students’ reasoning about energy flow in life science and discuss how advances in technology make it possible to score student essays and conceptual models automatically (Linn et al., 2014; Ryoo & Linn, 2014). Students come to science classrooms with many varied ideas about any science topic. Students typically develop these ideas using scientific practices, such as experimenting (e.g., the plants in the shade died), observing the natural world (e.g., plants eat dirt), culturallymediated collaborations with family members (e.g., plants need food), and analysis of everyday events (e.g., the Sun warms the air so plants can grow). These ideas form a repertoire that is often contradictory and confusing (Clark, 2006; diSessa, 1988). The process of knowledge integration emphasizes eliciting the ideas that students initially have to help them build on and refine their repertoire. It emphasizes adding new, scientifically normative ideas about scientific phenomena to these existing ideas. Although some of these new ideas contradict students’ views, many students do not spontaneously compare the two to determine which ideas are most productive. Rather, students just memorize the new ideas, and their existing ideas remain isolated. Adding ideas without integrating them with other ideas leads to a fragmented and fractured understanding (Gilbert & Boulter, 2000). Knowledge integration emphasizes opportunities to distinguish new ideas from students’ existing repertoire of ideas by using scientific practices, such as experimenting using scientific visualizations or analyzing data. These activities help students develop criteria to differentiate their initial ideas from new, scientifically normative ideas. They allow students to gather evidence by comparing their predictions to the results from scientific investigations. Knowledge integration promotes coher-

Assessments of Complex Thinking in Science

ent understanding by giving students opportunities to reflect on the results of their efforts to distinguish ideas. Students can develop an integrated understanding of scientific phenomena by applying the criteria they developed, sorting out the evidence they gathered from instruction, organizing their ideas around new experiences, and synthesizing their views into a coherent account of a scientific phenomenon. Inquiry instruction that emphasizes this knowledge integration process gives students multiple opportunities to articulate their ideas, explore new ideas, compare alternatives, and sort out their views to develop coherent accounts of scientific phenomena.

Conventional Science Assessments Research has shown that conventional assessment items often fail to measure the kind of learning and complex thinking processes emphasized in inquiry learning (Gotwals, Hokayem, Song, & Songer, 2013; Lomax, West, Harmon, Viator, & Madus, 1995). Conventional assessments generally rely on multiple-choice items that focus on measuring factual knowledge about a single scientific concept, rather than assessing students’ integrated understanding of science (Lane, 2004; Songer, 2006). For instance, a typical item on photosynthesis from the Trends in International Mathematics and Science Study (TIMSS) addresses superficial features about the role of chloroplasts using a multiple-choice format and fails to accurately measure how students make connections among different energy ideas to comprehend why chloroplasts absorb light energy and how this light energy is needed to make food (Figure 1). These multiplechoice items show whether students can select the correct answer, but they do not allow students to demonstrate why they made such a decision or to describe the evidence they used (Ennis, 1993). In addition, most students are only tested on the topics they studied in the latest unit and often quite superficially; there is seldom the expectation of applying concepts learned in prior

239

Assessment of Complex Thinking

Downloaded by [University North Carolina - Chapel Hill] at 07:52 22 July 2015

Figure 1. Example of Typical Science Assessment Measuring Energy Ideas in Photosynthesis.

material. Thus, contemporary assessments rarely measure coherent ideas or cumulative learning. Constructed response items, such as essays and models, have the potential to measure integrated understanding but do not always succeed. For example, some items in the National Assessment of Educational Progress and TIMSS allow students to construct responses, but most of these items are scored as complete or incomplete, based on the accuracy of students’ recall of factual knowledge in their explanations (Liu, Lee, & Linn, 2011). This scoring system does not capture the integrated understanding students need to become cumulative learners. Furthermore, the emphasis on recall of details in high-stakes assessments often motivates teachers to drill on recall of factual knowledge, rather than helping students apply concepts in new contexts or make connections among ideas using evidence (Shepard, 2000; Yeh, 2006). Paradoxically, activities that require students to interpret, connect, and elaborate their ideas are actually the most efficient way to help students store details so they can later be recalled (Bjork, Dunlosky, & Kornell, 2013). Designing assessments and rubrics that require these activities helps students prepare for both recall and knowledge integration tests. By using assessments that align with the goals of knowledge integration instruction, designers can communicate to teachers that such activities are valuable (Liu, Lee, Hofstetter, & Linn, 2008; Pellegrino, Chudowsky, & Glaser, 2001). Importantly, assessments that require students to integrate their ideas actually extend the inquiry process; conventional tests often interrupt inquiry by

240

asking for recall of details. In addition, the opportunity to score these items automatically makes them useful for classroom instruction (Linn et al., 2014).

Designing Instruction and Assessments for Integrated Understanding Partnership Design To design assessment items that measure students’ integrated understanding, we formed a partnership of science teachers, discipline experts, researchers, technologists, and psychometricians to align instruction and assessment. The partnership process ensured that each member’s expertise was respected and contributed to the professional development of the other participants (see Slotta & Linn, 2009). To improve understanding of energy flow in life science, the partnership developed and refined technology-enhanced inquiry units and assessment items on photosynthesis and cellular respiration using the Web-based Inquiry Science Environment (WISE), an online delivery system that logs student responses and helps teachers monitor student progress. The partnership first identified core energy concepts related to photosynthesis and cellular respiration addressed in California standards (see Table 1). The partnership used the knowledge integration framework to develop inquiry activities and assessments that help students articulate their ideas, add new ideas using technological features such as visualizations, distinguish among their initial and new ideas, and reflect on

†

†

†

†

Energy Source and Storage Energy Transformation and Release

Energy Transfer

(continued)

Light energy is used to start chemical Animals eat plants and the Students know plants are the primary Light energy from the chemical energy stored in reactions between water and sun is the main source source of matter and energy entering plants transfers to them carbon dioxide; as a result, of energy for plants. most food chains. (Grade 4) through food chains. sugar and oxygen are being created. Students know plants use carbon Plants use glucose as an Light energy is transformed into dioxide (CO2) and energy from energy source. chemical energy which is stored in glucose during photosynthesis. sunlight to build molecules of sugar and release oxygen. (Grade 5) Students know energy entering ecosystems as sunlight is transferred by producers into chemical energy through photosynthesis and then from organism to organism through food webs. (Grade 7) Students know matter is transferred over time from one organism to others in the food web and between organisms and the physical environment. (Grade 7) Students know the utility of energy sources is determined by factors that are involved in converting these sources to useful forms and the consequences of the conversion process. (Grade 7)

Standards

Photosynthesis †

Unit

Energy Concepts Addressed in the Inquiry Units

Table 1

Downloaded by [University North Carolina - Chapel Hill] at 07:52 22 July 2015

Ryoo and Linn Assessments of Complex Thinking in Science

241

Assessment of Complex Thinking Animals eat plants and the chemical energy stored in plants transfers to them through food chains. Chemical energy stored in glucose is released as a usable form in mitochondria during cellular respiration. Plants get chemical energy stored in glucose during cellular respiration.

Energy Stories and MySystem

†

† Cellular Respiration

Students know plant and animal cells break down sugar to obtain energy, a process resulting in carbon dioxide (CO2) and water (respiration). (Grade 5) Students know that mitochondria liberate energy for the work that cells do and that chloroplasts capture sunlight energy for photosynthesis. (Grade 7)

Energy Transfer Energy Transformation and Release Energy Source and Storage Standards Unit

Table 1 – (Continued)

Downloaded by [University North Carolina - Chapel Hill] at 07:52 22 July 2015

242

these experiences to coherently connect energy concepts and explain the processes of photosynthesis and cellular respiration (see Ryoo & Linn, 2012, 2014, for more details). The interactive, dynamic visualizations enabled students to explore abstract concepts of energy, such as where energy comes from, how energy is transformed and released, and how energy moves in the context of photosynthesis and cellular respiration (see Figure 2). The partnership drew on expert reviews by teachers and content experts to improve the activities, embedded assessments, and associated technologies, as well as to refine the sequence of the activities.

The partnership developed Energy Stories and MySystem to align with the goal of cumulative learning and the knowledge integration framework (Ryoo & Linn, 2010). These items extend prior knowledge integration assessments by supporting longer chains of reasoning and allowing students to express their ideas graphically (Lee, Liu, & Linn, 2011; Liu et al., 2008). This article illustrates how these item types can capture students’ sophisticated understanding of energy flow for photosynthesis and cellular respiration. Energy Stories ask students to reflect on the concepts they have learned and create narratives about energy concepts in domain-specific, everyday contexts. Energy Stories require more elaborate and sustained reasoning than prior knowledge integration items because, when writing Energy Stories, students incorporate various energy ideas into their everyday experiences. To create a successful story, students need to reflect on energy concepts from both photosynthesis and cellular respiration and generate a coherent narrative account of where plants get energy to grow, how energy is transformed, stored, and released, and where energy goes. Students are rewarded for elaborated links among ideas, clear connections between details, and complete arguments. MySystem assesses similar types of reasoning to those required for Energy Stories, but allows

Downloaded by [University North Carolina - Chapel Hill] at 07:52 22 July 2015

Ryoo and Linn

Assessments of Complex Thinking in Science

Figure 2. Screenshots of WISE Photosynthesis and Cellular Respiration Units.

students to graphically represent their understanding of how energy flows within a system using icons, arrows, and labels (Figure 3). MySystem captures students’ models of the connections among energy concepts, such as energy transformation, energy storage, energy release, and energy transfer. Students can drag icons into the workspace and connect these icons

with arrows to depict how energy moves from one representative icon to another. Students assign different colors to arrows to identify which type of energy is involved and to show how energy is transformed. Students can also annotate the relationships between energy concepts to provide more details. Compared to prior knowledge integration items, MySystem challenges

Figure 3. MySystem Interface.

243

Downloaded by [University North Carolina - Chapel Hill] at 07:52 22 July 2015

Assessment of Complex Thinking

students to translate their verbal understanding of energy flow into a visual representation and accurately depict the type of energy and form of energy flow for a complex process. Aligning with the main energy concepts emphasized in the photosynthesis and cellular respiration units, Energy Stories asked students to write a story about how a rabbit receives and uses energy from the Sun, and MySystem diagrams asked students to visually represent the same concepts as the Energy Stories. These assessments extend the knowledge integration activities in the instruction by encouraging students to reflect on the findings from inquiry activities and integrate multiple ideas they have learned. When used as embedded assessments, they give teachers an indication of students’ cumulative understanding of energy concepts at various stages of instruction. Developing Scoring Rubrics Knowledge integration scoring rubrics reward students for building a coherent narrative using normative energy ideas (Liu et al., 2011). The rubrics indicate scientifically valid links among ideas rather than detecting correct or incorrect ideas. The partnership created scoring rubrics for Energy Stories and MySystem diagrams by first analyzing students’ responses and identifying ways students linked ideas about energy source, energy transformation, energy storage, energy release, and energy transfer in photosynthesis and cellular respiration. We then distinguished four categories of links among these ideas: (a) energy transformation (how plants convert light energy into chemical energy), (b) energy storage (how chemical energy is stored in glucose, which then becomes an energy source), (c) energy release (how plants release energy stored in glucose), and (d) energy transfer (how energy is transferred from plants to other living organisms). The rubric rewards students for generating these links and combining them in a coherent way (see Table 2). To accommodate the complexity of Energy Stories and MySystem, we added one additional level called advanced complex understanding to the typical knowledge integration rubric. Based

244

on the number and coherence of scientifically valid links and normative energy ideas, the scores range from 1 (irrelevant or off-task) to 6 (advanced complex understanding). The knowledge integration rubric, thus, captures cumulative understanding of energy flow, rather than measuring isolated details about energy. Evaluating and Validating Assessments To validate Energy Stories and MySystem diagrams, we implemented them as pretestposttest measures in two middle school inquiry units on photosynthesis and cellular respiration. To establish the content and face validity of the items, we conducted an expert review. Discipline experts reviewed the items and rubrics and attested to their content validity. Expert science teachers reviewed the items and curricular units to ensure face validity. We studied implementation of the photosynthesis and cellular respiration units taught by two seventh-grade science teachers to 138 students in five classes for 15 days. We investigated instructional validity by determining how well the items captured student progress in making connections among multiple energy concepts and in explaining energy flow. We established baseline performance on a pretest and measured student progress after completing the photosynthesis unit and again after studying the cellular respiration unit. We analyzed overall knowledge integration scores, as well as students’ ability to make specific energy links emphasized in the units using the knowledge integration rubrics. We also analyzed four types of scientifically valid links identified in Energy Stories and MySystem diagrams across the pretest, postphotosynthesis test, and postcellular-respiration test. To measure student progress in understanding the flow of energy in life science, we compared student performance on Energy Stories and MySystem diagrams across the three tests using a repeated measures analysis of variance (ANOVA). The results revealed a significant main effect of time for both Energy Stories, F(2, 274) ¼ 179.58, p , .001, h2p ¼ .57, and MySystem, F(2, 274) ¼ 217.38, p , .001, h2p ¼ .61,

No answer or off-task

Off-task

No Link

1

2

Non-normative or scientifically invalid links and ideas

Description

Score KI Level

Mary, Plant use the heat energy from the sun to grow. Plants need sunlight, water, and fresh soil to grow.

I don’t know.

Sample Student Responses: Energy #Stories Sample Student Responses: MySystem

Knowledge Integration Rubric for Energy Stories and MySystem

Table 2

Downloaded by [University North Carolina - Chapel Hill] at 07:52 22 July 2015

(continued)

Ryoo and Linn Assessments of Complex Thinking in Science

245

246

Normative ideas without scientifically valid connections between ideas

Partial Link

Full Link

3

4

One scientifically valid and elaborated link between normative and relevant energy ideas

Description

Score KI Level

The energy comes from the sun. It is transformed using chloroplasts from light energy in to chemical energy in order to make glucose. [Energy Transformation] The plant uses the glucose to make more and the bunny eats the plant getting the glucose to feed itself.

The plants get there energy from the sun because that where we all get our energy. Also the energy is transformed from the sun letting us have it. The energy ends up in plants, lights in houses, all kinds of things that need energy to work. That’s were energy comes from and why we need it.

Sample Student Responses: Energy #Stories Sample Student Responses: MySystem

Table 2 – (Continued)

Downloaded by [University North Carolina - Chapel Hill] at 07:52 22 July 2015 Assessment of Complex Thinking

5

Complex Link

2 scientifically valid and The sun rays is absorbed into the plant’s chloroplast’s chlorophyll. elaborated links The sunlight doesn’t turn into between normative chemical energy until after it and relevant broke the co2 and h2o molecules. energy ideas [Energy Transformation] Then the molecules form together as glucose and the sunlight is turned into chemical energy and stored in the glucose made. Some of the glucose goes to the plant’s mitochondria and rest stay in the plants cells. [Energy Storage]

Downloaded by [University North Carolina - Chapel Hill] at 07:52 22 July 2015

(continued)

Ryoo and Linn Assessments of Complex Thinking in Science

247

248

6

Description

Advanced 3 or more scientifically valid and elaborated Complex links between Link normative and relevant energy ideas

Score KI Level The energy originally comes from the sun, in the form of light energy. While it is in the chloroplasts, it breaks up CO2 and H2O molecules, and in the meantime coverts into chemical energy. [Energy Transformation] When glucose forms from the CO2 and H2O’s atoms, the chemical energy then enters the glucose. [Energy Storage] In the mitochondria, the glucose molecules are broken up and this chemical energy is released to be used by the plant. [Energy Release] When the rabbit eats the plant, it is harvesting the chemical energy that has already been released, as well as the energy that is still in the glucose, all of which it can use to function and live. [Energy Transfer]

Sample Student Responses: Energy #Stories Sample Student Responses: MySystem

Table 2 – (Continued)

Downloaded by [University North Carolina - Chapel Hill] at 07:52 22 July 2015 Assessment of Complex Thinking

Downloaded by [University North Carolina - Chapel Hill] at 07:52 22 July 2015

Ryoo and Linn

Assessments of Complex Thinking in Science

indicating that students improved significantly in their cumulative understanding of energy flow in life science over time. Mean scores and standard deviations on the 3 tests are presented in Table 3. The results show that students demonstrated progress in their knowledge integration as they completed the two units. They performed significantly better on the photosynthesis posttest than they did on the pretest on both Energy Stories ( p , .001, d ¼ 1.45) and MySystem ( p , .001, d ¼ 1.60). Students also demonstrated significant gains from the photosynthesis posttest to the cell respiration posttest with a small effect size for Energy Stories ( p , .01, d ¼ 0.22) and MySystem ( p , .05, d ¼ 0.15). Student responses showed that, consistent with the knowledge integration perspective, both Energy Stories and MySystem diagrams captured the wide range of ideas students bring to science classrooms and documented how students developed a more integrated understanding of energy flow over time by adding normative ideas and generating coherent explanations (see examples in Table 4). They were also effective in documenting more complex and elaborated ideas than the knowledge integration items used in earlier research (Ryoo & Linn, 2012). Students demonstrated increased understanding of how energy flows by gradually making more elaborated connections among multiple energy ideas (Figure 4). For instance, on the photosynthesis posttest, many students articulated how plants convert light energy from the Sun to chemical energy with detailed descriptions of chemical reactions, such as how light energy breaks up carbon dioxide and water, in their posttest Energy Stories, reflecting their

experience with the dynamic visualizations. Extending their understanding from the postphotosynthesis test, students were better able to illustrate using MySystem, that not only plants, but also the rabbit, obtain chemical energy from glucose during cellular respiration on the postcellular respiration test. These item types build on prior knowledge integration assessments to document cumulative learning and serve linguistically diverse students including English language learners (ELLs), who speak a language other than English at home as their primary language. Energy Stories reward students for generating a more complex sequence of scientifically valid links than typical knowledge integration items to explain photosynthesis and cellular respiration. MySystem diagrams allow students to express their ideas about energy flow using icons, links, and arrows, reducing the language burden for ELLs and supporting a new form of communication. For instance, on the pretest, ELLs (M ¼ 3.24, SD ¼ 1.08) performed similarly to non-ELLs (M ¼ 3.23, SD ¼ 1.03) on MySystem but started with lower scores on Energy Stories (ELLs M ¼ 3.20, SD ¼ 0.81; non-ELLs M ¼ 3.25, SD ¼ 0.91). A validation study showed that both items showed similar, significant gains for ELLs and non-ELLs. The learning gains from the pretest Energy Stories to the final posttest Energy Stories were 1.80 (SD ¼ 1.37) for ELLs ( p , .001, d ¼ 1.82) and 1.82 (SD ¼ 1.38) for non-ELLs ( p , .001, d ¼ 1.82). The learning gains from the pretest MySystem to the final posttest MySystem were 1.88 (SD ¼ 1.30) for ELLs ( p , .001, d ¼ 1.74) and 1.86 (SD ¼ 1.38) for non-ELLs ( p , .001, d ¼ 1.69).

Table 3 Mean Scores and Standard Deviations Across the Three Tests

Pretest Photosynthesis posttest Cellular respiration posttest

Energy Stories

MySystem

3.23 (0.87) 4.78 (1.23) 5.04 (1.10)

3.23 (1.04) 4.93 (1.08) 5.10 (1.14)

249

Assessment of Complex Thinking

Table 4 Examples of Three Energy Stories and MySystem Diagrams Across the Units

Downloaded by [University North Carolina - Chapel Hill] at 07:52 22 July 2015

Example Pretest

The energy comes from the sun that goes into the chloroplast of the plant cells. There the energy gets transformed into plant food.

Photosynthesis posttest

Once upon a time there was a plant. The plant had a job of making food for his rabbit. To make energy-rich food for the rabbit, the plant’s chloroplasts collected water, light energy, and CO2 for the food that is called glucose. Glucose is a type of sugar used to help plants stay alive and grow. In the plant’s chloroplast, the light energy breaks apart the little water and CO2 atoms. When the light energy broke apart the atoms, it turned into chemical energy that decided to go inside the glucose and help give nutrients to the rabbit ant the plant. The glucose traveled around the body and the extra ones were stored in the roots and leaves. When the rabbit ate the plant, the plant gave away some of the glucose. Even though it was very complicated to make glucose, the plant was happy to be generous.

(continued)

250

Ryoo and Linn

Assessments of Complex Thinking in Science

Table 4 – (Continued)

Downloaded by [University North Carolina - Chapel Hill] at 07:52 22 July 2015

Example Cell respiration Once upon a time, there was a plant. The plant fed a rabbit named Bobby. Bobby ate the plant’s posttest nutritious leaves full of glucose. The plant woke up every morning just before dawn. To make glucose, the plant needs sunlight, water, and CO2. The plant collected sunlight and CO2 with his leaves. His roots soaked up the water. All the CO2, water, and sunlight gathered up the chloroplast where the water and CO2 atoms were broken apart and the sunlight was changed into chemical energy. The atoms were broken apart and the sunlight was changed into chemical energy. The atoms were assembled into glucose molecules and the chemical energy was stored in the glucose. The leftover atoms were made into oxygen. The glucose was stored in the leaves and roots for later use. Then Bobby came along and ate a leaf from the plant. The leaf went into Bobby’s stomach and his stomach cells extracted the glucose from the leaf. The glucose went into the mitochondria with oxygen and the glucose and oxygen were broken apart into smaller pieces (atoms). The chemical energy was taken away from the glucose and the atoms of the glucose and oxygen molecules assemble themselves into CO2 and water molecules.

Teacher Perceptions As further evidence for face validity, we conducted semistructured interviews with the two participating teachers individually for 45 min after they implemented the two inquiry units. Interview protocols focused on teachers’ use of the Energy Stories and MySystem assessments to customize their instruction to build on students’ ideas (i.e., “How did you help students make better connections about energy flow using MySystem?”), and teachers’ views on potential advantages and disadvantages of the MySystem

and Energy Stories assessments (i.e., “What did you find valuable about Energy Stories? In comparison to other assessment items, how does what you learn about students from MySystem and Energy stories differ?”). Each interview was audio recorded and fully transcribed for analysis. Two science teachers who implemented these assessments reported that MySystem and Energy Stories provide valid indicators of student performance. Both teachers reported that they were able to recognize that many students focused on the detailed processes captured by specific energy link categories, such as how

251

Downloaded by [University North Carolina - Chapel Hill] at 07:52 22 July 2015

Assessment of Complex Thinking

Figure 4. The Number of Scientifically Valid Links Identified in Energy Stories and MySystem.

plants absorb light energy from the Sun and what happens to light energy, water, and carbon dioxide in the chloroplast, rather than just explaining how energy moves from one place to another. In particular, teachers saw the potential of the Energy Stories as a way to assess more complex thinking in science. For example, one of the participating teachers mentioned that “the energy story is asking students to apply information. That’s very high-level of thinking.” The participating teachers also appreciated that MySystem gave students a new way to communicate their understanding of scientific concepts using a visual format. One of the teachers reported that allowing students to express their understanding with arrows and icons “gives them more flexibility and more creativity in being able to show what they know.” Another teacher also appreciated the advantage of MySystem diagrams as a way to measure more

252

complex thinking in students’ understanding by stating: I saw a big change from the pretest to . . . the [posttest] MySystem diagrams they were doing. They got much more sophisticated in their labels. They got more sophisticated in the way that they colored their arrows. They started to understand what are just connections and what are energy connections. So I thought that was really clear evidence of student learning and understanding.

Automated Scoring A common drawback to constructed response items, such as Energy Stories and MySystem, is that they are time-consuming to score. Teachers often have five or six classes of 30 to 40 students. Recent research shows that by taking advantage

Downloaded by [University North Carolina - Chapel Hill] at 07:52 22 July 2015

Ryoo and Linn

Assessments of Complex Thinking in Science

of natural language processing and developing algorithms to analyze MySystem diagrams, these types of items can be automatically scored (Linn et al., 2014). Furthermore in WISE units, these scores can be used to provide personalized guidance based on students’ ideas presented in their narratives and concept diagrams (Ryoo & Linn, 2014). Automated guidance on Energy Stories and MySystem diagrams can increase the impact of knowledge integration instruction by guiding learners, alerting teachers to students who are struggling, and freeing teachers to focus on those who most need help.

Conclusions Energy Stories and MySystem diagrams offer powerful inquiry assessments that can dramatically improve instruction and assessment. Content, face, and instructional validity studies show that the item types capture a detailed picture of students’ integrated understanding of scientific phenomena. Importantly, ELLs and non-ELL students started at the same level on MySystem and made similar, significant gains. These items, implemented in WISE, are available to teachers everywhere. WISE is open source and free. Many WISE units are available in Chinese and Spanish. Currently close to 10,000 teachers and 100,000 students use WISE. Energy Stories and MySystem diagrams strengthen instruction by adding knowledge integration activities rather than interrupting students to ask for recall of details. These items can be used to elicit ideas on pretests, to distinguish ideas when embedded in units, and to motivate reflection when featured on posttests. These item types extend the knowledge integration assessment options to measure more complex, sustained reasoning than previous item types. Automated scoring technologies can personalize instruction based on student scores on these items. Knowledge integration guidance can encourage students to monitor their own progress, fill in gaps in their understanding, and help teachers keep track of student activities. The

knowledge integration rubrics show a learning trajectory that students might follow in integrating energy transformation, energy storage, energy release, and energy transfer concepts in the context of photosynthesis and cellular respiration and can help teachers guide students. These items combined with automated scoring technologies illustrate for policy makers new possibilities for effective use of technology in science instruction. In conclusion, this article illustrates valid and promising ways to assess knowledge integration that contribute to learning outcomes. Using narrative explanations in Energy Stories and graphic representations in MySystem broaden opportunities to demonstrate understanding. These items document levels of understanding and help students, teachers, and policy makers appreciate the trajectories that students might follow to become cumulative learners. These assessments use evidence gathered during instruction rather than on one-shot state and national tests that are often poorly aligned with instruction. Energy Stories and MySystem provide a useful diagnostic tool and informative indicators to understand how students connect multiple scientific ideas and gradually develop complex thinking in science.

References Bjork, R. A., Dunlosky, J., & Kornell, N. (2013). Selfregulated learning: Beliefs, techniques, and illusions. Annual Review of Psychology, 64, 417– 444. Clark, D. B. (2006). Longitudinal conceptual change in students’ understanding of thermal equilibrium: An examination of the process of conceptual restructuring. Cognition and Instruction, 24, 467– 563. diSessa, A. A. (1988). Knowledge in pieces. In G. Forman & P. Pufall (Eds.), Constructivism in the computer age (pp. 49 – 70). Hillsdale, NJ: Erlbaum. Ennis, R. H. (1993). Critical thinking assessment. Theory into Practice, 32, 179– 186. Gilbert, J. K., & Boulter, C. J. (2000). Developing models in science education Dordrecht. The Netherlands: Kluwer.

253

Downloaded by [University North Carolina - Chapel Hill] at 07:52 22 July 2015

Assessment of Complex Thinking Gotwals, A. W., Hokayem, H., Song, T., & Songer, N. (2013). The role of core disciplinary ideas and practices in the complexity of large-scale assessment items. Electronic Journal of Science Education, 17, 1 –24. Lane, S. (2004). Validity of high-stakes assessment: Are students engaged in complex thinking? Educational Measurement: Issues and Practice, 23, 6 – 14. Lee, H. S., Liu, O. L., & Linn, M. C. (2011). Validating measurement of knowledge integration in science using multiple-choice and explanation items. Applied Measurement in Education, 24, 115 –136. Linn, M. C., & Eylon, B. -S. (2011). Science learning and instruction: Taking advantage of technology to promote knowledge integration. New York, NY: Routledge. Linn, M. C., Gerard, L., Ryoo, K., McElhaney, K., Liu, O. L., & Rafferty, A. N. (2014). Computer-guided inquiry to improve science learning. Science, 344, 155– 156. Liu, O. L., Lee, H. S., Hofstetter, C., & Linn, M. C. (2008). Assessing knowledge integration in science: Construct, measures and evidence. Educational Assessment, 13, 33 – 55. Liu, O. L., Lee, H. S., & Linn, M. C. (2011). Measuring knowledge integration: Validation of four year assessments. Journal of Research in Science Teaching, 48, 1079– 1107. Lomax, R. G., West, M. M., Harmon, M. C., Viator, K. A., & Madaus, G. F. (1995). The impact of mandated standardized testing on minority students. Journal of Negro Education, 64, 171– 185. Pellegrino, J., Chudowsky, N., & Glaser, R. (2001). Knowing what students know: The science and design of educational assessment. Washington, DC: National Academy Press.

254

Ryoo, K., & Linn, M. C. (2010). Students’ progress in understanding energy concepts in photosynthesis using visualizations. In K. Gomez, L. Lyons, & J. Radinsky (Eds.), Learning in the disciplines, proceedings of the 9th international conference of the learning sciences (Vol. 2, pp. 480 – 481). Chicago, IL: International Society of the Learning Sciences. Ryoo, K., & Linn, M. C. (2012). Can dynamic visualizations improve middle school students’ understanding of energy in photosynthesis? Journal of Research in Science Teaching, 49, 218– 243. Ryoo, K., & Linn, M. C. (2014). Comparison of specific and knowledge integration automated guidance for concept diagrams in inquiry instruction. In J. L. Polman, E. A. Kyza, D. K. O’Neill, I. Tabak, W. R. Penuel, A. S. Jurow, K. O’Connor, T. Lee, & L. D’Amico (Eds.), Learning and becoming in practice, proceedings of the 11th International Conference of the Learning Sciences (Vol. 3, pp. 1585– 1586). Boulder, CO: International Society of the Learning Sciences. Shepard, L. A. (2000). The role of assessment in a learning culture. Educational Researcher, 29, 4 – 14. Slotta, J. D., & Linn, M. (2009). WISE Science: Webbased inquiry in the classroom. New York, NY: Teachers College Press. Songer, N. B. (2006). BioKIDS: An animated conversation on the development of curricular activity structures for inquiry science. In R. Keith Sawyer (Ed.), Cambridge handbook of the learning sciences (pp. 355– 369). New York, NY: Cambridge University Press. Yeh, S. S. (2006). Tests worth teaching to: Constructing state-mandated tests that emphasize critical thinking. Educational Researcher, 30, 12 – 17.