Multimedia displays for conceptual discovery - Springer Link

Multimedia Systems (2006) 11(3): 236–248 DOI 10.1007/s00530-005-0004-y

R E G U L A R PA P E R

Kirsten R. Butcher · Sonal Bhushan · Tamara Sumner

Multimedia displays for conceptual discovery: information seeking with strand maps

Published online: 8 February 2006 c Springer-Verlag 2005

Abstract This article explores the use of a multimedia search interface for digital libraries based on strand maps developed by the American Association for the Advancement of Science. As semantic-spatial displays, strand maps provide a visual organization of relevant conceptual information that may promote the use of science content during digital library use. A study was conducted to compare users’ cognitive processes during information seeking tasks when using a multimedia strand maps interface, versus the textual search interface currently implemented in the Digital Library for Earth System Education. Quantitative and qualitative data from think-aloud protocols revealed that students were more likely to engage with science content (e.g., analyzing the relevance of science concepts with regard to task needs) during search when using the strand maps interface compared to those using textual searching. In contrast, students using a textual search interface engaged more frequently with surface-level information (e.g., the type of a resource regardless of its science content) during search and retrieval. As a multimedia search interface for digital libraries, strand maps appear to be promising tools to promote conceptual discovery and learning through content-based processes that promote learner engagement with relevant science knowledge. Keywords Information seeking · Strand maps · Educational digital libraries · Science learning · Educational technology 1 Introduction The continuing growth of online educational resources has meant that learners have an increasingly important need for K. R. Butcher (B) · S. Bhushan Digital Library for Earth System Education Program Center, University Corporation for Atmospheric Research, P.O. Box 3000, Boulder, CO 80307 E-mail: {kbutcher, sonal}@ucar.edu T. Sumner Department of Computer Science, University of Colorado at Boulder, Campus Box 430, Boulder, CO 80309-0430 E-mail: [email protected]

discovery methods that help them to explore and retrieve relevant information. As services that bring together, organize, and support connections to diverse collections of educational materials, digital library systems have great promise as tools to support online learning. However, as successful as digital libraries have been to date, retrieval of relevant resources remains a significant problem in educational settings. Especially in science, subtle differences between concepts can make it difficult for learners without domain expertise to generate relevant search terms or even to effectively select the appropriate standard vocabulary from topic lists. Lacking relevant knowledge, learners often resort to selecting keywords from assignments or existing materials [1]. Previous research has found that science educators also experience difficulties locating digital learning materials [2]. Many K-12 science educators frequently must teach out of area in topic domains for which they lack training or confidence [3]. When considering both educators and learners, the number of individuals who come to digital libraries with low domain knowledge is strikingly high. As such, digital libraries should be designed to serve as cognitive tools that support library users to engage in conceptual learning, in addition to supporting information search [4]. Our research efforts are exploring how conceptual browsing interfaces can support learning of science concepts during digital library search tasks. We are creating concept browsing interfaces and supporting web services based on the science concepts and strand maps developed by the American Association for the Advancement of Science [5, 6]. These concepts represent nationally recognized learning goals developed through a long-term collaborative process involving pedagogical experts, scientists, science educators, and school districts across the country [7, 8]. The strand maps provide an ideal foundation for conceptual browsing interfaces because they provide a visual representation of science concepts that supports students and educators in making connections between key ideas. The context for our research is the National Science Digital Library (NSDL). NSDL is intended to support


educational science reforms by providing educators and learners with online access to a large number of educational materials. NSDL encompasses a number of disciplinespecific digital library systems, including the communityled Digital Library for Earth System Education (D LESE). D LESE seeks to support learning and research in Earth system science education by providing interactive and effective access to high-quality online educational resources for all educational levels in both formal and informal settings. Given the diversity of their user base, NSDL and D LESE face significant challenges in connecting learners to the relevant conceptual knowledge and educational interventions that are appropriate to their specific informational needs and learning goals [9, 10]. In this article, we first describe prior work demonstrating learner difficulties with information search and discuss the theoretical rationale for conceptual browsing interfaces as tools to overcome these observed difficulties. Next, we describe one of the interfaces we have developed to provide conceptual support for search tasks. Finally, we report the results of a study investigating the effects of this interface on the cognitive processes performed by users when using this conceptual browsing interface versus an existing textual search interface.

2 The challenge of conceptual discovery Educational digital libraries face an extraordinary challenge in finding successful ways to support conceptual discovery. In this article, we use the term conceptual discovery to refer to the ability of learners to find new information and ideas that can inform and refine their conceptual understandings of a specific knowledge domain. As such, conceptual discovery requires more than simple information retrieval. It requires learners to make connections between the content of library resources and their specific knowledge needs, and to consider the relationship of new information to the larger knowledge domain. Previous research illustrates that issues of learners’ prior knowledge, problem definition, and conceptual organization must be addressed in order to promote learning during information seeking tasks. However, the same research highlights the shortcomings of textual search interfaces for supporting these learner needs. In early stages of knowledge development, learners lack the requisite knowledge for generating the specialized and varied vocabulary necessary to locate relevant digital resources using keywords or phrases [11–13]. Kuhlthau [11] examined the information search processes of students writing academic papers and found that initial search strategies were characterized by confusion, uncertainty, and a lack of focused knowledge. Given their lack of knowledge, students often turned to physical methods to discover conceptually relevant information and to gain a better understanding about the topic over time. For example,

237

students used card catalogs to find potentially topic-relevant shelves in the library, which they would then browse to locate information that they had not identified by keyword searches or other formal strategies [11]. More recent research has found that even quite knowledgeable individuals in a domain can be limited by the failure of digital information databases to support conceptual browsing and discovery [12]. Kuhlthau and Tama found that lawyers doing research preferred printed texts over electronic retrieval mechanisms largely because electronic databases (1) failed to display information in a useful manner relevant to the learning task, (2) limited discovery due to strict requirements of keyword searching, and (3) failed to organize information in meaningful ways that prevented the user from becoming lost in digital resources. These results suggest that even experts in a content area may benefit from interfaces that are strategically designed to promote conceptual discovery through browsing, especially if the support is organized and presented in ways that are meaningful for a particular type of knowledge need. Kuhlthau’s [11, 12] research highlights the challenges of effective information seeking in digital libraries: lacking physical strategies that users can fall back on, digital library search interfaces must resolve long-standing problems resulting from discovery methods that rely upon user input for information retrieval. Because learners lack sufficient domain knowledge to generate reasonable search terms and because specialized expertise within a system is often necessary to know the correct terms with which resources are indexed, there is a fundamental mismatch between users’ vocabularies and system vocabulary [14].

2.1 Textual search in digital libraries The success of textual search interfaces has been considerable and likely stems from their simplicity and familiarity; users tend to be experienced and comfortable with online textual search methodologies. The existing D LESE search interface (www.dlese.org) is an example of a successful textual interface onto a digital library discovery system (see Fig. 1). As seen in the top panel of Fig. 1, the textbox provided on the left-hand side of the homepage allows learners to search for resources using keywords, phrases, or other terms chosen by the user. D LESE also supports the use of optional vocabulary terms (as seen in the bottom panel of Fig. 1) to focus the search results based on the type of resources needed; for example, users can search for resources that include lesson plans and scientific illustrations. Experienced digital library users with ample knowledge of a science domain may prefer textual search interfaces, such as the D LESE interface, for their simplicity and efficiency. However, as discussed above, many users in educational settings do not have adequate science understanding and can benefit from interfaces that support conceptual discovery.

238

K. R. Butcher et al.

Fig. 1 The Dlese textual search interface.

3 Approach: semantic-spatial displays Multimedia search interfaces have the potential to change learners’ interactions with digital libraries by providing multiple forms of cognitive support for exploration and evaluation of knowledge and its organization in a domain. When appropriately designed, multimedia interfaces can change the ways that learners pursue information by engaging them early on with the science content that informs conceptual discovery. Although multimedia can include a wide variety of resources, including interactive visual displays, sounds, pictures, animations, voice recognition, and other complex

sensory experiences, we define multimedia tools as those that include at least two modes of sensory experience. This definition is similar to that proposed by Mayer [15], who argues that multimedia learning involves processing of verbal and visual components. Previous research in cognitive psychology and education has repeatedly demonstrated benefits when well-designed multimedia tools are used for learning [15–19]; however, research has not addressed whether multimedia interfaces can change the cognitive behaviors of users seeking information in digital environments. We are particularly concerned with supporting learners who have limited domain knowledge to engage with


science content during digital library searches in order to promote opportunities for learning during search tasks. Previous research has shown that qualitatively different search processes characterize experts and domain novices; whereas students with little domain knowledge spend more time formatting and modifying queries, experts spend more time scanning and reading texts [20]. In short, domain novices tend to engage in surface-based processes whereas experts tend to engage in content-based processes. It is not terribly surprising that experts are better able to find relevant resources by analyzing informational content; however, it does suggest that less knowledgeable students can better approximate expert behavior by attending to science content early and often during a search. Surface-based search and evaluation processes are characterized by the tendency to focus on aspects of the interface that are not relevant to domain information (e.g., components of the resource such as pictures or activities regardless of their science content). In contrast, content-based processes are characterized by a focus on science content that is related to the task at hand (e.g., assessing the relevance of plate tectonics resources in preparing a lesson plan on the cause of earthquakes). Promoting the use of content-based processes during digital library search can be considered an important first step in promoting domain learning—that is, the progression from surface-based to content-based search and evaluation processes represents an important step toward reasoning with science content. The attempt to match information needs with science concepts characterizes a type of reasoning with conceptual information that should promote understanding. Further, the increased use of contentbased processes rather than surface-based processes characterizes a movement toward more expert behaviors that focus on the conceptual relevance of resources during information seeking. Semantic-spatial displays refer to a multimedia representation in which information is presented both verbally (the semantic content) and visually (the spatial representation of the content). Although the use of semantic-spatial representations is a relatively novel approach to facilitating contentbased search processes in digital libraries, previous work in psychology and education has addressed the cognitive impact of these types of representations for learning. Knowledge maps refer to a particular type of semanticspatial display in which content information or ideas are located in nodes and labeled links are used to depict the relationships between nodes; as such, knowledge maps reflect both the conceptual information in a domain and its theorized cognitive structure. Previous research on knowledge maps has found that organized visual representations of semantic information can change learning processes and learning outcomes when compared to text-only representations. Knowledge maps repeatedly have been found to facilitate students’ memories for central ideas (also referred to as the macrostructure of a text) when knowledge maps are used during learning [21–23]. Further, student engagement with knowledge maps in a domain can transfer to learning in

239

other formats; explicit prior training with knowledge maps has been found to improve memory for non-mapped information studied in a text format [21]. Overall, the body of research evidence on knowledge maps suggests that these semantic-spatial tools may help students identify and integrate key concepts during learning. Interestingly, the benefits of knowledge maps have been shown to be greatest for students with low verbal ability or low prior knowledge of a domain [22]. The usefulness of knowledge maps for learners with limited domain knowledge has important potential applications for digital library development. Engaging learners with science content requires support in helping learners identify and understand key information. As multimedia displays that visually organize conceptual information according to underlying relationships, semantic-spatial displays may provide the right types of support to promote content-based processing of domain information by learners with limited domain knowledge. Strand maps are an example of semantic-spatial displays and are similar to knowledge maps in their form and structure (Fig. 2). These maps consist of node-link diagrams organized around topics important to science literacy (e.g., weather and climate, flow of energy in ecosystems, conservation of matter). Each strand map provides an overview of K-12 learning goals for a particular topic organized into “strands” reflecting key concepts within that map (e.g., heat, water cycle, atmosphere, and climate change are strands within the Weather and Climate Map). Each strand is further cross-referenced by grade level (K-2, 3–5, 6–8, 9–12). High-level descriptions of science concepts are provided in the nodes, while the links depict how science concepts both support and depend upon each other. These links between concepts illustrate how learners’ understandings should become increasingly sophisticated over the course of their education. It should be noted that strand maps are not the only multimedia interface that could support conceptual discovery in digital libraries; we use strand maps as a test case for assessing the effectiveness of a multimedia search interface in supporting conceptual discovery for educational tasks.

4 The Strand Map Service To investigate the potential benefits of multimedia interfaces for conceptual discovery, we have implemented and evaluated a system based on the strand maps just described. The Strand Map Service (the “Service”) supports the needs of two audiences: K-12 educators and learners, and digital library developers. These audiences are supported through the provision of two kinds of public interfaces: (1) graphical concept browsing interfaces for end-users, and (2) a programmatic web service interface that allows digital library developers to easily create concept browsing interfaces for use in their own libraries.

240


6-8 Heat can be transferred through materials by the collisions of atoms or across space by radiation. If the material is fluid, [such as air or water], currents will be set up in it that aid the transfer of heat. 4E/M3

3-5

When warmer things are put with cooler ones, the warm ones lose heat and the cool ones gain it until they are all at the same temperature. A warmer object can warm a cooler one by contact or at a distance. 4E/E2

K-2 The sun warms the land, air,and water. 4E/P1

to and from CONSERVATION OF ENERGY

to and from CONSERVATION OF MATTER to and from STATES OF MATTER

The cycling of water in and out of the atmosphere plays an important role in determining climatic patterns. Water evaporates from the surface of the earth, rises and cools, condenses into rain or snow, and falls again to the surface. The water falling on land collects in rivers and lakes, soil, and porous layers of rock, and much of it flows back into the ocean. 4B/M7

When liquid water disappears, it turns into a gas (vapor) in the air and can reappear as a liquid when cooled, or as a solid if cooled below the freezing point of water. Clouds and fog are made of tiny droplets of water . 4B/E3

Water can be a liquid or a solid and can go back and forth from one form to the other. If water is turned into ice and then the ice is allowed to melt, the amount of water is the same as it was before freezing. 4B/P2

heat

Water left in an open container disappears, but water in a closed container does not disappear. 4B/P3

This is a section of a map called “Weather and Climate.” The whole map consists of 22 concepts, 7 of which are shown here. The arrows indicate how one concept supports the ideas in the next concept. Dotted lines show connections to other maps (e.g., Conservation of Matter). The three boxes on the left side form a strand called “heat” and the four boxes on the right side form a strand called “water cycle.” Three grade ranges are shown on the map. For example, the three benchmarks at the bottom of the map are for grades K-2. The full map extends into grades 9-12.

water cycle

Fig. 2 Excerpt of strand map on weather and climate.

4.1 End-user interfaces Graphical concept browsing interfaces based on the Service enable K-12 educators and learners to: • Discover educational resources that support selected concepts • Browse concepts and their interconnections by exploring interactive, concept map visualizations • Enhance their own content knowledge by using the service to examine important background information on concepts, such as related prior research on student conceptions and student learning, related educational standards, and assessment strategies to check student understanding. A wide variety of end-user interfaces based on concept maps and concept browsing can be created using the strand map service. One example interface, created for D LESE, is shown in Fig. 3. This particular interface was used in the experiment described later in this article. The right side of this interface illustrates a “map view” generated by the Service; i.e., a visual display of one complete strand map. Users can explore this map view via direct manipulation; selecting strand names or grade-levels will enable the user to “drill down” and explore smaller portions of this map. Learners or teachers can also explore this map, and other strand maps, by manipulating the twist-down folders on the left side of the interface. That is, in addition to providing visualizations of maps and map components, the Service also provides visualizations that support exploring across map boundaries. Finally, the user can elect to retrieve resources from D LESE that support a particular concept by pressing one of the “view related resources” links embedded in the map nodes. This causes a list of search results

to be displayed containing brief descriptions of relevant resources. These brief descriptions include information about each resource, such as the resource’s title, URL, gradelevel, a short textual summary, and relevant educational standards.

4.2 Web service interface The programmatic web service interface enables digital library developers to easily construct concept browsing interfaces appropriate to the needs of their specific library audiences using dynamically generated visual components provided by the Service. Figure 4 illustrates the overall Service architecture and briefly describes the major components of the Service. The Service builds on recent advances in visualization components [24] and programmatic interfaces to knowledge organization systems [25]. The Service is architected to support a “spectrum of interoperability” to maximize its utility for a broad range of NSDL projects [26]. Specifically, library developers can create interfaces and services by making calls to our web service interface or by harvesting conceptual information using the Open Archives Initiative Protocol for Metadata Harvesting (OAI-PMH) server [27]. Rather than creating static presentations of strand maps, the Service middleware generates visualizations of maps and map components from information modeled in the benchmarks repository. The information modeled in this repository is drawn from the Benchmarks and Atlas publications [5, 6], as well as other AAAS materials. Library developers create concept browsing interfaces by requesting information from the Service middleware using a web service interface: the concept space interchange protocol (CSIP) [28].


241

Fig. 3 The strand map search interface implemented in DLESE. Users can choose to explore the conceptual relations depicted or to retrieve relevant resources by clicking one of the concepts

Fig. 4 Architecture of the strand map service

242

CSIP supports three types of requests: (1) service description (which returns information about the capabilities and version of this instance of the Service), (2) submit resource (which is used to submit additional benchmark information to the Service), and (3) query (which is used to request AAAS information and visual components useful for creating concept browsing interfaces). Details on the web service protocol are described elsewhere [29, 30]. When the user performs an action in a client interface, the interface makes an information request to the Strand Map Service, such as “retrieve all the concepts associated with a particular strand.” The Service returns the requested information as XML or as scalable vector graphics (SVG) [31]. SVG is a technical format for the exchange of graphical information in web-based interfaces [31]. The SVG option enables developers to easily construct concept browsing interfaces from interactive visual components that are dynamically generated by the Service. Using this option, the same information is returned to the interface as in the XML option, but it is already embedded in a visual component that can be directly displayed and interacted with. The visual component generator (VCG) takes the information extracted from the benchmarks repository and renders it into a semantic-spatial display of a strand map in SVG format [29, 32]. One design requirement that emerged from discussions with NSDL developers was that libraries want to choose from a flexible range of approaches for indicating correspondence between a concept and library resources. Therefore, a Query Registration Service is provided that allows digital libraries to specify particular retrieval methods to use when searching their collections for conceptual information.

5 The study: assessing the influence of strand maps A study was conducted in order to determine how different search interfaces may influence the cognitive search processes of digital library users when performing educationally relevant tasks. A multimedia strand maps interface was contrasted with the standard D LESE textual search interface. Because the strand maps interface provides the user with a visual organization of knowledge as well as rich semantic content about the domain, it was hypothesized that the strand maps interface would promote increased use of content-based (science-focused) processes when performing information seeking tasks in D LESE. In contrast, because textual search interfaces are directed by user input we predicted that learners with limited domain knowledge using the D LESE textual interface would fall back on surface-level search processes. These users are more likely to focus on operational strategies—such as selecting vocabulary options or typing in keywords—to successfully retrieve resources with less stringent attention to relevant science content.


5.1 Research methodology Because the strand maps are designed to promote understanding of concepts important to science education, students with limited domain knowledge were used to approximate educators with low science knowledge engaging in typical educational search tasks. Twelve undergraduate students from the University of Colorado, Boulder participated in the study; each received partial credit toward a research participation requirement for an introductory psychology class. The average age of participants was 19 (min: 18, max: 22); six participants were male and six were female. In a demographic form completed at the end of the research session, all students rated themselves as being comfortable and experienced with computer use; average response was 5.5 (min: 4, max: 7) on a scale from 0 (very low) to 7 (very high). In addition, students reported that they frequently used digital methods to search for academic information. When asked how often they used a computer to research information related to class topics, average response was 5.5 (min: 3, max: 7) on a scale from 0 (never) to 7 (2+ times a day). Students were randomly assigned to one of two experimental conditions; half the participants performed the study using the D LESE textual search interface (shown in Fig. 1) while the other half performed the study using a strand maps interface for D LESE resource retrieval (shown in Fig. 3). On average, the participants took about 30 min to complete the study. Participants first were given an informed consent form, which explained the procedures of the study, stated that the students would be audio-recorded during the tasks, and explained steps that would be taken to ensure confidentiality of the data. After consent was obtained, students began the main experimental procedure. Because the strand maps are particularly relevant for educators, a series of four tasks (see Table 1) were developed to represent typical information-seeking needs that an educator may have when teaching an unfamiliar topic in Earth science. Each task required the participant to take on the role of an educator in middle or high school with the goal of finding a specific kind of resource that could be used to teach desired concepts in class. In order to understand if the strand maps search interface influenced search processes during task completion, participants in both groups were asked to think aloud as they completed each task. The same set of tasks was used for both groups of participants (the strand maps group, and the textual search group). During the think-aloud procedure, participants in both groups were asked to talk aloud about their onscreen choices while using the system, as well as to verbalize the reasoning behind their actions. Students were asked to explain everything they were doing or trying to do, what they were thinking, and whatever questions, ideas or thoughts they had along the way. The experimenter emphasized that there were no right and wrong answers during verbalization and that the more the student talked, the more it would help the researchers understand how people


243

Table 1 Task descriptions representing information needs that may be faced by educators Task 1 Carl is an 8th grade science teacher. He’s new at this. He used to be a substitute teacher and has just obtained this full time position. He’s teaching science for the very first time in his life. In his school, teachers work as a team during the summer months to develop modules and lesson plans for the upcoming school year. Currently, his team is working on modules related to rocks and sediments. The team is going to meet this afternoon. Carl wants to find some information related to sedimentary rock, specifically—some interactive media or images on the topic of sedimentary rock to use for teaching his class. He’s heard from his colleagues that D LESE is a good source for such materials. Task 2 Jack is a 10th grade science teacher who has volunteered to fill in for Jan, a 7th grade science teacher, while she’s out sick. Jan was supposed to teach her class about changes in the Earth’s surface this week. She suggests Jack come up with a classroom activity based on changes in the Earth’s surface. One of the topics that Jack teaches in his 10th grade class relates to earthquakes. He wants to teach the 7th graders something related to this topic. Jack often uses D LESE in order to find activities and detailed text on material he teaches in his 10th grade class. He decides to check out what D LESE has to offer. He wants to find out which concepts he needs to teach the 7th graders, in addition to a classroom activity that support these concepts. Task 3 Sally has just graduated from CU, where she was an education major, and has found a job as a fifth grade teacher. One of her teaching responsibilities is science. She is preparing material for the science class. One of the topics in the syllabus is composition of rocks. She wants to teach some age-appropriate concepts in class. She knows that last year the students learned that rocks come in different sizes and shapes. She wants to find teaching materials to help her teach the students how to classify rocks according to their characteristics. Task 4 Josie teaches 11th grade Math, but she’s substituting for her colleague, Alicia, this week, who teaches 11th grade science, who’s on leave. Before she left, Alicia was teaching her class about the age of rocks or fossils. She has left a note for Josie, suggesting that Josie give the class a lab activity on the topic. She also suggests D LESE as a possible source for lab activities on science topics.

work with the system. A normal think-aloud protocol is nondirective [33]; the only probe used by the examiner after the initial instructions is when participants stop verbalizing for some time, at which point they are reminded that they need to think aloud. However, because we were interested in the cognitive search processes in which students would engage during task completion, the think-aloud protocol was modified slightly to elicit cognitively oriented process information from students. The experimenter prompted the student after vague or incomplete comments with questions such as “Why would you say that?” or “How did you arrive at that conclusion?” These prompts were necessary in order to capture the underlying processes in which users were engaging during task completion. All verbal protocols were audio recorded and transcribed. 5.2 Verbal protocol analysis Verbal data collected during the think-aloud protocol were analyzed for differences in the cognitive processes represented by the verbalizations. After verbal protocols were transcribed, each protocol was separated into a series of complex propositions for coding. A complex proposition is approximately equal to an idea unit that represents one thought or action for analysis [34, 35]. Two raters coded each proposition according to the type of process it represented. Protocols were scored twice; after initial scoring, raters discussed category criteria and reviewed all protocols for scoring accuracy. Nine major categories were coded; see Table 2 for an explanation and example of each category. Seven of these categories are organized under three major types of

comprehension processes relevant to digital library search: planning, performance, and evaluation. The other two categories were needed to score propositions that lacked content (Filler/Miscellaneous) or were otherwise uncodable (Search–Other). It is important to note that planning, performance, and evaluation processes are not linear in nature but tend to be used iteratively during an overall search task. As learners begin to plan their digital library searches, they attempt to coordinate their understanding of the task, the system, and their needs. Thus, statements related to these types of planning-relevant processes were separated into two categories: Monitoring and Task propositions. After assessing the task and their understanding, learners begin to search for relevant material in the library. During search performance, learners tend to talk about both what they are seeing on the screen and their search strategies. These general categories are referred to as “Interface” and “Strategy” statements, but each category can occur at a surface level (where statements do not reflect science content) or at a content level (where statements directly refer to relevant domain science). Thus, four Search–Performance categories were scored: Interface–Surface, Interface–Content, Strategy–Surface, and Strategy–Content. Finally, once learners retrieved a resource, they needed to make a decision about its usefulness during an evaluation phase. The Evaluation category refers to the learner’s overall decision to use or not to use the resource. Within the evaluation category, reasoning subcategories reflected the type of rationale that students gave for their evaluations; these subcategories included reasoning related to the surface-level characteristics of the resource (without mention of science content), content-level

244


Table 2 Categories and subcategories for propositional analysis Category

Description

Example

Relevant stage

Filler or Miscellaneous

Statements with no analyzable content.

“Um” or “Let’s see.”

N/A

Monitoring

Statements that assess one’s understanding

“I think that . . .”

Search Planning

Task

Statements that relate to information or goals given in the experimental task descriptions.

“He wants to learn about the rock cycle. . .” or “It says they are seventh graders. . .”

Search Planning

Interface–Surface

Statements about on-screen information that did not include science content.

“I see a lot of pictures on this page. . .”

Search Performance

Interface–Content

Statements about on-screen information that include reference(s) to science content or knowledge.

“It says that dissolved minerals form solid rock again.”

Search Performance

Search–Surface

Statements about search strategies or behaviors reflecting surface-level information and no science content.

“I need to find pictures so I’m going to. . .”

Search Performance

Search–Content

Statements about search strategies or behaviors that include science content or knowledge.

“I need to find information about this plate tectonics so I’m going to. . .”

Search Performance

Search–Other

Statements about search strategies that were otherwise uncodable.

“I just thought I’d look back. . .”

N/A

Evaluation

Statements that reflected decisions about information relevance or usefulness.

“I think this is too general” or “This one seems like a good choice.”

Search Evaluation

Evaluation rationale (subcategories of evaluation)

Reasoning–Surface: Rationale for accepting or rejecting a resource based on superficial characteristics of the resource.

“This has a lab activity.”

Search Evaluation

Reasoning–Content: Rationale for accepting or rejecting a resource based on science content or knowledge

“This talks about the causes for earthquakes in depth.”

Search Evaluation

Reasoning–Task: Rationale for accepting or rejecting a resource based on the demands of a given task

“This activity would work better for fifth graders.”

Search Evaluation

Reasoning–Other: Rationale for accepting or rejecting a resource based on criteria other than those listed above.

“This is more like what I did as a kid.”

Search Evaluation

reasoning about the resource (related to science domain information), task-level reasoning (assessing appropriateness based on task description), and other reasons that were varied and lacked theoretical relevance (e.g., just “liking” a resource better). The Filler/Miscellaneous and the Search–Other categories were not included in final analyses due to the unfocused nature of statements they contained. In addition, the Reasoning-Other subcategory was rarely used and therefore was not included in final analyses. Thus, seven major categories and three subcategories were analyzed; interrater correlations averaged 0.74 for major categories (min: .63 for Task, max: 0.83 for Evaluation) and 0.69 for subcategories (min: 0.65 for Reasoning–Content, max: 0.75 for Reasoning–Task). Table 3 shows a selection of complex propositions from participant protocols and the coded categories assigned to each statement.

5.3 Results 5.3.1 Quantitative results In order to compare the two groups (strand maps interface vs. textual search), the number of complex propositions of each type were averaged across raters and tasks. Based on our experimental questions, a repeated measures analysis of variance (ANOVA) was used to test for group differences in the frequency with which students engaged in contentbased and surface-based processes (in the interface and strategy categories). Results showed a significant two-way interaction (F(1,10) =9.2, p