Instr Sci DOI 10.1007/s11251-013-9263-8
Task-oriented reading of multiple documents: online comprehension processes and offline products Øistein Anmarkrud • Matthew T. McCrudden • Ivar Bra˚ten Helge I. Strømsø
•
Received: 27 May 2012 / Accepted: 2 January 2013 Ó Springer Science+Business Media Dordrecht 2013
Abstract We explored readers’ judgments of text relevance and strategy use while they read about a controversial scientific issue in multiple conflicting documents using a thinkaloud methodology and had them write a short essay after reading. Participants were university-level students. There were three main findings. First, readers discriminated between more- and less-relevant information while they read. Second, the frequency with which they used strategies differed while they read more- and less-relevant information. Specifically, while they read more-relevant information, students were more likely to build connections between that information and information in other texts. Third, their judgments of more-relevant segments as relevant and their evaluation of less-relevant information while they read were related to the quality of students’ essays after they read. We discuss how the findings may contribute to the literature on task-oriented reading of multiple documents. Keywords Multiple-documents literacy Task-oriented reading Strategic processing Text relevance
Introduction In task-oriented reading, readers use one or more texts to complete an assigned task or a self-selected reading goal (Bra˚ten and Strømsø 2010; Gil et al. 2010; Vidal-Abarca et al. 2010; Wineburg 1991). Consider the following example. Suppose a teacher gives her Ø. Anmarkrud H. I. Strømsø University of Oslo, Oslo, Norway M. T. McCrudden Victoria University of Wellington, Wellington, New Zealand e-mail:
[email protected] I. Bra˚ten (&) Department of Educational Research, University of Oslo, Blindern, P.O. Box 1092, 0317 Oslo, Norway e-mail:
[email protected]
123
Ø. Anmarkrud et al.
students multiple documents from different sources about whether cell phone use poses a health risk. She then gives them the following task instruction: ‘‘Imagine a close friend has told you that she experiences discomfort when using her cell phone. She has asked you for advice. You are now going to study these texts in order to provide your friend with wellgrounded advice.’’ This task can be demanding because readers need to be able to identify, evaluate, and integrate task-relevant information within and across various documents that contain overlapping, unique, and conflicting messages (Britt et al. 1999; Cerda´n and VidalAbarca 2008; Rouet 2006; Rouet and Britt 2011; Wiley et al. 2009). In such instances of task-oriented reading, it is critical that readers judge information as more- and less-relevant and process it in ways that enable them to meet their reading goals. Text relevance refers to the perceived instrumental value of text information in relation to an individual’s goal or purpose for reading (Lehman and Schraw 2002; McCrudden et al. 2011; McCrudden and Schraw 2007). Text relevance differs from text importance, which generally refers to text elements that are essential for understanding a text’s main ideas (Cirilo and Foss 1980; Schraw et al. 1993). Text information that is perceived as more instrumental to a goal is judged to be more relevant, whereas information that is perceived as less instrumental is judged to be less relevant (McCrudden et al. 2011). Thus, information that is important for understanding a text may not necessarily be relevant to a particular reading task. Online data, in which the term online refers to readers’ moment-by-moment processing while they read text (e.g., printed page, screen-based), and offline data, in which the term offline refers to the products of a post-reading activity, have shown that readers allocate more attention to more-relevant information than to less-relevant information, and recall more of this information (Rapp and Mensink 2011). For instance, reading time data have shown that readers spend more time reading more-relevant information than less-relevant information, and often recall more of this information (McCrudden and Schraw 2007). Similarly, eye-tracking data have shown that readers allocate more visual attention towards more-relevant information than towards less-relevant information (e.g., Burton and Daneman 2007; Kaakinen and Hyo¨na¨ 2011; Lewis and Mensink 2012), and often recall more of this information. However, while reading time and eye-tracking methodologies provide information about how readers’ allocate their attention, they only provide indirect evidence about readers’ comprehension processes while they read. There is limited research using think-alouds to measure readers’ online comprehension processes while they read moreand less-relevant information (Kaakinen and Hyo¨na¨ 2005), particularly in the context of reading multiple documents (Bra˚ten and Strømsø 2003; Wiley et al. 2009). Thus, more research is needed to understand what readers are doing when they spend more time on more- and less-relevant information and how these activities relate to the offline products of reading. Therefore, the purpose of this study was to explore readers’ online comprehension processes and offline products in task-oriented reading. More specifically, we investigated readers’ judgments of text relevance and their strategy use while they read multiple documents, and their relations to post-reading essay performance (i.e., written argumentation). Participants were university-level students. We used Rouet and Britt’s (2011) MD-TRACE model (for Multiple Document Task-based Relevance Assessment and Content Extraction), as a frame for investigating this topic. Task-oriented reading of multiple documents Task-oriented reading is a goal-directed activity such that an individual reads to meet a goal (McCrudden and Schraw 2007). When task-oriented reading is externally-induced, readers use one or more texts to complete an assigned task or goal. The MD-TRACE model
123
Task oriented reading
describes the online processing of multiple documents in task-oriented reading as multifaceted (see Rouet and Britt 2011 for a detailed description of the model). According to the MD-TRACE model, a reader interprets task instructions, then develops a reading goal and a task model (i.e., a mental representation of the expected outcome of reading), both of which can be modified during the course of reading. The task model then drives a reader’s plans for pursuing goals through subsequent processing activities because it enables the reader to assess the relevance of information in relation to an assigned task. The assessment of relevance consists of three distinct steps, although the steps may take place in various order and may occur in parallel (Rouet and Britt 2011). In the first step, readers judge the task-relevance of the available information. In the second step, readers process the text contents (or potentially skim or ignore it). Their perceptions of the information, such as its relevance or the difficulty associated with understanding the information, influence whether and how they will process that information. In the third step, readers create/update a documents model, which consists of readers’ understanding of the documents’ contents, their evaluation of the sources, and judgments about relations between/among documents (e.g., complementary, contradictory). Readers may use a variety of cognitive processes during the second and third steps (Bra˚ten and Strømsø 2011; Strømsø et al. 2003; Wineburg 1991; Wolfe and Goldman 2005), as they attempt to strategically construct meaning while they read (Afflerbach et al. 2008). To illustrate relevance assessment, suppose a student is given three documents and is asked to determine whether cell phone use poses a health risk and to provide advice for a friend. Thus, her goal is to provide sound advice for a friend and she believes that reading will enable her to meet this goal. Further, she will need to have criteria in mind for judging the relevance of text information. In the first document, she reads about the wave frequencies that various cell phones produce. In the second document, she reads that cell phone companies charge different amounts for similar service plans. Finally, in the third document, she reads that wave frequencies above a certain level are considered harmful. For the assigned task, the information about the wave frequencies that various cell phones produce in the first document is relevant; however, it is incomplete for determining whether the frequencies pose a health risk until she relates them to information in the third document about the range of frequencies that are deemed harmful. However, the student must be aware that the initial information about cell phone wave frequencies was incomplete and then link it to the information about dangerous wave frequencies, and be able to judge that the information about service plans in the second document is less relevant to the task. Thus, it is important for students to be able to judge the relevance of information and to use strategies that enable them to meet their goals. For instance, the student may judge the information about the cost of service plans to be irrelevant and choose not to think about it further in an effortful way. Conversely, the student may judge the information about cell phone wave frequencies to be relevant and may seek to link this information to other information. When a student is given an assortment of documents, they can reasonably assume that the information in them is generally relevant to the task. However, when students must identify the documents themselves, the identification of relevant information is even more difficult, and underscores the need to have clear criteria for judging the relevance of information. Reading multiple documents highlights the goal-directed nature of successful taskoriented reading because of the additional time and energy needed to identify and link relevant information between and among documents (Bra˚ten et al. 2011; Goldman et al. 2010). For instance, the need for deliberate processes, such as linking, monitoring, and evaluating may increase during multiple documents comprehension as readers attempt to
123
Ø. Anmarkrud et al.
actively construct meaning from overlapping, unique, and sometimes conflicting pieces of information from different sources (Bra˚ten et al. in press; Bra˚ten and Strømsø 2011). In the present study, we used Rouet and Britt’s (2011) MD-TRACE model as a frame to investigate readers’ online processing of multiple documents during task-oriented reading. In particular, we were interested in their judgments of text relevance and their use of the following three strategies while they read more- and less-relevant segments: (1) linking (e.g., building connections between text segments or between text segments and prior knowledge), (2) monitoring (e.g., noting comprehension problems or discrepant information between sources), and (3) evaluation (e.g., using information about source of text to evaluate and interpret a text segment within that text). We used think-aloud protocols to measure these online processes. Think-aloud protocols can provide information about reading processes that are consciously available to readers and codeable in language (Ericsson and Simon 1993; Magliano et al. 1999; Trabasso and Magliano 1996), which provides insights into readers’ moment-bymoment cognitive processes. While many cognitive processes that occur during reading are automatic or are not codable in language, proficient readers typically have an awareness of their deliberate processes (Graesser et al. 1994) and tend to report them when thinking aloud (Magliano et al. 1999; Magliano and Millis 2003; Trabasso and Magliano 1996). Although thinking-aloud while reading may alter readers’ spontaneous processing of the text (Fletcher 1986), especially among younger readers (Cote et al. 1998), thinking-aloud has received extensive validation as a tool to reveal comprehension processes in reading (Afflerbach 2002; Cote´ and Goldman 1999; Magliano and Graesser 1991; Magliano and Millis 2003; Magliano et al. 1999; Zwaan and Brown 1996). How readers process multiple documents in task-oriented reading has implications for the offline products of reading (e.g., Bra˚ten et al. in press; Bra˚ten and Strømsø 2011; Cerda´n and Vidal-Abarca 2008; Cerda´n et al. 2009; Rouet et al. 2001; Wiley et al. 2009; Wolfe and Goldman 2005). In the present study, the offline product was a brief essay in which the participants judged the health risk of cell phone use. Previous research that has investigated the link between online processes and offline products has generally measured offline products using measures of free or cued recall. In this study, students developed a written argument about whether cell phone use poses a health risk. We chose this task because it prompted students to form an opinion on a controversial topic, which is arguably one of the main reasons that people read multiple documents. Constructing a good argument on a controversial issue when sources provide unique, overlapping, and conflicting information can be challenging for many readers (Kuhn 2009; Larson et al. 2009; Wolfe et al. 2009). Thus, it is important to understand the links between readers’ online comprehension processes and offline products in task-oriented reading from multiple documents about a controversial topic. The present study We asked university students to read multiple documents that contained unique, overlapping, and conflicting messages about whether cell phone use poses a health risk. Their task was to be able to provide advice to a close friend who had experienced discomfort when using her cell phone. Students thought-aloud while they read multiple documents. After they read, they wrote an essay about whether they thought using a cell phone posed a health risk. We analyzed the think-aloud data to investigate readers’ online processes while they read more- and less-relevant text segments. According to the MD-TRACE model (Rouet and Britt 2011), readers assess the relevance of text information, process more- and less-relevant information differently, and
123
Task oriented reading
update their ongoing understanding of the texts. We developed three research questions to evaluate these predictions. First, do readers discriminate between more- and less-relevant information while they read? If readers assess the relevance of text information, then readers should judge the relevance of information differently while they read. That is, readers should make more positive judgements while they read more-relevant segments (i.e., judge the information to be more-relevant); similarly, readers should make more negative judgments while they read less-relevant segments (i.e., judge the information to be less-relevant). We predicted that readers would discriminate between more- and lessrelevant segments while they read because task instructions help readers develop standards of relevance, or criteria for identifying information’s value for accomplishing a reading goal (Kintsch 1998; Lorch et al. 1993; McCrudden et al. 2010; van den Broek et al. 2001; van den Broek et al. 1995). Second, does strategy use differ while individuals read more- and less-relevant information? According to the MD-TRACE model, readers’ perceptions of text information, such as its relevance, influence whether and how they will process it. We assessed readers’ use of three particular strategies: linking, monitoring, and evaluating. If readers’ judgments of relevance influence whether and how they process information, then their strategy use should differ while they read more- and less-relevant information. For example, once readers judge information to be more-relevant, they should engage in more effortful processing, such as linking, to integrate that information with their developing mental representation of the text. Conversely, once readers judge information to be less-relevant, they should engage in less effortful processing of that information. That is, readers’ strategy use should differ while they read more-relevant sentences than while they read less-relevant sentences because they should be willing to engage in more effortful processing while they read information that is more-relevant to the task (van den Broek et al. 1999). We predicted that readers would use strategies differently while they read more- and less-relevant information because readers tend to allocate more attention and processing effort towards more-relevant information (Kaakinen and Hyo¨na¨ 2005; Wiley et al. 2009). Third, are readers’ relevance judgments and strategy use while reading related to postreading essay quality? We investigated the correlations among relevance judgments, strategy use, and essay quality. Thus, we wanted to know whether discriminating between and strategically processing more- and less-relevant information were related to essay quality. Readers remember more-relevant information better than less-relevant information (e.g., Kaakinen and Hyo¨na¨ 2011; McCrudden and Schraw 2007). However, constructing a written argument involves skills beyond memory for information (Britt and Larson 2003; Rouet 2006). Constructing arguments on the basis of multiple information sources is regarded as an essential academic skill and can be viewed as an essential skill for participation in democratic discourse. Investigating the relation between online processes and written argumentation can enable researchers to better understand how readers engage with multiple documents and form arguments, which can inform future research. Thus, we explored whether relevance judgments and strategy use were related to essay quality.
Method Participants and setting Participants were students (n = 51) at a large state university in southeast Norway, who were enrolled in an introductory course in educational science. The course was a
123
Ø. Anmarkrud et al.
mandatory foundation course for students enrolled in bachelor programs in education and special education. The sample included 43 females and 8 males, with an overall mean age of 22.1 (SD = 2.6). The majority (84.3 %) were white, native-born Norwegians whose first language was Norwegian and who had completed their secondary education in a Norwegian school. Most of the remaining participants had another Scandinavian language as their first language. The sample was relatively homogeneous in regard to socioeconomic status (i.e., middle class). All participants had at least 12 years of schooling before beginning university studies. In addition, 31 participants were in their first year of university studies, nine were in their second year, and 11 had completed two or more years. We measured participants’ background knowledge about cell phones and potentially related illnesses. Their knowledge about this issue was so low that their prior knowledge scores were not included in subsequent data analyses. Participants were recruited from a lecture for students (n = 175) enrolled in the educational science course. Those who volunteered (n = 51) provided their contact information and were later contacted by research assistants for individual appointments. The participants completed the reading task individually in a university lab, directed by a trained research assistant. The participants did not receive any course creditis, and participation was not a part of any class assignment. Each of participant was paid $50, equivalent to 2 h work as a research assistant. Materials Documents Participants read six documents that presented different perspectives on cell phones and potential health risks. These documents were authentic in the sense that they were obtained via the World Wide Web and their contents were not altered. All of the documents were in Norwegian. At the beginning of each document, source information was presented in the form of author’s name and credentials, publisher, publication, document type, and date of publication. The first document was a 554-word excerpt from a textbook in science for upper secondary education. This document explained the functioning of cell phones and electromagnetic radiation in relatively neutral, academic terms, and included a paragraph on how researchers can investigate potential health risks related to cell phone use. The second document was a 444-word public information text from the National Radiation Protection Agency (NRPA) which stated that researchers have not substantiated a link between cell phone use and serious illness (e.g., cancer), but that some precautions, such as use of hands free and SMS and brief conversations, should still be taken given a lack of evidence (e.g., lack of longitudinal studies). The third document was a 473-word article from a popular science magazine called Illustrated Science, in which a research reporter described a recent, yet unpublished review article by an Australian professor and brain surgeon, Dr. Khurana, who argued that there is solid scientific evidence for the link between cell phone use and certain brain tumors. This document also included a paragraph which referred to a British researcher expressing concerns about the potential health risks related to radiation from wireless computer networks, regarded as similar to cell phone radiation. The fourth document was a 483-word debate article in a health care magazine called Today’s Medicine written by a chief engineer representing the Cell Phone Operators’ Association, who took issue with Dr. Khurana’s message and argued that, given the current evidence, the cancer risk associated with cell phone use is exaggerated, also citing the World Health Organization and other independent research reviews in support of his
123
Task oriented reading
view. The fifth document was a 464-word text written by a reporter in a U.S. power critical magazine called the Idaho Observer, which generally emphasizes advocacy journalism and populist political issues. The article stated that cell phone use doubtlessly causes cancer, citing unspecified scientific journals and researchers in support of this view, and also claimed that the telecom industry, in alliance with dishonest politicians, conceals this proven fact from the public. Finally, the sixth document was a 395-word newspaper article written by a journalist in a Norwegian liberal daily newspaper that presented an interview with a musician diagnosed with brain tumor who suspected that his heavy cell phone use might have been the cause of his disease. Table 1 provides an overview of the documents. Apart from the science textbook excerpt (Document 1), the other documents contained partly conflicting information. Document 2 critiqued the existing research base and argued that there is no evidence that cell phone use poses a health risk, but still recommended certain precautions. Documents 3 and 5 argued that cell phone use increases the risk of brain cancer. Document 4 directly opposed this view and argued that cell phone use does not pose a health risk. Document 6 presented the life story of a cell phone user who attributed his brain cancer to cell phone use. As an indication of text difficulty, we computed readability scores for each of the six texts, using the formula proposed by Bjo¨rnsson (1968), which is based on sentence length and word length. Vinje (1982) reported that public information texts from the Norwegian government had a readability score of 45, while texts in the Norwegian code of laws had readability scores ranging from 47 to 63. As can be seen in Table 1, the readability scores of the documents that we used in this study ranged from 39 to 57. The mean readability score for the six documents was 50.3 (SD = 6.3), suggesting that the reading material was somewhat more difficult than that encountered in public information texts. Essay task Participants were asked to write a brief essay on the issue discussed in the documents. Specifically, they were given the following written instruction: ‘‘You are now going to write a short essay where you judge the health risk of cell phone use. Base your response on the texts that you just read and try to express yourself clearly and elaborate the information—preferably in your own words. Justify your conclusions by referring to the sources you have been working with.’’ Procedure Participants completed each task individually, directed by a trained research assistant. First, they completed a demographic questionnaire. Second, they practiced thinking aloud for 15 min while they read two expository documents on a different topic on a computer screen. Prior to reading the practice documents, students were instructed to think aloud and report on whatever they thought and did while they read. They were also informed that prompts would be given if they remained silent for more than 30 s. The following instruction was given to each participant: ‘‘I want you to say everything you are thinking and doing while you read. You can choose whether you want to read silently, aloud, or combine these two ways of reading, but I want you to continuously say everything that you are thinking and doing aloud while you work with the documents. If you do not say anything for a while, I will remind you to speak. Do you understand what you are to do?’’ Third, participants entered a browser showing an offline search results page in Googlelayout listing six search results corresponding to the six documents described in the
123
123
Type of document
Textbook in nature studies for upper secondary education
Public information text
Popular science article
Debate article
Polemic
Newspaper article
No.
1.
2.
3.
4.
5.
6.
Norwegian liberal daily
Idaho Observer
Today’s Medicine
Illustrated Science
Journalist
Journalist
Chief engineer representing cell phone industry
Research reporter
Interviews person with brain tumor who suspects that this was caused by heavy cell phone use
395
39
50
464
Cites scientific journals and researchers to support the view that cell phones undoubtedly cause cancer and claims a conspiracy involving the industry and politicians to conceal this fact
55
483
Takes issues with the message of document 3 and argues that the cancer risk related to cell phone use is exaggerated
49
473
57
52
Readability score
Cites researchers arguing that radiation from cell phones and wireless networks poses serious health risks
444
States that it is not documented that cell phone use causes cancer but recommends some precautions
National Radiation Protection Agency
National Radiation Protection Agency
554
Explains the functioning of cell phones and electromagnetic radiation in relatively neutral, academic terms
Science teachers
Number of words
Publishing house
Content
Author
Publisher
Table 1 Overview of the six documents
Ø. Anmarkrud et al.
Task oriented reading
Fig. 1 Screenshot of the search result page
‘‘Materials’’ section (see Fig. 1). The search terms shown on the page were cell phone ? radiation, and each search result displayed the title of the document, the name of the publication, and its uniform resource locator (URL). The six search results appeared on the page in the same order as the documents were described above. At the top of the search results page was a link labeled ‘‘Task’’, and participants were orally instructed to click on this link and read the following task instruction before accessing and reading the documents: ‘‘Imagine that a close friend has told you that she experiences discomfort when using her mobile phone. She has asked you for advice and you have searched the Internet for information about the topic. The search resulted in six results that you would like to take a closer look at. You are now going to study these texts in order to provide your friend with well-grounded advice.’’ The task instruction also informed participants that they could access, read, and re-read the six documents in any order they preferred, and that they could return to the search results page whenever they wanted by using the arrows at the top of the browser. They were also told that they could re-access and re-read the task instruction at any time, and that they should keep on saying aloud what they were thinking and doing while they worked with the documents. Verbalizations as well as reading times and movements from page to page were exactly recorded by the MoraeÒ usability software. After participants had accessed and read all six documents, the browser was closed and they were given a folder containing the essay task. Fourth, for the essay task, participants were given two lined sheets that were stapled together, with the instruction for the essay task printed at the top of the first sheet. Participants thus wrote the essays by hand. They were asked to study the instruction carefully before starting on their essay and also told that
123
Ø. Anmarkrud et al.
they had to finish it within 20 min. The six documents were not available during the essay task. Coding schemes and data scoring Coding scheme and scoring of relevance judgments We defined judgments of text relevance as a comment made while reading a text segment (i.e., a sentence or a group of sentences) that indicated the instrumental value of the segment in relation to the task instruction (i.e., study to provide your friend some wellgrounded advice; McCrudden et al. 2011). There were two types of relevance judgments: positive and negative. Positive relevance judgments were comments in which the participant identified the segment as more-relevant to the task (e.g., ‘‘It seems that I’m coming to something about the heating of body tissue. Then I think that this is more relevant for that task’’). Negative relevance judgments were comments in which the participant identified the segment as less-relevant to the task (e.g., ‘‘I believe that it is totally irrelevant to compare this [health risk of cell phone use] to the number of people who dies of tobacco, the point is if it’s dangerous or not’’). We identified 211 judgments of text relevance (M = 4.14, SD = 3.36, max = 15, min = 0). The vast majority of the judgments were one-sentence comments (e.g., ‘‘This information isn’t relevant to tell this friend of mine’’), but in some instances, the comments that constituted a judgment were longer (e.g., ‘‘I think about my friend when I read this. Cell phones send out more signals when you talk, so she would get more radiation when she talks, compared to when she listens. This is something I would have to tell her’’). Next, we distinguished between positive and negative judgments of text relevance. We identified 138 positive judgments of text relevance (M = 2.71, SD = 2.54, max = 11, min = 0) and 73 negative judgments of text relevance (M = 1.43, SD = 1.37, max = 5, min = 0). The first author coded all think-aloud protocols for positive and negative relevance judgments. Then, the fourth author independently coded a random subset (37 %) of the 211 episodes of relevance judgments using the same scoring rubric. The agreement on the coding of the utterances into positive and negative relevance judgments was 100 %. Coding scheme and scoring of strategies We defined ‘‘strategy’’ as a comment made while the individual read a text segment (i.e., a sentence or a group of sentences) for the purpose of controlling or modifying document comprehension (cf., Afflerbach et al. 2008; Cote et al. 1998; Strømsø et al. 2003). We developed a coding scheme based on codes used in previous research (see Table 2 for definitions and exemplar quotes; Afflerbach and Cho 2009). The coding scheme included seven subcategories, which were organized into three main categories (i.e., linking, monitoring, and evaluation). We identified 326 instances of strategy use (M = 6.37, SD = 6.22, max = 26, min = 0). The vast majority of these instances were one utterance long, but in some instances, more than one utterance constituted an instance. The first author scored all of the think-aloud protocols using the coding scheme. Then, the third author used the same coding scheme to independently score a random subset (20 %) of the 326 instances. Agreement on the main categories (i.e., linking, monitoring, and evaluation) was 93.9 %. Regarding the scoring of the utterances into the 7 subcategories, the overall agreement was 89.4 %. Specifically, agreement on the subcategories of linking, monitoring, and evaluating, respectively, was
123
Making a connection between a text segment and background knowledge
Text-to-background knowledge
Source
Using information about source of text to evaluate and interpret text segment within that text
Perceiving diverse and complementary views on one topic across different texts
Across-text
Evaluation
Awareness of comprehension problem within a particular text
Within-text
Monitoring
Making a connection between text segments from different texts
Definition
Text-to-text
Linking
Cognitive process
6
0
7
17
5
0
4
6
7
Frequency: lessrelevant segments (n = 26)
43
Frequency: morerelevant segments (n = 84)
Table 2 Coding scheme for think-aloud comments, frequencies, and exemplar quotes
‘‘Initially, I see that this is written by the National Radiation Protection Agency, and then I’m thinking that these people know what they are writing about.’’
‘‘In some articles, cell phones are really dangerous, whereas in others they’re not bad at all.’’
‘‘Do they exaggerate the danger? Yes. Who says this?’’
‘‘And it seems that he is concerned that radiation from cell phones harms the person using it, whereas use of computers can harm people in the surroundings. Just like passive smoking; that’s something I think about right now.’’
‘‘Then I think about the first texts I read, particularly the one from the National Radiation Protection Agency. There I was told that it is not actually proven that cell phones are harmful.’’
Exemplar quotes
Task oriented reading
123
123
Using information about source cited within a text to evaluate and interpret a particular text segment
Comparing the reliability and validity of sources and information across different texts
Holistic source
Definition
Embedded source
Cognitive process
Table 2 continued Frequency: lessrelevant segments (n = 26) 5
3
Frequency: morerelevant segments (n = 84) 6
1
‘‘I believe that many people will be frightened by Rolf Teigen’s story and become afraid that something like that may happen to them. But this is only a single case, and then it becomes difficult to conclude anything from it compared to the other texts.’’
‘‘They refer to the WHO [World Health Organization]; then I suddenly feel that I trust them more than a random guy from Australia.’’
Exemplar quotes
Ø. Anmarkrud et al.
Task oriented reading
89.1, 94.4, and 87.5 %. All disagreements in the coding were settled through discussion between the two raters. Scoring of essays We scored the essays according to Reznitskaya et al.’s (2009) holistic scoring rubric, which focuses on several macro-level features that indicate the overall schematic structure of written arguments. This rubric is used to rate the overall quality of written argumentation on a 7-point scale and is consistent with theoretical assumptions regarding argumentation (e.g., Britt and Larson 2003; Reznitskaya and Anderson 2002; Toulmin 1958). The holistic scoring criteria that we used are displayed in Table 3. Our scoring criteria differed from Reznitskaya et al.’s in two main ways. One, we expanded ‘‘discussing opposing perspective’’ to ‘‘discussing opposing perspective(s) and the unsettled nature of the issue’’. Two, we made a distinction between mentioning opposing perspectives and discussing opposing perspectives. These adaptions were made to give credit to participants that were able to reflect upon the different perspectives and the lack of clear cut answers regarding the issue. Scoring the argumentative essays included five main steps. In the first step, we identified the participant’s position on the issue. Essays that did not state a position were given a score of 1. In the second step, we identified and counted the number of reasons that the participant used to support his/her position on the issue. In the third step, we identified whether the participant had included a counterargument to his/her position and whether the participant noted the controversial nature of the issue. When such information was identified, we Table 3 Rubric for scoring the essays for argumentative reasoning Score
Description
7
The essay contains five argument components: positions, supporting reasons, opposing reasons, elaborations, and rebuttals. There is a consistent discussion of opposing perspective(s) and the unsettled nature of the issue. The essay is well-structured and focused. No irrelevant information is included, repetition is low
6
The essay states a clear position on the issue supported by elaborated reasons. There is a consistent discussion of opposing perspective(s) and the unsettled nature of the issue. The essay is well focused
5
The essay states a clear position on the issue supported by elaborated reasons. There is some consideration of alternatives of chosen position and the unsettled nature of the issue, but it is not well-developed. There is little or no attempt at reconciling the alternative positions in own argumentation. The essay may contain irrelevant or repetitive information
4
The essay contains a position on the issue supported by 4 or more distinct or elaborated reasons. Alternative perspective(s) and the unsettled nature of the issue may be mentioned but are not discussed;
3
The essay contains a position on the issue supported by 4 or more distinct or elaborated reasons. Alternative perspective(s) and the unsettled nature of the issue are not mentioned or discussed. There is a lot of irrelevant and/or repetitive and/or inconsistent information
2
The essay contains a position on the issue supported by fewer than 4 reasons. The reasons are not elaborated. Alternative perspective(s) and the unsettled nature of the issue are not mentioned or discussed
1
The essay is underdeveloped and it is not possible to identify a position on the issue. The essay may contain irrelevant information. Alternative perspective(s) and the unsettled nature of the issue are not mentioned or discussed
Note The rubric is adapted from Reznitskaya et al. (2009)
123
Ø. Anmarkrud et al.
considered whether it was just mentioned or discussed and elaborated. In the fourth step, we examined the structure and the amount of repetition in the essays. In the last step, we searched for the five components of a strong argument (position, supporting reasons, opposing position and reasons, elaborations, and rebuttals). The first author scored all essays using the scoring rubric displayed in Table 3. Then, a random subset (25 %) of the essays was independently scored by the fourth author using the same rubric, with the two raters agreeing on the scores of 77 % of those essays. The difference between the raters on an essay was never more than 1 point, resulting in an interrater reliability coefficient (Pearson’s r) of .88. All disagreements were resolved through discussions between the two raters. Classifying the more- and less-relevant text segments Classifying information within the documents as more- and less-relevant involved three main steps. First, the first and the fourth author independently parsed the six texts into text segments, reaching an agreement of 98 % on the segmentation. The few disagreements were resolved through discussion between the two raters. In most instances, the text segments consisted of one or two sentences. Combined, the six documents consisted of 100 text segments. A text segment was defined as a sentence or group of sentences constituting an idea unit. In situations where the text segments comprised of more than one sentence, we validated the segmentation by examining the participant’s think-alouds to see if verbalizations were directed at the segment as one semantic unit, or toward individual sentences within that segment. An example of a two sentence text segment is: ‘‘So far, however, the research basis is too unsubstantiated to be able to claim a link between cell phones and cancer. And several recent studies have in fact rejected that such links exists’’. The idea unit captured from this text segment was there is a lack of research evidence linking cell phone use to cancer. Second, we obtained expert ratings from seven active researchers within the field of multiple documents reading and/or text relevance. The experts received the segmented texts and were asked to judge the relevance of each text segment in relation to the reading task using a 4-point scale (1 = Irrelevant, 2 = Somewhat irrelevant, 3 = Somewhat relevant, 4 = Relevant). Third, based on the mean score of the expert ratings of text relevance for each of the text segments, we identified the 25 % most relevant text segments and the 25 % least relevant text segments in the document set. The non-parametric omnibus Friedman test indicated statistically significant differences in the experts’ mean ranks of the most- and least-relevant text segments, v2(1, n = 7) = 7.000, p \ .008. The number of more- and less relevant text segments identified by the experts in the six documents are displayed in Table 4.
Results Our first research question was: Do readers discriminate between more- and less-relevant information while they read? To address this question, we conducted a 2 (judgment type: positive or negative) 9 2 (segment type: more-relevant or less-relevant) within-subjects ANOVA for relevance judgments. Means and standard errors are in Table 5. The main effect of judgment type was significant, F(1, 50) = 19.02, MSE = 1.12, p \ .01, g2 = .276. Participants made more positive judgments (M = 1.06, SE = .15) than negative judgments (M = .41, SE = .07). The main effect of segment type was also significant,
123
Task oriented reading Table 4 Number of more- and less-relevant text segments identified by experts in the six documents Publisher
Type of document
Number of more-relevant text segments (n = 26)
Number of less-relevant text segments (n = 26)
1. Publishing house
Textbook in nature studies for upper secondary education
2
5
2. National Radiation Protection Agency
Public information text
9
0
3. Illustrated Science
Popular science article
7
2
4. Today’s Medicine
Debate article
5
3
5. Idaho Observer
Polemic
1
7
6. Norwegian liberal daily
Newspaper article
2
9
F(1, 50) = 19.91, MSE = 1.01, p \ .01, g2 = .285. Participants made more judgments at more-relevant segments (M = 1.05, SE = .15) than at less-relevant segments (M = .42, SE = .07). However, these main effects were qualified by a significant judgment type 9 segment type interaction, F(1, 50) = 35.75, MSE = 1.72, p \ .01, g2 = .417. To follow-up the interaction, we ran dependent-sample t tests to assess readers’ relevance judgments while they read more- and less-relevant segments. With respect to more-relevant segments, readers made more positive judgments (M = 1.92, SE = .30) than negative judgments (M = .18, SE = .05), t(50) = 5.72, p \ .01, d = .80. Similarly, with respect to less-relevant segments, readers made more negative judgments (M = .65, SE = .12) than positive judgments (M = .20, SE = .07), t(50) = 3.34, p \ .01, d = .47. Thus, readers were more likely to judge more-relevant segments positively, and they were more likely to judge less-relevant segments negatively, which suggests that readers discriminated between more- and less-relevant information while they read. Our second research question was: Does strategy use differ while individuals read moreand less-relevant information? To address this question, we conducted a 2 (segment type: more-relevant or less-relevant) 9 3 (strategy type: linking, evaluation, and monitoring) within-subjects ANOVA. Means and standard errors are in Table 5. The main effect of segment type was significant, F(1, 50) = 20.35, MSE = 0.54, p \ .01, g2 = .289. Participants used strategies more frequently at more-relevant segments (M = .55, SE = .09) than at less-relevant segments (M = .17, SE = .05). The main effect of strategy type was also significant, F(2, 100) = 7.04, MSE = 0.45, p = .01, g2 = .123. Post hoc tests using LSD showed that participants used linking (M = .53, SE = .10) and evaluation (M = .37,
Table 5 Means for sentence type on each outcome measure (standard errors in parentheses)
More-relevant segments
Less-relevant segments
Relevance judgments Positive
1.92 (0.30)
0.20 (0.07)
Negative
0.18 (0.05)
0.65 (0.12)
Linking
0.92 (1.29)
0.14 (0.40)
Evaluation
0.47 (0.78)
0.27 (0.63)
Monitoring
0.25 (0.56)
0.10 (0.30)
Strategy use
123
Ø. Anmarkrud et al.
SE = .09) more frequently than monitoring (M = .18, SE = .04). However, these main effects were qualified by a significant segment type 9 strategy type interaction, F(1.64, 82.1) = 7.38, MSE = 0.52, p \ .01, g2 = .129. To follow-up the interaction, we ran dependent-sample t tests to compare readers’ use of each strategy for more- and lessrelevant segments. Readers used linking more frequently [t(50) = 4.21, p \ .01, d = .59] at more-relevant segments (M = .92, SD = 1.29) than at less-relevant segments (M = .14, SD = .40). However, readers did not use evaluation more frequently at more-relevant segments (M = .47, SD = .78) than at less-relevant segments (M = .27, SD = .63) [t(50) = 1.87, p = .07, d = .26], nor did they use use monitoring more frequently at morerelevant segments (M = .25, SD = .56) than at less-relevant segments (M = .10, SD = .30) [t(50) = 1.66, p = .10, d = .23], although, overall, there was a trend for readers to use these strategies more frequently while they read more-relevant segments. That readers used linking more frequently at more-relevant segments indicates that there were differences in readers’ online comprehension processes while they read more- and less-relevant information. Our third question was: Are relevance judgments and strategy use while reading related to post-reading essay quality? To address this question, we conducted a correlation analysis including essay quality, relevance judgments, and strategies (see Table 6). On the essaytask, the mean score was 2.61 (SD = 1.90), which indicated that the overall quality of the essays was low given that the maximum possible score was 7. This topic is addressed in more detail in the ‘‘Discussion’’ section. With respect to relevance judgments, there was a positive correlation between essay quality and positive judgments at more-relevant text segments, (r = .33, p = .016). With respect to strategies, there was a positive correlation between essay quality and evaluation at less-relevant text segments (r = .41, p = .003). Thus, judging more-relevant segments as relevant and evaluating less-relevant information were related to the quality of participants’ written argumentation, which indicated that relevance judgments and strategy use were related to post-reading essay quality. In addition, there were several statistically significant correlations pertaining to relevance judgments and strategies. With respect to relevance judgments, there was a positive correlation between positive judgments at more-relevant segments and negative judgments at less-relevant segments (r = .36, p = .009). Thus, there was a relation between judging some information as more-relevant and other information as less-relevant. With respect to strategies, linking at more-relevant segments was positively-related to evaluating at morerelevant segments (r = .45, p = .001). Evaluating at less-relevant segments positivelyrelated to evaluating at more-relevant segments (r = .46, p = .001), linking at morerelevant segments (r = .30, p = .036), and linking at less-relevant segments (r = .32, p = .022). Thus, there was a relation between linking and evaluation strategy use. With respect to relevance judgments and strategies, there was a positive correlation between positive judgments at more-relevant segments and linking at more-relevant segments (r = .28, p = .045). Thus, there was a relation between judging some information as more-relevant and linking at more-relevant segments.
Discussion The purpose of this study was to investigate readers’ online comprehension processes during the reading of multiple documents, and the relations between online processes and offline products. More specifically, we investigated readers’ judgments of text relevance and strategy use while they read, and their relations to post-reading essay performance.
123
* p \ .05 (two-tailed), ** p \ .01 (two-tailed)
11. Less-relevant segments: Evaluating
10. Less-relevant segments: Monitoring
9. Less-relevant segments: Linking
8. More-relevant segments: Evaluating
7. More-relevant segments: Monitoring
6. More-relevant segments: Linking
5. Less-relevant segments: Negative judgments
4. Less-relevant segments: Positive judgments
3. More-relevant segments: Negative judgments
2. More-relevant segments: Positive judgments
1. Essay score
Variable
Table 6 Correlations among outcome variables
.334*
– –
2
1
–
-.056
-.068
3
–
.131
.053
.170
4
–
.026
.072
.363**
-.038
5 .264 .281*
–
-.062
-.196
-.092
6
.168
.152
–
-.055
.109
.033
-.027
7
.051
.023
.180
–
.131
.451**
.044
-.037
8
.020
–
.172
.019
.060
.087
-.038
-.160
-.011
9
–
.218
.140
-.152
.072
-.019
.003
.020
-.144
.244
10
–
.171
.320*
.459**
.080
.295*
-.040
-.048
-.039
.076
.406**
11
Task oriented reading
123
Ø. Anmarkrud et al.
Results showed that readers judged the relevance of more- and less-relevant segments differently and varied their strategy use while they they read more- and less-relevant segments. In addition, students’ judgments of relevance and strategy use were related to the quality of the their written arguments. We aimed to address three main research questions. First, do readers discriminate between more- and less-relevant information while they read? Think-aloud data indicated that readers discriminated between more- and less-relevant information, such that they accurately judged information as more- or less-relevant to the task. While they read morerelevant segments, readers were more likely to judge the segments to be more-relevant. Similarly, while they read less-relevant segments, readers were more likely to judge the segments to be less-relevant. One possible explanation for this finding is that the task instructions gave readers criteria for identifying task-relevant information. However, given that all participants received the same task instructions, this explanation is tentative. Nonetheless, it is consistent with experimental designs which have shown that task instructions affect the criteria readers use to identifying relevant information (e.g., McCrudden et al. 2010). Second, does strategy use differ while individuals read more- and less-relevant information? Think-aloud data indicated that strategy use differed while individuals read more- and less-relevant information. Readers used linking more frequently while they read more-relevant segments than while they read less-relevant segments. One possible explanation for this finding is that deliberately establishing links between different segments is an effortful process. Readers may be more inclined to use linking for morerelevant segments because the benefit of effortful processing outweighs the cost, whereas the same may not hold true for less-relevant information. Again, this explanation is tentative in the absence of experimental control, yet the findings are consistent with previous research that has used reading time and eye-tracking methodologies which generally show that readers allocate more attention to more-relevant information (e.g., Kaakinen and Hyo¨na¨ 2011; Lewis and Mensink 2012). Third, are relevance judgments and strategy use while reading related to post-reading essay quality? Judging more-relevant segments as relevant and evaluating less-relevant information were related to the essay quality, indicating that relevance judgments and strategy use were, indeed, related to post-reading essay quality. Judging more-relevant segments as relevant was related to essay quality. This finding suggests that the ability to identify relevant information helped readers determine whether cell phone use posed health risks. Similarly, evaluating less-relevant information was related to essay quality. This finding suggests that readers may have evaluated less-relevant information and decided to exclude this information from their essays because it was less-relevant. These explanations are speculative because it is unclear how readers ultimately decided what to include or exclude from their essays. Thus, one direction for future research would be to investigate the criteria readers use to include and exclude information from multiple documents while completing a writing task. The MD-TRACE model (Rouet and Britt 2011) provides a framework for investigating task-oriented reading in the context of multiple documents. According to this model, processing of multiple documents in task-oriented reading involves judgments of taskrelevance and processing of information in ways that enable readers to meet their reading goals. Our findings are consistent with this framework and with previous research that has shown that readers allocate attention differently to more- and less-relevant information while they read (e.g., Cerda´n and Vidal-Abarca 2008; Cerda´n et al. 2009; Rouet et al. 2001; Wiley et al. 2009). Data from the present study extends previous research by
123
Task oriented reading
providing insights in the contents of readers’ thoughts as they direct their attention towards more- and less-relevant information in the context of multiple documents reading. The present study provides empirical support for the processing assumptions described in the MD-TRACE model and extends the literature on readers’ online processing of multiple documents during task-oriented reading in three main ways. First, our data demonstrate that readers make conscious and deliberate attempts to discriminate between more- and less-relevant information while reading. Reading involves a combination of passive processes, which involve little or no strategy use, and effortful processes, which involve the use of active and strategic processes in the search for meaning (van den Broek et al. 2005). The presence of relevance judgments suggests that readers use effortful processes to judge the instrumental value of text information while they read. Second, these findings showed that strategy use differed while individuals read moreand less-relevant segments. In particular, readers used linking more often while they read more-relevant segments. This finding suggests that readers had different standards of coherence for information that they judged to be more- or less-relevant to their goals. Standards of coherence refer to readers’ criteria for determining the extent to which they understand a text (van den Broek et al. 2011; van den Broek et al. 1995). Standards of coherence influence the extent to which readers integrate text ideas with each other and with prior knowledge while they read. The fact that readers used linking more frequently at more-relevant segments suggests that readers had different standards of coherence for these segments. Third, the essays provided insights into links between online processes and offline products. Previous research in task-oriented reading has shown relations between online processes and offline products (Cerda´n and Vidal-Abarca 2008; Cerda´n et al. 2009; Kaakinen and Hyo¨na¨ 2011; McCrudden et al. 2010; Rouet et al. 2001; Wiley et al. 2009). In the present study, relevance judgments and strategy use during multiple documents reading were related to post-reading essay quality. It is possible that by evaluating information as less-relevant information, readers may develop a stronger understanding of more-relevant information (Nussbaum 2008). Nonetheless, only positive judgments at more-relevant segments and evaluation at lessrelevant segments were related to essay quality. Previous research has shown that students struggle to write well-reasoned arguments (e.g., Nussbaum 2008; Nussbaum and Schraw 2007; Wolfe et al. 2009; Wolfe and Britt 2008). Thus, it is possible that the participants struggled to transform their thoughts into written arguments. Future research should investigate relations between online processing in multiple documents reading and various measures of offline products (e.g., free recall, inference verification, written argumentation) to assess the extent to which different types of offline measures reflect students’ thinking about controversial topics such as the one in the present study. The present study made it possible to observe how readers engaged in inquiry-driven processing activities while they read multiple documents. It is important to note that we did not intend to compare the effect of task instructions on online processes and offline products; thus, readers did not receive different task instructions. Therefore, it is not possible to infer causality between the task instructions and subsequent processes and products. Thus, one direction for future research is to investigate whether and how different task instructions affect relevance judgments, strategy use, and performance (cf., Hagen et al. in press). We should also caution against generalizing these findings to readers from different populations (e.g., different ages) or to other reading topics. Thus, another direction for future research is to investigate online processes and offline products in multiple documents reading on various topics for readers from different populations.
123
Ø. Anmarkrud et al.
The low scores on the essay task are consistent with previous research which has shown that university students struggle when writing argumentative essays (e.g., Wolfe et al. 2009). The low scores on the essay task make the links between online processes and essay scores provide tenous support for the claim that some online processes are related to essay scores, whereas other processes are not related. As such, more research is needed to understand the links between online processes and written argumentation. Future research could look at ways to in which multiple document tasks could be used as an instructional tool to promote the development of written argumentation. For example, students could read multiple documents on a controversial topic and critique different writing samples. Then they could read an additional set of documents on a different topic and write their own argument. The present study has several instructional implications. First, educators may consider ways to help students develop criteria for identifying more- and less-relevant information while they read. Second, educators may consider helping students develop a repertoire of strategies for thinking about and using this information. In both of these cases, a teacher could use cognitive modeling to scaffold students’ use of relevance judgments and strategy use. Lastly, educators might consider the ways to use multiple documents to help students develop arguments and then provide practice and feedback on their written work. Readers’ ability to construct meaning from multiple documents with overlapping, unique, and conflicting information is integral to the development of reasoning and ultimately influence what people think about controversial topics that have consequences for individuals and groups. Hopefully the result of the present study will motivate researcher to further understand online processes and offline products in task-oriented, multiple documents reading.
References Afflerbach, P. (2002). Verbal reports and protocol analysis. In M. L. Kamil, P. B. Mosenthal, P. D. Pearson, & R. Barr (Eds.), Handbook of reading research (Vol. III, pp. 87–103). Mahwah, NJ: Erlbaum. Afflerbach, P., & Cho, B. (2009). Identifying and describing constructively responsive comprehension strategies in new and traditional forms of reading. In S. Israel & G. Duffy (Eds.), Handbook of research on reading comprehension (pp. 69–90). New York: Routledge. Afflerbach, P., Pearson, P. D., & Paris, S. G. (2008). Clarifying differences between reading skills and reading strategies. The Reading Teacher, 61, 364–373. Bjo¨rnsson, C. H. (1968). La¨sbarhet [Readability]. Stockholm: Liber. Bra˚ten, I., Ferguson, L. E., Anmarkrud, Ø., & Strømsø, H. I. (in press). Prediction of learning and comprehension when adolescents read multiple texts: The roles of word-level processing, strategic approach, and reading motivation. Reading and Writing: An Interdisciplinary Journal. Bra˚ten, I., Gil, L., & Strømsø, H. I. (2011). The role of different task instructions and reader characteristics when learning from multiple expository texts. In M. T. McCrudden, J. P. Magliano, & G. Schraw (Eds.), Relevance instructions and goal-focusing in text learning (pp. 95–122). Greenwich, CT: Information Age Publishing. Bra˚ten, I., & Strømsø, H. I. (2003). A longitudinal think-aloud study of spontaneous strategic processing during the reading of multiple expository texts. Reading and Writing: An Interdisciplinary Journal, 16, 195–218. Bra˚ten, I., & Strømsø, H. I. (2010). When law students read multiple documents about global warming: Examining the role of topic-specific beliefs about the nature of knowledge and knowing. Instructional Science, 38, 635–657. Bra˚ten, I., & Strømsø, H. I. (2011). Measuring strategic processing when students read multiple texts. Metacognition and Learning, 6, 111–130. Britt, M. A., & Larson, A. (2003). Construction of argument representations during on-line reading. Journal of Memory and Language, 48, 749–810.
123
Task oriented reading Britt, M. A., Perfetti, C. A., Sandak, R., & Rouet, J. F. (1999). Content integration and source separation in learning from multiple texts. In S. R. Goldman, A. C. Graesser, & P. van den Broek (Eds.), Narrative, comprehension, causality, and coherence: Essays in honor of Tom Trabasso (pp. 209–233). Mahwah, NJ: Erlbaum. Burton, C., & Daneman, M. (2007). Compensating for a limited working memory capacity during reading: Evidence from eye movements. Reading Psychology, 28, 163–186. Cerda´n, R., & Vidal-Abarca, E. (2008). The effects of tasks on integrating information from multiple documents. Journal of Educational Psychology, 100, 209–222. Cerda´n, R., Vidal-Abarca, E., Martı´nez, T., Gilabert, R., & Gil, L. (2009). Impact of question-answering tasks on search processes and reading comprehension. Learning and Instruction, 19, 13–27. Cirilo, R. K., & Foss, D. J. (1980). Text structure and reading time for sentences. Journal of Verbal Learning & Verbal Behavior, 19, 96–109. Cote´, N., & Goldman, S. R. (1999). Building representations of informational text: Evidence from children’s think-aloud protocols. In H. van Oostendorp & S. R. Goldman (Eds.), The construction of mental representations during reading (pp. 169–193). Mahwah, NJ: Erlbaum. Cote, N., Goldman, S. R., & Saul, E. U. (1998). Students making sense of informational text: Relations between processing and representation. Discourse Processes, 25, 1–53. Ericsson, K. A., & Simon, H. A. (1993). Protocol Analysis: Verbal reports as data. Cambridge, MA: The MIT Press. Fletcher, C. R. (1986). Strategies for the allocation of short-term memory during comprehension. Journal of Memory and Language, 25, 43–58. Gil, L., Bra˚ten, I., Vidal-Abarca, E., & Strømsø, H. I. (2010). Summary versus argument tasks when working with multiple documents: Which is better for whom? Contemporary Educational Psychology, 35, 157–173. Goldman, S. R., Lawless, K. A., Gomez, K. W., Braasch, J., MacLeod, S., & Manning, F. (2010). Literacy in the digital world: Comprehending and learning from multiple sources. In M. G. McKeown & L. Kucan (Eds.), Bringing reading research to life (pp. 257–284). New York: Guilford. Graesser, A., Singer, M., & Trabasso, T. (1994). Constructing inferences during narrative comprehension. Psychological Review, 101, 371–395. ˚ . M., Braasch, J. L. G., & Bra˚ten, I. (in press). Relationships between spontaneous note-taking, Hagen, A self-reported strategies, and comprehension when reading multiple texts in different task conditions. Journal of Research in Reading. Kaakinen, J. K., & Hyo¨na¨, J. (2005). Perspective effects on expository text comprehension: Evidence from think-aloud protocols, eyetracking and recall. Discourse Processes, 40, 239–257. Kaakinen, J. K., & Hyo¨na¨, J. (2011). Online processing of and memory for perspective relevant and irrelevant text information. In M. T. McCrudden, J. P. Magliano, & G. Schraw (Eds.), Text relevance and learning from text (pp. 223–242). Greenwich, CT: Information Age Publishing. Kintsch, W. (1998). Comprehension: A paradigm for cognition. Cambridge, England: Cambridge University Press. Kuhn, D. (2009). Do students need to be taught how to reason? Educational Research Review, 4, 1–6. Larson, A., Britt, M. A., & Kurby, C. (2009). Improving students’ evaluation of informal arguments. Journal of Experimental Education, 77, 339–365. Lehman, S., & Schraw, G. (2002). Effects of coherence and relevance on shallow and deep text processing. Journal of Educational Psychology, 94, 738–750. Lewis, M. R., & Mensink, M. C. (2012). Prereading questions and online text comprehension. Discourse Processes, 49, 367–390. Lorch, R. F., Lorch, E. P., & Klusewitz, M. A. (1993). College students’ conditional knowledge about reading. Journal of Educational Psychology, 85, 239–252. Magliano, J. P., & Graesser, A. C. (1991). A three-pronged method for studying inference generation in literary text. Poetics, 20, 193–232. Magliano, J. P., & Millis, K. K. (2003). Assessing reading skill with a think-aloud procedure. Cognition and Instruction, 21, 251–283. Magliano, J. P., Trabasso, T., & Graesser, A. C. (1999). Strategic processes during comprehension. Journal of Educational Psychology, 91, 615–629. McCrudden, M. T., Magliano, J. P., & Schraw, G. (2010). Exploring how relevance instructions affect personal reading intentions, reading goals, and text processing: A mixed methods study. Contemporary Educational Psychology, 35, 229–241. McCrudden, M. T., Magliano, J. P., & Schraw, G. (2011). Relevance in text comprehension. In M. T. McCrudden, J. P. Magliano, & G. Schraw (Eds.), Text relevance and learning from text (pp. 1–17). Greenwich, CT: Information Age Publishing.
123
Ø. Anmarkrud et al. McCrudden, M. T., & Schraw, G. (2007). Relevance and goal-focusing in text processing. Educational Psychology Review, 19, 113–139. Nussbaum, E. M. (2008). Using argumentation vee diagrams (AVDs) for promoting argument–counterargument integration in reflective writing. Journal of Educational Psychology, 100, 549–565. Nussbaum, E. M., & Schraw, G. (2007). Promoting argument-counterargument integration in students’ writing. Journal of Experimental Education, 76, 59–92. Rapp, D. N., & Mensink, M. C. (2011). Focusing effects from online and offline reading tasks. In M. T. McCrudden, J. P. Magliano, & G. Schraw (Eds.), Text relevance and learning from text (pp. 141–164). Greenwich, CT: Information Age Publishing. Reznitskaya, A., & Anderson, R. C. (2002). The arguments schema and learning to reason. In C. C. Block & M. Pressley (Eds.), Comprehension instruction: Research-based best practices (pp. 319–334). New Yourk: Guilford. Reznitskaya, A., Kuo, L., Glina, M., & Anderson, R. C. (2009). Measuring argumentative reasoning: What’s behind the numbers? Learning and Individual Differences, 19, 219–224. Rouet, J.-F. (2006). The skills of document use. Mahwah, NJ: Erlbaum. Rouet, J.-F., & Britt, M. A. (2011). Relevance processes in multiple document comprehension. In M. T. McCrudden, J. P. Magliano, & G. Schraw (Eds.), Text relevance and learning from text (pp. 19–52). Greenwich, CT: Information Age Publishing. Rouet, J.-F., Vidal-Abarca, E., Bert-Erboul, A., & Millogo, V. (2001). Effects of information search tasks on the comprehension of instructional text. Discourse Processes, 31, 163–186. Schraw, G., Wade, S. E., & Kardash, C. A. (1993). Interactive effects of text-based and task-based importance on learning from text. Journal of Educational Psychology, 85, 652–661. Strømsø, H. I., Bra˚ten, I., & Samuelstuen, M. S. (2003). Students’ strategic use of multiple sources during expository text reading. Cognition and Instruction, 21, 113–147. Toulmin, S. E. (1958). The use of arguments. Cambridge, MA: Cambridge University Press. Trabasso, T., & Magliano, J. P. (1996). Conscious understanding during comprehension. Discourse Processes, 21, 255–287. van den Broek, P., Bohn-Gettler, C., Kendeou, P., Carlson, S., & White, M. J. (2011). When a reader meets a text: The role of standards of coherence in reading comprehension. In M. T. McCrudden, J. P. Magliano, & G. Schraw (Eds.), Text relevance and learning from text (pp. 123–139). Greenwich, CT: Information Age Publishing. van den Broek, P., Lorch, R. F, Jr., Linderholm, T., & Gustafson, M. (2001). The effects of readers’ goals on inference generation and memory for texts. Memory & Cognition, 29, 1081–1087. van den Broek, P., Rapp, D. N., & Kendeou, P. (2005). Integrating memory-based and constructionist processes in accounts of reading comprehension. Discourse Processes, 39, 299–316. van den Broek, P., Risden, K., & Husebye-Hartmann, E. (1995). The role of readers’ standards for coherence in the generation of inferences during reading. In R. F. Lorch & E. J. O’Brien (Eds.), Sources of coherence in reading (pp. 353–374). Hillsdale, NJ: Erlbaum. van den Broek, P., Young, M., Tzeng, Y., & Linderholm, T. (1999). The landscape model of reading: Inferences and the online construction of memory representation. In H. van Oostendorp & S. R. Goldman (Eds.), The construction of mental representations during reading (pp. 71–98). Mahwah, NJ: Erlbaum. Vidal-Abarca, E., Man˜a´, A., & Gil, L. (2010). Individual differences for self-regulating task-oriented reading. Journal of Educational Psychology, 102, 817–826. Vinje, F. E. (1982). Journalistspra˚ket [The journalist language]. Fredrikstad, Norway: Institute for Journalism. Wiley, J., Goldman, S. R., Graesser, A. C., Sanchez, C. A., Ash, I. K., & Hemmerick, J. (2009). Source evaluation, comprehension, and learning in internet science inquiry tasks. American Educational Research Association Journal, 46, 1060–1106. Wineburg, S. (1991). Historical problem solving: A study of the cognitive processes used in the evaluation of documentary and pictorial evidence. Journal of Educational Psychology, 83, 73–87. Wolfe, C. R., & Britt, M. A. (2008). The locus of the myside bias in written argumentation. Thinking and Reasoning, 14, 1–27. Wolfe, C. R., Britt, M. A., & Butler, J. A. (2009). Argumentation schema and the myside bias in written argumentation. Written Communication, 26, 183–209. Wolfe, M. B. W., & Goldman, S. R. (2005). Relations between adolescents’ text processing and reasoning. Cognition and Instruction, 23, 467–502. Zwaan, R. A., & Brown, C. M. (1996). The influence of language proficiency and comprehension skill on situation-model construction. Discourse Processes, 21, 289–327.
123
