Half-Full or Half-Empty

Journal of Literacy Research, 42:94–99, 2010 Copyright © Taylor & Francis Group, LLC ISSN: 1086-296X print/1554-8430 online DOI: 10.1080/10862960903583301

Half-Full or Half-Empty Isabel L. Beck University of Pittsburgh

When I received the four evaluation studies that appear in this edition of Journal of Literacy Research for commentary, I assumed some folks who read this issue would compare the four state evaluation studies to the Reading First Impact Study Final Report (RFISFR) (Gamse, Jacob, Horst, Boulay, & Unlu, 2008), so I reread that report. For those of you who have not looked at RFISFR, the two big findings were that there was (a) a positive and significant impact on first-grade decoding, the only grade at which it was measured; and (b) no effect on comprehension at grades one, two, and three. What came to mind was, “Hmm : : : half-empty or half-full?” The half-empty interpretation would be something like, “Who cares that they can decode, if they can’t comprehend what they read.” The half-full interpretation would be that improved decoding puts a reader in position for improved comprehension if the other components (e.g., fluency, vocabulary) are in good shape or repaired. A serious problem with educational research is that when something does not work out completely, the field too often fails to build on what did work, or to figure out why a component might not have worked, fix it, and measure again. The half-full, half-empty notion turned out to be a lens through which I started to read the four studies. Upon completion, I rated each of them as half-full. My reason for the half-full categorization is that there were many positive trends, some of which were shown to be significant, that the field could build on and move the glass toward beyond half-full. With that in mind, let us look briefly at a few findings that contributed to my half-full rating for each of the reports. In “Reading First in Florida: Five Years of Improvement,” Foorman, Petscher, Lefsky, and Toste (2010) assessed achievement with the SAT 10 in first and second grades, and the Florida Comprehensive Assessment Test (FCAT) in third. A change score was calculated as percentage of increase in students’ reading Correspondence should be addressed to Isabel L. Beck, University of Pittsburgh, 3939 O’Hara Street, Pittsburgh, PA 15260, USA. E-mail: [email protected]

94

Downloaded from jlr.sagepub.com by guest on January 10, 2016

COMMENTARY

95

at or above grade level and decrease of students at high risk for grades 1, 2, and 3 across five years of implementation. There was a tendency for the percentage of students reading at grade level to increase and those at high risk to decrease. The percentage change from the first to fifth year ranged from 6– 9%; statistical analyses were not done, however, because of large attrition and changing tests (B. Foorman, personal communication, November 2009). Given the above data, some will view that Florida finding as half-empty because of the lack of statistical analyses. Others, myself included, will view the finding as half-full. Who would not celebrate a 6–9% change for a district, a school, a class, a student? Let us look at another Florida finding. In a comparison of students who had been in Reading First (RF) schools from kindergarten through third grade with those who moved once, twice, and more than twice, mobility significantly reduced achievement at both the end of first and third grades. This is in contrast with the RFISFR. The Florida authors view their data as encouraging, and do I. “Encouraging with a direction to build on” is my definition of half-full. In “Second Opinions on the Reading First Initiative: The View from Utah,” Dole, Hosp, Nelson, and Hosp (2010) report impact through results on the Utah Language Arts Criterion-Referenced Tests (CRTs). The phrase in Utah’s title refers to the authors’ point that their results are in contrast to the RFISFR results. Specifically, Utah’s study indicates that RF schools made greater gains than comparison schools or the state totals in percentage of students who performed proficiently on the CRT. Significance tests for those data were not done for reasons similar to Florida, including student attrition and changing measures (J. Dole, personal communication, November 2009). Tests of significance were done on whether there was a difference in achievement in the amount of time spent in RF. There was. The group who spent three years in RF averaged the highest proportion of students who scored at the proficient level. The authors make the point that the RF data are particularly supportive of the RF intervention because it is likely that the comparison schools benefited from RF components. That is, the state of Utah began to require RF features, such as “a three-hour literacy block, use of reading coaches, ongoing progress monitoring, and adoption of a scientifically-based core reading program” (Dole, Hosp, Nelson, & Hosp, 2010, p. 36). It is not certain the extent to which the state’s endorsement of RF features were incorporated into the non-RF schools, but given that Utah is a small state, the ability for the state to enact its instructional views is likely greater than for larger states. Having brought up the size of Utah, it is interesting that Utah is too small a state to conduct an appropriate hierarchical analysis, even if they (researchers) could have secured all the comparison school data needed (J. Dole, personal communication, November 2009). We all need to remember what we all know: States are different from each other, as are districts, schools, and indeed students.


96

BECK

“Reading First in Pennsylvania: Achievement Findings After Five Years” (Bean, Draper, Turner, & Zigmond, 2010) is a third-party evaluation commissioned by the state of Pennsylvania. RF’s impact was assessed with the Pennsylvania System of School Assessment (PSSA), which starts assessment at the third grade. The researchers provide a rationale for only using the PSSA thirdgrade scores because the RF mandate was that third-grade students’ performance be at grade level. The data for five years show year-by-year increases, and the increases from year 1 to year 5 were significant. The Pennsylvania study reports an interesting analysis of achievement in terms of percentage of PA RF schools whose slopes indicated success in “the two goals of Reading First: an increase in the percentage of students at grade level and a reduction in the percentage of students who were seriously below grade level at the end of five years” (Bean, Draper, Turner, & Zigmond, 2010, p. 5). The slope data were divided into three categories: (a) schools that showed both an increase in percent proficient and a decrease in percent at risk—79.2%, (b) schools that showed neither increase or decrease—9.1%, and (c) schools that showed either increase or decrease— 11.7%. My response to those data was, “Wow!” That’s almost 80% of schools, 122 schools that accomplished the two goals. If you did not peruse the table and figure that presented slope information, they are worth seeing. Certainly I had not a moment’s hesitation in characterizing Pennsylvania’s results as half-full. “Reading Achievement in Reading First Schools in Michigan” (Carlisle, Cortina, & Zeng, 2010) was assessed with the Iowa Test of Basic Skills (ITBS), Reading Comprehension subtest. There was significant progress in the improvement of the percentage of students reading at or above grade level for grades 1 and 2 but not grade 3. Significant decreases in the percentage of students reading below grade level were found for grades 1 and 2 but not for grade 3. Carlisle, Cortina, and Zeng suggest several potential explanations for the disappointing results for third grade. One is a sampling issue: RF students were not compared to non-RF students; rather, they were compared to the group used to develop norms for the ITBS. Another explanation was that the tests got harder and required knowledge of more difficult vocabulary and background knowledge. Relatedly, there is an inference that can be made from the following statement in the article: “The test may be increasingly difficult for students, given the comprehension curriculum in RF schools (p. 66).” I jumped to the inference that the authors might be suggesting that the RF comprehension curriculum was inadequate. In my view, a potentially important consideration for why the lack of a third grade effect on comprehension in Michigan and, for that matter, first, second, and third grade effects in the RFISFR, is the way comprehension was likely taught. Recall that the National Reading Panel (NRP; 2000) report was the basis for the “big five” in RF practice. The explicit and direct teaching of phonics was based on hundreds and hundreds of scientific studies accumulated over, at least,


COMMENTARY

97

10 years. Keith Stanovich, a most careful and prominent reading researcher, characterized the research on early reading as “one of the most well established conclusions in all of behavioral science” (Stanovich, 2000, p. 415). Moreover, the ways to teach phonics explicitly and directly is rather obvious, so that the translation of the decoding research into practice is not so complicated. That RF accomplished a positive result in decoding is the most clear research-to-practice manifestation I know of. In contrast, the scientific research base for comprehension instruction, at least at the time the RF initiative was designed, was thinner than for decoding. Among the reasons is that the topics are more complicated than decoding and far more difficult to have the kind of clear research-to-practice path than is the case for decoding. Hence the NRP’s discussion of comprehension was based on limited research. The RF schools were, of course, required to teach comprehension and punted to reliance on the basals’ comprehension strands (which, at the time, had punted to “best practices,” which from my informal review, I would have difficulty calling “best”). RF’s teacher professional development for comprehension was limited compared to professional development for phonics. The support for that assertion comes from many interactions with various state and local RF folks and the state of the research on comprehension that was provided in the NRP. In terms of comprehension, there was an emphasis on comprehension strategies in the NRP, which was incorporated into the primary grades curricula. Two problems are that virtually all the research that the NRP used to support the use of comprehension strategies was done with students beyond the primary grades. There were questions then (Baker, 2002; Carver, 1987), and there are questions now about the value of comprehension strategies (Gerston, 2009; McKeown, Beck, & Blake 2009). Now let me raise one issue that is relevant to the four studies and the RFISFR. All of the studies in one way or another had some mention of engaging or needing to engage in classroom observations (some call it “assessing degree of implementation”). Whatever it is called, the idea is that we need to know what actually goes on in classrooms. The RFISFR used several instruments to collect classroom data. For instance, one instrument tallied such matters as the teacher’s use of explicit instruction, the teacher’s provision of practice opportunities for students, and the amount of time students were being engaged with each of the “big five.” Throughout much of my career, I have thought that the information such categorizations give us is quite limited, mostly because they are silent as to the quality of what’s going on in a category. For instance, let us say that an observer records several instances of a teacher providing explicit instruction. I have seen good explicit instruction. I have seen convoluted and inadequate explicit instruction. And I have seen a lot in between.


98

BECK

So how could big studies capture quality? Some video or audio transcripts of classroom lessons and their subsequent analysis would allow much insight supportive of improving instruction; it certainly has for my colleagues and me at the University of Pittsburgh. I say this here because first, I can, and second, I want to make a plug to the field to get inside of categories. Folks may think about the cost of audio or video transcripts and choke. Well, I’d like to see the difference in cost between (1) some well-timed transcription of classroom lessons, and (2) the careful training of observers and their time making observations, as was done in RFISFR. I tend to be single-minded—let me acknowledge that sometimes that is good and sometimes not good—in what I think will improve instruction. Improving instruction was not the goal of RFISFR, nor is it the goal of the four studies under consideration, but collecting data that has the potential to improve instruction is not disallowed. Including actual transcripts of classroom lesson discourse would be enormously helpful in moving us toward deeply understanding what teachers actually say and do, which is particularly important for designing professional development. I want to tell the four teams of researchers how much I appreciated the disaggregated data they each provided for students who were economically disadvantaged, who were not proficient in English, and who have learning disabilities. This disaggregation begins to provide information about differential outcomes. When reading the Carlisle, Cortina, and Zeng article, I was reminded that the federal government required such disaggregated data for state RF evaluations. I wondered why the federal government did not require their subcontractors to do the same for the national study. While reading the four present studies, I often felt overt appreciation for the work in which the four evaluation teams engaged, especially in the face of many problems. They were not in positions to design studies before the fact; in more than a few cases, they could not get the data that would have allowed for a meaningful comparison; there was much instability at the school level (e.g., principals left); at the classroom level (e.g., new teachers came, and trained teachers left); at the state level (e.g., in one case there were 11 different directors of that state’s RF Initiative); and on and on it went. However, the four evaluation teams persevered and have given the field some useful results and some matters to think about. Finally, with some hesitation, I tell you about a persistent image that I unintentionally evoked when thinking about RFISFR and the four present studies. It was of an elephant—intelligent, steady footed, sure where it was going— and four beavers—smart, exceptionally busy, using what was around to build something useful. I think that image may be pretty corny and I was going to ask some colleagues whether they thought I should include it here. But I didn’t. I decided, “What the heck—I’m emeritus.”


COMMENTARY

99

REFERENCES Baker, L. (2002). Metacognition in comprehension instruction. In C. C. Block & M. Pressley (Eds.), Comprehension instruction: Research-based best practices (pp. 77–95). New York: Guilford Press. Bean, R., Draper, J., Turner, G., & Zigmond, N. (2010). Reading first in Pennsylvania: Achievement findings after five years. Journal of Literacy Research, 42(1), 5–26. Carlisle, J. F., Cortina, K. S., & Zeng, J. (2010). Reading achievement in reading first schools in Michigan. Journal of Literacy Research, 42(1), 49–70. Carver, R. P. (1987). Should reading comprehension skills be taught? In J. E. Readance & R. S. Baldwin (Eds.), Research in literacy: Merging perspectives (Thirty-sixth Yearbook of the National Reading Conference, pp. 115–126). Rochester, NY: National Reading Conference. Dole, J. A., Hosp, J. L., Nelson, K. L., & Hosp, M. K. (2010). Second opinions on reading first: The view from Utah. Journal of Literacy Research, 42(1), 27–48. Foorman, B. R., Petscher, Y., Lefsky, E., & Toste, J. R. (2010). Reading first in Florida: Five years of improvement. Journal of Literacy Research, 42(1), 71–93. Gamse, B. C., Jacob, R. T., Horst, M., Boulay, B., & Unlu, F. (2008). Reading First impact study final report. National Center for Education Evaluation and Regional Assistance, Institute of Education Sciences. Washington, DC: U.S. Department of Education. (NCEE 2009-4038) Gerston, R. (2009, October). No success like failure? Examining the ‘no effects’ phenomenon in education research. Education Week, pp. 24–26. McKeown, M. G., Beck, I. L., & Blake, R. G. K. (2009). Rethinking reading comprehension instruction: A comparison of instruction for strategies and content approaches. Reading Research Quarterly, 44(3), 218–253. National Reading Panel. (2000). Teaching children to read: An evidence-based assessment of the scientific literature on reading and its implications for reading instruction (NIH Pub. No. 004754). Washington, DC: National Institutes of Health. Stanovich, K. E. (2000). Progress in understanding reading: Scientific foundations and new frontiers. New York: Guilford Press.

Isabel L. Beck is professor emeritus at the University of Pittsburgh. Dr. Beck has conducted extensive research on decoding, vocabulary, and comprehension and has published many journal articles and books on these topics. Her 2002 book Bringing Words to Life: Robust Vocabulary Instruction (with Margaret McKeown and Linda Kucan) has become a bestseller. Dr. Beck’s work has been acknowledged by numerous awards, including the Oscar S. Causey award for outstanding research from the National Reading Conference, and the William S. Gray award from the International Reading Association. She is also the recipient of the Contributing Researcher Award from the American Federation of Teachers for “bridging the gap between research and practice,” which is Dr. Beck’s hallmark. Most recently she was elected to the National Academy of Education. Dr. Beck can be reached at 3939 O’Hara Street, University of Pittsburgh, Pittsburgh, PA 15260. E-mail: [email protected]
