56 schools - their writing; and apply standard English conventions (Massachusetts Department of Education,. 2001 ... While the list is not comprehensive, a few.
DEVELOPING A WRITING OBSERVATION PROTOCOL
Developing a Writing Instruction Observation Protocol: Implications for Practice, Research, and Policy
Andrea Winokur Kotula and Cynthia Mata Aguilar Education Development Center, Inc. Terrence Tivnan Harvard University Graduate School of Education
1
DEVELOPING A WRITING OBSERVATION PROTOCOL
Suggested citation: Kotula, A. W., Aguilar, C. M., & Tivnan, T. (2014). Developing a writing instruction observation protocol: Implications for practice, research, and policy. Waltham, MA: Education Development Center, Inc. Copyright © 2014 by Education Development Center, Inc. Education Development Center, Inc. is a global nonprofit organization that creates learning opportunities for people around the world, empowering them to pursue healthier, more productive lives. For more information, visit edc.org.
2
DEVELOPING A WRITING OBSERVATION PROTOCOL
3
Abstract The focus of this article is on the design of a Writing Instruction Observation Protocol to capture the details of writing lessons; the components of effective writing instruction, as indicated by research; the practices that teachers are using when they teach writing; and the effectiveness of those practices, as documented by classroom observers using our protocol. This singular focus on writing was essential, not only because it could inform our research but also because of the Common Core State Standards’ focus on writing, and the potential need in the field for such a protocol. Over a two-year period, we observed fourth-and fifth-grade writing instruction in 259 classrooms in 56 randomly selected treatment and control schools in six districts. What we discovered in the classroom can guide teacher preparation programs, shape professional inservice training, direct curriculum, influence policy, and ultimately support the development of improved writing skills among students.
DEVELOPING A WRITING OBSERVATION PROTOCOL
4
Developing a Writing Instruction Observation Protocol: Implications for Practice, Research, and Policy
If today’s students expect to compete in a global economy, they must possess the skills to think critically and creatively, solve problems, collaborate, and especially communicate (Partnership for 21st Century Skills, 2011). “Writing has never been more important than in this digital age. It is almost inconceivable to achieve academic success without good writing skills” (College Board Advocacy and Policy Center, 2010, p. 2). Writing is essential in the business sector as well (cf. Levy & Murnane, 2004; Murnane & Levy, 1996; National Governors Association Center for Best Practices, 2005), with many arguing that along with being extremely important for daily job performance (National Commission on Writing, 2003), good writing affects hiring and promotions (ACT, 2005). However, “despite the existence of solid models of how to proceed . . . writing is often poorly taught. The reality is that a lot of writing instruction is perfunctory” (College Board’s National Commission on Writing, 2006, p. 12). Teachers in grades 4-6 report that they devote only about 1.25 hours per week on writing instruction; their students spend about two hours each week writing. Moreover, these teachers indicate that they felt unprepared to teach writing after graduating from their teacher preparation programs (Gilbert & Graham, 2010). While much has been written about what constitutes “good” reading instruction, writing instruction has taken a back seat. The National Reading Panel (National Institute of Child Health and Human Development, 2000) has found more than 100,000 published studies about reading. In contrast, the landmark meta-analysis of writing research by Graham and Perin (2007a, 2007b) identifies
DEVELOPING A WRITING OBSERVATION PROTOCOL
5
approximately 600 documents on writing instruction for students in grades 4–12. In fact, writing has been called “the neglected element of American school reform” (National Commission on Writing, 2003, p. 9). The 10 College and Career Readiness Anchor Standards for Writing, part of the Common Core State Standards for English Language Arts & Literacy in History/Social Studies, Science, and Technical Subjects (CCSS) (National Governors Association Center for Best Practices, Council of Chief State School Officers, 2010), become increasingly challenging as students progress in school. In the early grades, students begin to learn how to write opinion pieces, informational/explanatory texts, and narratives. However, in grade six, for example, they must write not only opinion pieces, but arguments that link a concluding statement to the argument and that incorporate claims, logic, evidence, and credible sources. By the time they reach grades eleven to twelve, students need to “write arguments to support claims in an analysis of substantive topics or texts, using valid reasoning and relevant and sufficient evidence” (p. 45). They are expected to develop counterclaims, include words, phrases, clauses, and syntax that “link the major sections of the text, create cohesion, and clarify the relationships between claim(s) and reasons, between reasons and evidence, and between claim(s) and counterclaims” (p. 45). Writing under pressure adds another dimension to the task, and students must “have the flexibility, concentration, and fluency to produce high-quality first draft text under a tight deadline as well as the capacity to revisit and make improvements to a piece of writing over multiple drafts when circumstances encourage or require it” (p. 41). Student Writing Outcomes While calls to improve student writing are not new, the stakes are far higher today than ever before. However, across all grades, only about one-quarter of the students in our nation’s schools
DEVELOPING A WRITING OBSERVATION PROTOCOL
6
are proficient in writing. The most recent National Assessment of Educational Progress (NAEP) Writing Assessment at grades 8 and 12 (National Center for Education Statistics [NCES], 2012) found that only 27% of students at both grades performed at or above the proficient level (“solid academic performance”) in writing; just 54% of eighth graders and 52% of twelfth graders performed at the basic level, and 20% of eighth graders and 21% of twelfth graders obtained writing scores below this level. The NAEP most recently assessed fourth-grade student writing in 2002. Findings from this assessment revealed that only 28% of fourth graders obtained scores at or above the proficient level. Fully 58% of fourth graders scored at the basic level—revealing only a “general grasp” of the writing task—with 14% demonstrating writing skills below this level (Persky, Daane, & Jin, 2003). The writing of students among certain racial and ethnic groups suffers even more. In 2011, 34% of white eighth-grade students obtained scores at or above the proficient level, while only 14% of Hispanic and 11% of black students did so; similarly, 35% of white twelfth-grade students obtained scores at or above the proficient level compared to only 11% of Hispanic and 9% of black twelfth graders (NCES, 2012). There was also a significant gap in writing performance among different racial and ethnic groups for fourth graders in 2002. Whereas 34% of white students obtained scores at or above the proficient level, only 17% of Hispanic and 14% of black students scored at those levels (Persky et al., 2003). While one would expect student writing proficiency to improve over time, findings from the NAEP (Persky et al., 2003) suggest that it does not. By twelfth grade, one in five students still cannot compose texts without a significant number of grammar, spelling, and punctuation errors (National Commission on Writing, 2003). Even among college-bound students, just over
DEVELOPING A WRITING OBSERVATION PROTOCOL
7
one in three high school graduates (36%) cannot meet the readiness benchmarks set for collegelevel writing (ACT, 2013). Overview of Research In response to the national student writing proficiency crisis, we engaged in a four-year research project funded by the US Department of Education, Institute of Education Sciences, to study a writing curriculum for fourth- and fifth-grade classrooms in six school districts. As part of that study, we developed a Writing Instruction Observation Protocol (see Appendix) for classroom observations specific to writing instruction. In this article, we detail our design of the protocol— among the first such instruments in the field and the first that is based on research, was shown to have moderate to high inter-rater reliability, and was used in a study with large numbers of teachers to gauge its feasibility. We also examine one set of findings from our research: the strategies that our study’s teachers used when they taught writing—as documented by classroom observers using our protocol—and the effectiveness of those strategies, as determined by research on the components of effective writing instruction. We examined the observations across treatment conditions to capture a “snapshot” of current writing instruction in our study schools. What we have discovered in the classroom, if validated in future studies, can guide teacher preparation programs, shape professional in-service training, influence policy, direct curriculum, and ultimately support the development of improved writing skills among students. Method Participants We observed writing instruction of 259 teachers (203 fourth-grade classes and 56 fifth-grade classes) in 56 randomly selected treatment and control schools in six school districts in Massachusetts (see Table 1). Districts ranged in size from 13,373 students to 4,496 students.
DEVELOPING A WRITING OBSERVATION PROTOCOL
8
Districts are classified by NCES locale codes according to their distance to urbanized areas. Five of our study districts are considered “suburbs, large,” or a “territory outside a principal city and inside an urbanized area with population of 250,000 or more.” The sixth district is considered a “city, small,” or a “territory inside an urbanized area and inside a principal city with population less than 100,000” (NCES, n.d.) All are Title I districts. There was a high percentage of low-income students in the study (those eligible for free or reduced lunch), 76%. Thirty-two percent received Limited English Proficiency (LEP) services during the study or up to two years prior to it, and 17% were enrolled in special education during either or both years of the implementation. Twenty-five percent of the students were White, 52% Hispanic, and 6% Black. Teacher experience varied from beginning to veteran teachers with over 20 years of experience within and across schools and districts, and participating teachers taught between one and five writing classes in their schools. Procedure The data collection for the study took place over two years. In Year 1, we studied the students in fourth-grade classrooms. In Year 2, we studied a second cohort of fourth-grade classrooms and followed the first cohort of students into their fifth-grade classrooms. We observed all of the fourth-grade teachers who taught writing in Year 1 (2010—2011) in five districts in Massachusetts, or 121 lessons, with a few exceptions because of scheduling or availability issues. In Year 2 of the implementation (2011–2012), one district dropped out of the study due to competing interventions; thus, we observed teachers in four of the Year 1 districts engage a new cohort of fourth-grade students in writing lessons, and we also observed fourth-grade teachers’ writing lessons in a new district that joined the study. In addition, we observed fifth grade
DEVELOPING A WRITING OBSERVATION PROTOCOL
9
teachers. We observed all of the new fourth- and fifth-grade teachers of writing in Year 2 and about 40% of the returning fourth-grade teachers. Of the 178 lessons observed in Year 2, 54 were new fourth grade teachers, 96 were new fifth grade teachers, and 28 were returning fourth grade teachers. We determined the order in which we observed teachers by randomizing within each district within time blocks. If a teacher taught more than one section, we observed one of the classes, making adjustments to the random order of observations for geographical constraints. Five districts participated each year—or a total of six districts across the two years. Measures Development of an observation protocol for writing lessons. To capture the details of the writing lessons, we needed a teacher observation protocol that focused specifically on writing instruction. When it became apparent that no such protocol existed for our purposes, we created our own. In the process of developing it, we needed to (1) identify the elements of “good” writing instruction; (2) examine national and state writing standards, frameworks, and assessments; and (3) examine other observation protocols for formatting suggestions. Elements of good writing instruction. Over the past two decades, a series of metaanalyses and additional research studies have found that teachers need to learn how to teach writing effectively (Coker & Lewis, 2008; Graham & Perin, 2007a). Unfortunately, teachers and curriculum designers tend to believe that writing instruction is easier than it is (Hillocks, 2008). The research identified the following effective teaching practices that support positive writing outcomes:
Providing explicit instruction as part of a systematic writing curriculum that uses a process writing approach (Graham, McKeown, Kiuhara, & Harris, 2012; Graham &
DEVELOPING A WRITING OBSERVATION PROTOCOL
10
Perin, 2007a, 2007b), maintaining a balance among materials, activities, learning tasks, and teacher instruction (Hillocks, 2008)
Scaffolding students’ writing practices to help them while they are learning to be better writers (Graham & Perin, 2007a, 2007b)
Teaching writing strategies for the writing process, including such stages as planning, drafting, revising, and editing (Bui, Schumaker, & Deshler, 2006; Graham et al., 2012; Graham & Perin, 2007a, 2007b; Hillocks, 1986). Providing comments about student writing paired with an opportunity for revision. (Hillocks 1986 found that students gain almost twice as much on measures of their writing as those who just receive comments for the teacher.)
Building students’ capacity to evaluate their own writing using guides or scales (Hillocks, 1986)
Providing instruction of prewriting, summarization, and sentence combining skills (Graham et al., 2012; Graham & Perin, 2007b; Hillocks, 1986).
Ensuring that students have time to study and emulate good models of writing (Graham & Perin, 2007b)
Engaging students in inquiry activities (Graham & Perin, 2007b; Hillocks, 1986)
Helping students to set product goals (Graham et al., 2012; Graham & Perin, 2007b)
Teaching text structure, transcription (spelling, handwriting, keyboarding), and selfregulation (e.g., goal setting, self-assessment); using imagery and creativity; assessing student writing and providing feedback; and increasing time spent writing (Graham et al., 2012)
DEVELOPING A WRITING OBSERVATION PROTOCOL
11
Providing opportunities for peers to assist each other in collaborative writing (Graham et al., 2012; Graham & Perin, 2007b). Engaging students in student-led small-group discussions (Hillocks, 1986, 2008).
Using word processing equipment (Graham et al., 2012; Graham & Perin, 2007a, 2007b)
Including multiple purposes for writing, using varied genres, and reflecting real-world writing needs (Coker & Lewis, 2008; Graham & Perin, 2007a, 2007b)
Focusing on the quality of the writing, not just on the lower-level skills of writing mechanics or word count (Deatline-Buchman & Jitendra, 2006)
All of the above findings on the elements of good writing instruction informed our design of items for our Writing Observation Protocol. National and state standards, frameworks, and assessments. To ensure that our Writing Instruction Observation Protocol reflected expectations for “good writing” in the fourth and fifth grades as set by national and state standards and assessments, we compared the fourth-grade NAEP writing objectives and scoring rubrics to the writing standards and scoring rubrics from our study state’s English language arts curriculum the Massachusetts English Language Arts Curriculum Framework. The NAEP is generally considered the gold standard writing assessment. We also examined the scoring rubrics from the five states with the highest fourthgrade NAEP writing scores in 2002. We found a very close alignment between the Massachusetts standards and the NAEP objectives. In both, students need to write for a variety of purposes; use both narrative and informational formats; write for different audiences; make use of various stimulus materials; employ different stages of a writing process that includes revision; demonstrate organization in
DEVELOPING A WRITING OBSERVATION PROTOCOL
12
their writing; and apply standard English conventions (Massachusetts Department of Education, 2001; Persky et al., 2003). Our examination of the NAEP and MCAS scoring rubrics indicated that the rubrics disagree only on the role of spelling in good writing and writing instruction. The NAEP employs one six-point holistic rubric, while MCAS uses a six-point rubric for “topic/idea development” and a four-point rubric for “standard English conventions.” The NAEP includes spelling only as part of conventions—which is just one element of the holistic score. MCAS does not assess spelling on its rubric for “standard English conventions” and only considers spelling if those errors interfere with communication. We did not include a spelling item in our observation protocol. We examined the writing scoring rubrics of the five states with the highest 2002 NAEP writing scores for three reasons: to determine how they define good writing; to see if they share skill foci with NAEP or MCAS, or both; and to ascertain if there are additional essential areas that they assess. We found close alignment between the NAEP, the MCAS, and the writing scoring rubrics of the five states—Connecticut, New York, North Carolina, Delaware, and Massachusetts. (Note that Delaware assesses writing at the fifth—not fourth—grade level. Also, New York does not assess writing from a stand-alone prompt. Fourth graders in New York answer two extended-response questions: one each in the reading and listening sections.) The writing scoring rubrics for the NAEP and the top-scoring states were perfectly aligned in the skill areas of Organization, Details, Sentence Structure, Word Choice, Revision, and Editing. We have already discussed the alignment with Massachusetts. The differences among the other states are as follows:
Mechanics: not assessed in Connecticut and New York.
DEVELOPING A WRITING OBSERVATION PROTOCOL
Spelling: only assessed in North Carolina.
Grammar and Usage: not assessed in Connecticut or New York.
Audience and Purpose: assessed in New York and Delaware.
13
Because of the close alignment of the scoring rubrics, we did not add items for mechanics, spelling, grammar and usage, or audience and purpose to our protocol. Sample classroom observation protocols. To inform the format of our Writing Instruction Observation Protocol, we reviewed numerous classroom observation protocols to identify their structure, categories, and rating systems. While the list is not comprehensive, a few examples follow: The Oregon Teacher Observation Protocol (OTOP) (Wainwright, Flick, & Morrel, 2003), which consists of 10 items that are scored holistically along a 1–4 scale, or N/O when not observed. We found the use of a holistic scale especially helpful. PEP Observation Protocol (Howard, Shoemaker, Peach, & Tomlinson, 2007), which is an adaptation of the OTOP and adds two capsule description items that summarize the lesson. We incorporated the check-off format of the capsules in some of our items. English Language Observation Development Tool (Sapien-Melchor, 2006), which targets English language development classes but provides rich detail in a check-off format. Four Dimensions of Classroom Practice (Disston, 1997), which begins each item with “How does the teacher,” a question that guided our team’s focus toward what both the teacher and the student are doing during the lesson. In summary, developing our Writing Instruction Observation Protocol was a recursive process; we continuously cross-checked the sources to determine the appropriate placement and wording of items. In addition, we added items the research literature identifies as good
DEVELOPING A WRITING OBSERVATION PROTOCOL
14
instruction, although neither NAEP nor our study state measures them (e.g., planning, revising, and providing feedback to students). The observation protocol. Our Writing Instruction Observation Protocol includes a series of questions and checklist items that represent relatively objective considerations about the organization of the class, the way teachers explain lessons and interact with students, and the types of interactions and feedback included in lessons. The observer’s task is to notice and record these activities during writing lessons. The protocol includes the following four sections (Sections 2 and 3 constitute the two primary blocks of the instrument): Section 1—Introduction: This brief section focuses on the teachers, how they introduce the lessons, and how set the stage for the day’s work. The section addresses three questions fundamental to any lesson: (1) “What instruction is the teacher providing?”; (2) “What are students doing during skill/strategy instruction/practice?”; and (3) “What is the teacher doing while the students are working?” Section 2—Skills/Strategies Instruction/Practice: Section 2 provides 14 questions about what the teachers are doing during this phase of the lesson; what the students are doing and how they are organized; and whether activities involve the whole class, small groups, or individual students. If the lesson focuses on composing, and not on “instruction/practice,” the observer has a space for checking off the section as “not applicable.” The observer also notes whether or not the teacher gives students feedback on their writing as part of the skills/strategies instruction/practice and, if so, the kinds of feedback. Section 3—Composing: Section 3 consists of 10 questions on the portion of the lesson that involves student writing and composing. In this section, the observer notes how the
DEVELOPING A WRITING OBSERVATION PROTOCOL
15
students are working as they compose and in what part of the composing process they are engaged. As in Section 2, the observer has a space to check off the section as “not applicable”—if the lesson focuses on instruction/practice instead of composing—and space to document if the teacher provides feedback and, if so, what kinds of feedback. Section 4—Closure: Section 4 contains miscellaneous questions about whether the teacher provides closure to the lesson and assigns homework that involves writing. Section 4 also asks the observer to document what other adults in the room, aside from the teacher, are doing during the lesson, and provide space for the titles of other adults. The protocol provides space at the end of the page for the observer’s comments. All sections have subcategories for the observer to check off. Some sections record the number of minutes spent on the task and include a category for “other.” Throughout, space is included for observer comments as needed. We designed the protocol so that an observer could easily record whether the teacher provides instruction in skills or strategies, practice, or both and whether the lesson includes composing. The protocol also enables the observer to determine what the teacher and students are doing during different parts of the lesson. We recognized that some features of the questions in the protocol are not appropriate or necessary to all lessons. For example, a writing session that focuses entirely on completing a draft might not include an instructional component and, in that case, guided practice would not be relevant. Similarly, it might be unnecessary to show an example or model a skill or strategy for a lesson that is a continuation from a previous day. Protocol rating scale. After completing Sections 1 through 4 and documenting the details of the writing lesson, observers use the following scale to rate the effectiveness of nine
DEVELOPING A WRITING OBSERVATION PROTOCOL
16
specific features of the lesson (e.g., “How effective was the lesson objective?” “How effective was the feedback?”): 1 = not effective 2 = somewhat effective 3 = effective 4 = very effective Observers indicate if they did not observe a category or if it did not apply to the lesson. Then, to capture the effectiveness of the writing instruction, observers provide an overall rating, which enables them to rank lessons along the same four-point Likert scale. Observers were trained not to average the individual category ratings because some categories are more essential to a specific lesson than others. For a number of reasons—different lessons have different objectives, students vary in their knowledge and skills, some teachers use a writing curriculum and others fashion their own, lessons may share characteristics and yet differ notably in their overall effectiveness—the category ratings were to be used as guidelines and the overall rating scored holistically. The observers provided the summary rating of the overall effectiveness of the writing session using the same 4-point scale of effectiveness. We relied on the observer’s overall rating rather than on an average of the nine individual ratings for two reasons. First, an average of the nine ratings might not capture the variability among lessons of each category’s importance. A second reason for relying on the observer’s overall rating rather than on an average of the nine individual ratings emerged when we looked at the results. As it turned out, the ratings of the nine categories of lesson effectiveness were highly consistent with one another and also highly correlated with the overall rating. The internal consistency reliability (Cronbach’s coefficient alpha) for a composite variable based on an
DEVELOPING A WRITING OBSERVATION PROTOCOL
17
average of the nine ratings was .87, indicating that the lessons rated as more effective on one dimension (e.g., effectiveness of the lesson objective) also tended to be rated as more effective on other dimensions (e.g., feedback or classroom climate). In addition to this relatively high level of internal consistency, the correlation between the average of these ratings and the observer’s overall rating was .86 (p < .001); in other words, the observers’ ratings of overall effectiveness were also consistent with their ratings based on the separate features of instruction. We found both the overall summary rating and the ratings of separate features to be useful parts of the observation protocol and believe they help the observers. Moreover, the separate ratings could be valuable in enabling observers to examine specific features of the lessons. In principle, using an average of several ratings provides an assessment of effectiveness that is potentially more finegrained, but the single overall rating offers a simple and easy-to-use assessment with virtually identical results. Observer training. During the initial implementation of the classroom observations, observers received a full-day training on the use of the protocol conducted by the principal investigator and site coordinator, both experienced literacy researchers. Because of the importance of consensus on terminology, our research team created common definitions for observers to apply to the following terms: “direct vs. indirect instruction,” “discussion,” “modeling vs. examples (models),” “feedback,” “guided practice,” “lesson integrity,” and “classroom climate.” For example, we defined “guided practice” as “a purposeful means by which students can learn, practice, and/or apply new skills/strategies.” To be considered using a guided practice, a teacher needs to be guiding the students—walking around and helping or leading a whole-class or small-group practice. Reviewing a worksheet or homework was not considered guided practice.
DEVELOPING A WRITING OBSERVATION PROTOCOL
18
Inter-rater reliability. As part of the process of developing the observation protocol, multiple observers viewed a sample of writing lessons at the beginning of each year so we could assess the levels of rater agreement and reach an acceptable level of inter-rater reliability. A member of our research team observed classrooms with the new observers, comparing and discussing results after each observation. Two observers rated 34 lessons and three observers rated 24 lessons for a total of 58 lessons. Each rated the lessons independently, with discussions occurring only after the lessons and the ratings were completed. We randomly selected classrooms in order to ensure a variety of settings in this phase of the study. The observers tended to show high levels of agreement when rating the objective features of the lessons—whether the teacher introduced the lesson (from 89 to 96% for the three questions) or whether the instruction included feedback to students about their writing (91%), for example. We examined the percentage of agreement, using Cohen’s kappa to adjust for agreement “by chance” when analyzing the classroom activities variables. Sections 1-4 contained 30 questions that included some subquestions, resulting in a total of 39 items for analysis. For these 39 items, the inter-rater agreement averaged more than 90% (mean = 91, median = 93, SD = 5.3). We analyzed the effectiveness ratings by determining the percentage of agreement, the correlations of the ratings, and the intraclass correlation. For the nine ratings that focused on the effectiveness of the instruction, the inter-rater agreement was still adequate, with an average percentage of exact agreement of just less than 80% (mean = 78, median = 78, SD = 5.9). For the summary rating of the overall effectiveness of the lesson, the raters showed exact agreement of 82%, and the average correlation of the raters’ summary ratings was also adequate (r = .73). Administering the protocol. Each year after we completed our inter-rater reliability study, two observers conducted the remaining observations, one per classroom. Before we
DEVELOPING A WRITING OBSERVATION PROTOCOL
19
conducted any visits, we sent a letter reviewing the procedures and assuring teachers and principals of the confidentiality of the data we collected. To minimize scheduling conflicts, we informed teachers, principals, and district liaisons one week prior to the proposed class visit. Our observers did not share the protocol or provide feedback to the teachers. Results and Discussion Descriptive statistics and overall reliability information on the ratings of lesson effectiveness are presented in Table 2. Observers rated most lessons as “effective” (about 72% of all lessons), while they rated about 11% as “very effective” and 18% as “somewhat effective.” Introducing the Lessons and “Skills/Strategies Instruction Practice” Table 3 presents information on how frequently teachers—across treatment and control schools—used a variety of instructional features while introducing the lessons and during the “Skills/Strategies Instruction/Practice” portion of the lessons. This table indicates the overall frequency of each feature (the “most popular” features are listed first) and also shows the relative frequency of the features of the lessons rated as “very effective,” “effective,” and “somewhat effective.” We created three sets of classrooms based on their effectiveness rating and compared them with a series of cross tabulations. Chi-square tests of independence were used to identify which differences were statistically significant. Introducing the lessons. One of the significant findings was in the way that teachers set the stage for the day’s work and introduced the writing lessons. In almost all cases, they provided an explicit statement of the lesson objective(s) (in 91% of all of the lessons observed). However, they did so in 100% of the lessons rated as “very effective,” but in only 82% of the lessons rated as “somewhat effective.” Similarly, teachers linked a previous lesson to a current lesson in 96% of the lessons rated as “very effective,” but in only 65% of the lessons rated as “somewhat
DEVELOPING A WRITING OBSERVATION PROTOCOL
20
effective.” Activating prior knowledge was also a strategy found significantly more often in lessons rated as “very effective.” We observed it in 74% of those rated “very effective,” but in only 46% of the “somewhat effective” lessons. These three features of lessons—explicit objectives, connections with earlier lessons, and activation of students’ prior knowledge—were the clearest features that distinguished the lessons that were given the highest overall ratings from lessons given ratings of only “somewhat effective.” Skills and strategy instruction and/or practice. While significant variation appeared in the portion of the lessons that focused on the instruction and practice of skills and strategies, the most common approach emphasized direct skill or strategy instruction (96% of the lessons) and practice (94%) during class. In many lessons, teachers also provided examples, asked openended questions, explained how the skills would be helpful, and modeled the process of using the skills. These approaches to explaining, questioning, providing models, and modeling, while observed in 74%, 59%, 50%, and 47% of the lessons, respectively, were observed only slightly more often in the highest-rated lessons; and the differences were statistically significant only in the case of “providing examples of writing related to the lesson”; teachers provided examples in 89% of the lessons rated “very effective” compared to only 63% of the lessons rated as “somewhat effective.” The use of discussion, small-group work, or computers (i.e., keyboarding), as identified by the literature as important elements of effective writing instruction, was rare. It was not common for teachers to engage in class discussions (14% in small groups; 8% whole class).Teachers modeled the skill or strategy they were teaching in fewer than half of the lessons (47%). Because this practice was observed only slightly more often in the highest-rated lessons, our findings did not indicate a statistically significant difference on the effectiveness of
DEVELOPING A WRITING OBSERVATION PROTOCOL
21
teacher modeling as an instructional practice. However, 89% of the lessons in which teachers provided examples of the focal skill or strategy were rated “very effective,” strengthening the evidence in favor of using that practice in writing instruction. In summary, direct instruction was the predominate approach to teaching writing skills or strategies, and teachers usually provided time to practice the relevant skills or strategies. They often posed open-ended or probing questions and elicited student responses. In general, our findings point to the effectiveness of three specific practices in teaching writing skills and strategies: working directly with individual students, providing individualized feedback, and providing examples of the skill or strategy that is the focus of the lesson. Student activity. During the “Skills/Strategies Instruction/Practice” phase of the lesson, students tended to work alone (72%) and write on paper (76%), often giving oral responses to teacher questions (63%). They worked less frequently in small groups (13%) or with partners (26%), and we rarely observed them using computers (1%). Teacher activity. When students were engaged in seatwork, teachers generally circulated around the classroom (76%), often stopping to engage with individual students (77%), and addressed the whole class (70%). We found that when teachers worked with individual students, the effectiveness rating of the instruction increased: 82% of the lessons rated “very effective” included teachers working with individual students compared to 59% of the lessons rated “somewhat effective.” It was much less likely for teachers to work with small groups of students during the “Skills/Strategies Instruction/Practice” phase of the writing lessons (14%). Feedback to students. During the “Skills/Strategies Instruction/Practice” phase of the lessons, most teachers provided feedback to students about their writing (98%); this feedback was mostly verbal (94%), sometimes given to individuals and sometimes to the whole class. We
DEVELOPING A WRITING OBSERVATION PROTOCOL
22
saw the same pattern when teachers gave feedback to individual students as we did when they worked with individuals. Teachers provided feedback to individual students in about 90% of the lessons rated “effective” or “very effective” compared to 68% of the lessons rated “somewhat effective.” Teachers very rarely gave written feedback or any formal grades to students during this part of the lessons. In sum, our findings show that the effectiveness of lessons increased when teachers both work individually with students and provide them with individualized feedback. Student composing and writing in class. In addition to lessons on writing skills and strategies, most of the lessons that we observed also included time for students to write and compose in class. Table 4 presents the results from this phase of the lessons in order of frequency, along with the relative frequencies for the lessons rated as “very effective,” “effective,” and “somewhat effective.” Students benefited from the same kinds of instruction in this part of the lesson that effectively supported their efforts to learn skills and strategies. For instance, 83% of the lessons rated “very effective” included teacher-provided examples of writing that related to instruction compared to only 29% of the lessons rated as “somewhat effective.” Interestingly, this was not a common strategy, occurring in less than one-third of the lessons. Similarly, highly rated lessons were also more likely to have teachers actively model writing as part of the lesson; 33% that were rated “very effective” compared to 7% that were rated “somewhat effective,” Student activity. Similar to what we saw when observing the skills and strategies portion of the lessons, students almost always worked alone (98%) and wrote on paper (97%) during the composing part of the lesson. We occasionally saw them working with partners (12%) or in small groups (8%), but the predominant mode was for them to write on their own. The fact that
DEVELOPING A WRITING OBSERVATION PROTOCOL
23
they almost always worked on their writing tasks alone during the composing part of the lessons cannot be taken as indicative of any inadequacy in the instruction; writing is in and of itself a solitary activity. However, given clear direction and rubrics, small-group work may well succeed at any grade level, and discussion and collaborative writing have been noted in the literature as important components of good writing instruction. Because of its limited use in our study (8%), we can draw no conclusions about its effectiveness. In the lessons that we observed, students generally wrote drafts (78%) about a specific topic (76%) and occasionally engaged in the planning part of the writing process (23%) or in revising earlier work (15%). We very rarely observed students using computers, tablets, and/or laptops to compose (1%). Word processing is an essential writing practice, and it is a concern that it was not observed in more lessons. Teacher activity. Because the focus was on having students work on drafts, teachers tended to circulate around the classroom (93%) and work with individual students (94%) during the composing phase of the lessons. Teachers occasionally addressed the whole class (33%), although this was less common while students were composing, and they sometimes worked with small groups of students (13%). Teacher feedback to students. Virtually all teachers provided some type of feedback on writing (97%) to at least some of the students, almost always in the form of individual comments given verbally (91%) as the teachers moved about the room. Much less frequently, teachers provided whole-class feedback (34%). Written feedback was also rare (17%). Sharing writing. Students frequently shared their writing (in 74% of the lessons). Sharing usually involved individual students reading their compositions aloud to the whole class (68%); only rarely did we see students sharing with partners (8%) or in small groups (3%).
DEVELOPING A WRITING OBSERVATION PROTOCOL
24
Writing genres. In most cases (81%), students worked on imaginative or literary writing. Writing in these genres was very common in the most effective lessons (95% of the “very effective” lessons compared to 72% of the “somewhat effective” lessons). But many lessons also focused on informational writing (43%), and over the course of the school year most classes undertook a variety of writing tasks and topics, such as poetry, opinion pieces, or responding to literature. For some lessons, teachers used a mixed approach in which some students wrote informational reports while others composed fictional or imaginative pieces. The fact that imaginative or literary writing was the most common mode of writing in the lessons we observed suggests potential challenges for teachers and students alike as they come to terms with the CCSS. While the CCSS were not fully implemented at the time of these observations, they had been published, and our study state’s schools have agreed to adopt them. Although it should be noted that the fourth-grade statewide composition test writing prompt was a narrative format, this still does not prepare students for the more demanding genres that constitute the backbone of the CCSS: the expectation that by sixth grade students will be able to write opinion and informational pieces that establish a clear, supporting point of view. Students who are not proficient in writing informational text will not be able to meet the new standards. Even more importantly, they will not be prepared for higher education or the workplace, in which informational text predominates. Closure. About two-thirds (68%) of the lessons came to a formal end with the teachers summarizing some of the key ideas that were covered. Teachers provided closure in 81% of the lessons rated “very effective” compared to 56% of the lessons rated “somewhat effective.” The assignment of writing instruction homework was very rare—in only 8% of the lessons we observed. However, we are not aware of any research that demonstrates a relationship between
DEVELOPING A WRITING OBSERVATION PROTOCOL
25
the assignment of homework and effective writing. Of that 8%, homework involving writing was commonly used as a way to give more practice in specific skills (50%) and to provide practice with other kinds of writing (31%). Study Limitations One limitation of this study is that we only observed each teacher once during the study, with the exception of about 40% of the returning fourth grade teachers in Year 3. Consequently, we may not have adequately captured typical teaching practices in all cases. Our hope is that because we observed teachers in random order and because we observed so many lessons, we were able to obtain a “snapshot” of all the instruction. However, we note that it is possible that we overestimated or underestimated the prevalence of some of the teaching practices. A second limitation is that the classroom observers were trained by the same research team that created the observation protocol. Therefore, the observers rated the effectiveness of each lesson according to the nine features that the team selected and deemed important and were influenced by the other features on the protocol that they observed or did not observe. While we think that all the items on the protocol are driven by the research literature on writing and effective instruction, we may have inadvertently introduced error into the ratings by our own beliefs. Third, we are aware that the relationships we found between the features of instruction and the effectiveness ratings were determined through a series of cross tabulations and that these correlations do not necessarily imply causality. Finally, although the items in our protocol are based on what the professional literature identifies as good writing instruction, we suggest the need for an analysis of student outcome data to explore the predictive validity of the protocol. Implications for Practice, Policy, and Research
DEVELOPING A WRITING OBSERVATION PROTOCOL
26
We created the Writing Instruction Observation Protocol to capture the details of lessons that focus specifically on writing instruction in the intermediate grades. This singular focus on writing was essential not only because of the needs of our research study but also because of the importance of writing in general—both in school and in adult life—and because of the emphasis on writing in the CCSS. Researchers, principals, writing coaches, and other teacher leaders need such a protocol to evaluate and recommend practices, as do teachers themselves if they intend to examine their own classroom effectiveness. The Writing Observation Protocol makes it possible to accurately observe how teachers engage in a very challenging task: helping students improve their writing. The NAEP results, the multiple reports citing the urgency of improving writing instruction in our schools, and the new rigor within the CCSS all compel us to examine our findings with an eye toward practice, future research, and policy. The ability to write a personal narrative, once the centerpiece of many state assessments at the fourth-grade level, is now not enough to demonstrate writing proficiency and growth. Our project’s 259 classroom observations provide baseline data for the state of writing instruction in today’s classrooms. Below are some implications and recommendations based on our findings. Practice. The following practices—which our observers rated as effective, based on the findings from the literature that inform our protocol—should be included daily in writing instruction: Explicitly state the objective of the writing session, explain how the lesson connects to previous instruction—even to instruction from prior years—and activate students’ prior knowledge about the topic.
DEVELOPING A WRITING OBSERVATION PROTOCOL
27
During the instruction or practice of skills or strategies, work with individual students and provide feedback to them about their writing. Feedback needs to be focused, clear, and timely. Written feedback can provide students with documentation to which they can refer when reviewing their writing. When students are composing, provide examples of the skill or strategy that is the focus of the lesson or of the type of writing. Engage students in a discussion about the challenges and successes they experience when using the examples to inform their own writing. Model how to compose the type of writing that is required of students. Help students understand and talk about the different types of writing and the approaches to implementing each of them. We also recommend research-based practices that we did not observe or only observed rarely: Include more informational/expository writing instruction and practice. Work in dyads or small groups, as collaboration has been found to be an effective instructional practice. Provide small-group or whole-class discussions in which students reflect on what their peers and their teachers have said and build on the discussion with new comments. Use computers and/or tablets in the composing process. Research. Our findings indicate many implications for further research. Although there was no significant difference among the various effectiveness ratings for many of the items on our protocol, additional research with a larger and especially more varied sample is needed to test this out. For example, how important is it to tell students how or why what they are learning
DEVELOPING A WRITING OBSERVATION PROTOCOL
28
will help them as writers? As well, are lessons more effective when the teacher asks open-ended and/or probing questions? How important is it for students to share their writing? Are lessons more effective when students share their writing with the whole class than when they work with partners or in small groups? Are discussions about writing in small groups more effective than they are with the whole class? While the amount of instructional time for writing was not a focus of our study, what exactly is the optimal amount of writing instruction and practice per day or week? Research has identified some effective practices. However, that research is scant. We would like to see more research designed to discover additional effective practices, along with a concerted effort by researchers to partner with school districts in randomized control trials to determine which writing practices are truly effective. Moreover, some of the current decisions about effectiveness have been based on a limited number of studies, several of which were of poor quality. It is important to determine—in real classrooms—which combinations of practices help students become better writers. Finally, while we identified several clusters of instructional features, we do not know if any particular combination leads to more effective lessons. There was no single pattern or approach that was consistently associated with higher observer ratings of overall effectiveness. Are there particular clusters that help to improve student writing more than other clusters? Additional research might tell us. Policy. Several policy implications can be drawn from our findings. School administrators need to review their current writing curricula to ensure that they reflect the CCSS and the research on writing. By fourth grade, students are required to write opinion papers on literature and expository texts; their curriculum needs to support the progressive development of these skills. Thus may be important for administrators to examine the types of writing instruction
DEVELOPING A WRITING OBSERVATION PROTOCOL
29
provided in the early grades to ensure it includes more than lessons on writing personal narrative. It is also essential that districts and schools provide professional development that focuses on building teachers’ knowledge and expertise in teaching writing, particularly in using the practices that we (and others) have found to be effective. This training needs to be intensive and ongoing, with mentoring by trained personnel. To enable these changes, school leaders will need to create a culture of writing by allowing staff to examine student writing, perhaps with a structured protocol. They might also consider cultivating in-house teacher-leaders who can support all teachers in writing across the curriculum. School leaders of all kinds could take advantage of the protocol used in this study to examine and inform writing instruction in their classrooms. Finally, to bring about significant change in teaching practices, colleges and universities with teaching preparation programs will need to stress writing instruction, with an emphasis on the CCSS and research on effective writing practices.
DEVELOPING A WRITING OBSERVATION PROTOCOL NOTE: The research reported here was supported by the Institute of Education Sciences, US Department of Education, through Grant Number R305A090479 to Education Development Center, Inc. The opinions expressed are those of the authors and do not represent views of the Institute or the US Department of Education.
30
DEVELOPING A WRITING OBSERVATION PROTOCOL
31
References ACT. (2005). Crisis at the core: Preparing all students for college and work. Washington, DC: Author. Retrieved from http://www.act.org/research/policymakers/pdf/crisis_report.pdf ACT. (2013). The condition of college and career readiness 2013. Washington, DC: Author. Retrieved from http://www.act.org/research/policymakers/cccr13/index.html Bui, Y. N., Schumaker, J. B., & Deshler, D. D. (2006). The effects of a strategic writing program for students with and without learning disabilities in inclusive fifth grade classes. Learning Disabilities Research & Practice, 21(4), 224–260. Coker, D., & Lewis, W. (2008, spring). Beyond writing next: A discussion of writing research and instructional uncertainty. Harvard Educational Review, 78(1), 231–251. College Board Advocacy and Policy Center. (2010). Teachers are the center of education: Writing, learning and leading in the digital age. New York: College Board. College Board’s National Commission on the Writing. (2006). Writing and school reform. New York: College Board. Deatline-Buchman, A., & Jitendra, A. K. (2006, winter). Enhancing argumentative essay writing of fourth-grade students with learning disabilities. Learning Disability Quarterly, 29, 39–54. Disston, J. (1997). Seeing teaching in videos: Using an interpretive video framework to broaden pre-service teacher development. (Master’s project) MACSME Program, University of California, Berkeley. Gilbert, G., & Graham, S. (2010). Teaching writing to elementary students in grades 4–6: A national survey. The Elementary School Journal, 110(4), 494-518.
DEVELOPING A WRITING OBSERVATION PROTOCOL Graham, S., McKeown, D., Kiuhara, S., & Harris. K. R. (2012). A meta-analysis of writing instruction for students in the elementary grades. Journal of Educational Psychology, 104(4), 879–896. Graham, S., & Perin, D. (2007a). What we know, what we still need to know: Teaching adolescents to write. Scientific Studies of Reading, 11(4), 313–335. Graham, S., & Perin, D. (2007b). Writing next: Effective strategies to improve writing of adolescents in middle and high schools – A report to Carnegie Corporation of New York. Washington, DC: Alliance for Excellent Education. Hillocks, G., Jr. (1986). Research on written composition: New directions for teaching. Urbana, IL: ERIC Clearinghouse on Reading and Communication Skills and the National Conference on Research in English. Hillocks, G., Jr. (2008). Writing in secondary schools. In C. Bazerman (Ed.), Handbook of research in writing: History, society, school, individual, text (pp. 311–329). New York: Lawrence Erlbaum. Howard, M., Shoemaker, B., Peach, H., & Tomlinson, D. (2007). PEP observation protocol. Appalachian Mathematics and Science Partnership, Partnership Enhancement Program. Retrieved from http://www2.research.uky.edu/amsp/pub/ Sharepoint%20Toolkit%20Documents/OTOP%20adapted%20for%20PEP%202.doc Levy, F., & Murnane, R. (2004). The new division of labor: How computers are creating the next job market. Princeton, NJ: Princeton University Press. Massachusetts Department of Education. (2001). Massachusetts English language arts curriculum framework. Malden, MA: Author.
32
DEVELOPING A WRITING OBSERVATION PROTOCOL
33
Murnane, R., & Levy, F. (1996). Teaching the new basic skills: Principles for educating children to thrive in a changing economy. New York: Free Press. National Center for Education Statistics. (n.d.). Common core of data: Identification of rural locales. Retrieved from http://nces.ed.gov/ccd/rural_locales.asp National Center for Education Statistics. (2012). The Nation's Report Card: Writing 2011 (NCES 2012–470). Washington, DC: U.S. Department of Education, Institute of Education Sciences. Retrieved from http://nces.ed.gov/nationsreportcard/pdf/main2011/2012470.pdf National Commission on Writing. (2003). The Neglected R: The need for a writing revolution. Retrieved from http://www.host-collegeboard.com/advocacy/writing/publications.html National Governors Association Center for Best Practices. (2005). Reading to achieve: A governor’s guide to adolescent literacy. Retrieved from http://www.nga.org/cms/home/ngacenter-for-best-practices/center-publications/page-edu-publications/col2-content/maincontent-list/reading-to-achieve-a-governors-g.html National Governors Association Center for Best Practices, Council of Chief State School Officers. (2010). Common core state standards for English language arts & literacy in history/social studies, science, and technical Subjects. Washington DC: Author. National Institute of Child Health and Human Development (NICHD). (2000). Teaching children to read: An evidence-based assessment of the scientific research literature on reading and its implications for reading instruction: Reports of the subgroups (NIH Publication No. 00-4754). Washington, DC: U.S. Government Printing Office. Partnership for 21st Century Skills. (2011). Framework for 21st century learning. Washington, DC: Author. Retrieved from http://www.p21.org/overview/skills-framework
DEVELOPING A WRITING OBSERVATION PROTOCOL
34
Persky, H. R., Daane, M. C., & Jin, Y. (2003). The nation’s report card: Writing 2002 (NCES 2003-529). U.S. Department of Education. Institute of Education Sciences. National Center for Education Statistics. Washington, DC: Government Printing Office. Sapien-Melchor, R. (2006, January). English language development observation tool. San Diego County Office of Education. Retrieved from http://www.sdcoe.net/lret2/eld/pdf/ ELDobservationtool.pdf Wainwright, C. L., Flick, L., & Morrel, P. (2003). The development of instruments for assessment of instructional practices in standards-based teaching. Journal of Mathematics and Science: Collaborative Explorations, 6, 21–46.
Table 1 Schools and Classroom Sample for the Observations of Writing Instruction Lessons
Frequency
Teachers observed
259
Grade level
4 5
203 56
School district
A B C D E F
5 schools 9 schools 15 schools 14 schools 7 schools 6 schools
Total
56 schools
2010–2011 2011–2012
119 140
School year
Percent
78.4 21.6
45.9 54.1
DEVELOPING A WRITING OBSERVATION PROTOCOL
36
Table 2 Descriptive Statistics for the Summary Ratings of the Effectiveness of the Lessons
Rated item*
Observed Mean n of Lessons**
SD
Min
Max
1. Lesson objective 2. Activation of prior knowledge 3. Skill/strategy instruction 4. Models or modeling 5. Guided practice 6. Feedback 7. Academic challenge 8. Lesson integrity 9. Classroom climate
231 219
2.88 2.81
.54 .60
1 1
4 4
232
2.07
.60
1
4
196 224 250 255 257 257
2.92 2.94 2.93 2.98 2.93 3.04
.69 .60 .52 .44 .68 .74
1 1 1 2 1 1
4 4 4 4 4 4
Composite (mean of all nine ratings) a
258
2.92
.42
1.4
4.0
Single item overall rating of effectiveness b
257
2.93
.53
2
4
*Rating scale for each item: 1 = not effective; 2 = somewhat effective; 3 = effective; 4 = very effective. ** Observed number of lessons varies slightly because certain ratings were not applicable for some lessons. The composite rating was based on the mean of the relevant ratings. a. Cronbach’s alpha (internal-consistency reliability) for composite scale = 0.87. b. Correlation of composite rating and single-item rating = 0.86, p < .0001.
DEVELOPING A WRITING OBSERVATION PROTOCOL
37
Table 3 Features of Writing Instruction during the Introduction and the Skills and Strategies Phase of the Lessons
Percentage of Lessons in Which Feature Was Noted Phase of the Writing Lesson
Observed Variable
Across All Lessons Lessons Lessons Rated Rated Somewhat Effective Effective
Lessons Rated Very Effective
Introducing the Writing Lesson Teacher explicitly stated 91 the writing session’s objective Teacher explained how 80 previous lessons relate to today’s writing session Teacher activated prior 63 knowledge relevant to today’s writing session
82
92
100
*
65
80
96
*
46
66
74
*
96
90
97
96
94
95
93
100
74
63
75
89
59
65
56
78
Skills and Strategies During Instruction/Practice Teacher provided direct skill/strategy instruction Teacher provided students with in-class time to practice the focus skills/strategies. Teacher provided example of writing related to the instruction and/or practice. Teacher asked openended and/or probing questions.
*
DEVELOPING A WRITING OBSERVATION PROTOCOL
38
Table 3 Features of Writing Instruction during the Introduction and the Skills and Strategies Phase of the Lessons Teacher explained how or why skills/ strategies/ processes will help students as writers (the “why” of instruction). Teacher modeled the process of using focus skills/ strategies. Teacher provided indirect skill/strategy instruction. Teacher engaged students in small group discussion. Teacher adjusted whole class and/or small group skill/strategy instruction and/or practice based on observations of perceived student needs. Teacher engaged students in whole class discussion.
50
39
51
58
47
48
46
58
19
21
20
19
14
20
11
23
12
5
14
11
8
15
5
19
76
64
79
70
72 63
56 73
76 60
67 70
60
64
59
67
37
33
38
41
27
20
31
19
27
30
27
19
What Were Students Doing? Student Activities During Instruction Students worked on paper. Students worked alone. Students gave oral responses. Students worked with class. Students wrote single words or phrases. Students wrote single sentences (not connected). Students marked text.
~
DEVELOPING A WRITING OBSERVATION PROTOCOL
39
Table 3 Features of Writing Instruction during the Introduction and the Skills and Strategies Phase of the Lessons Students worked with a partner. Students wrote more than one sentence (connected text). Students worked with small group. Students worked in “other” way Students worked on computer.
26
18
26
33
25
15
28
19
13
23
11
15
12
10
12
26
1
0
1
4
77
59
80
82
*
76
62
79
78
~
70
72
68
74
14
18
14
7
4
5
4
4
98
100
98
96
96
97
95
96
What Was the Teacher Doing? During student work time, the teacher worked with individuals. During student work time, the teacher circulated around the room. During student work time, the teacher addressed whole class. During student work time, the teacher worked with small groups. During student work time, the teacher did “other.” Teacher Feedback to Students During the Lessons Teacher gave students feedback on their writing or oral responses. If teacher gave
DEVELOPING A WRITING OBSERVATION PROTOCOL
40
Table 3 Features of Writing Instruction during the Introduction and the Skills and Strategies Phase of the Lessons feedback, was the type verbal? If teacher gave feedback, was the type in comments? If teacher gave feedback, was format individual? If teacher gave feedback, was format whole class? If teacher gave feedback, was format small group? If teacher gave feedback, was the type in writing? If teacher gave feedback, was the type in ‘other’ (explain)? If teacher gave feedback, was the type in grades? If teacher gave feedback, was the type in rubrics?
94
92
95
96
87
68
91
89
72
76
69
82
14
21
15
4
7
5
9
4
3
8
2
0
0
0
0
0
0
0
0
0
~ p < .10. * p < .05, comparisons across the lessons rated somewhat effective, effective, and very effective.
*
~
DEVELOPING A WRITING OBSERVATION PROTOCOL
41
Table 4 Features of Writing Instruction during the Composing Phase of the Lessons
Phase of Observed Variable the Writing Lesson
Composing and writing instruction: Student composing and writing in class Teacher provided example of writing related to the composition lesson. Teacher explained how or why skills/strategies/processes will help students as writers (the why of instruction)? Teacher modeled relevant process of writing. Teacher adjusted whole class and/or small group instruction based on observations of perceived student needs.
Percentage of Lessons in Which Feature Was Noted Across Lessons Lessons Lessons All Rated Rated Rated Lessons Somewhat Effective Very Effective Effective
31
29
29
83
28
38
22
40
18
7
22
33
12
15
12
6
98 97
95 95
98 97
100 100
12
13
13
5
8
5
7
20
7
5
8
5
Students during the composing time Students worked alone. Students worked on paper. Students worked with a partner. Students worked with small group. Students worked with
*
~
DEVELOPING A WRITING OBSERVATION PROTOCOL
42
Table 4 Features of Writing Instruction during the Composing Phase of the Lessons class. Students worked on computer. Students worked in ‘other’ way (explain).
1
0
2
0
1
3
1
0
78 76
80 80
80 77
57 62
23
28
22
30
15
20
12
24
6
10
6
5
3
3
3
5
3
3
4
5
1
0
2
0
0
0
1
0
94
95
94
91
93
93
94
91
33
43
32
24
Student tasks during composing time Students wrote to a topic. Students drafted composition. Students planned/organized composition. Students revised composition. Students edited composition. Students gave or received feedback on composition. Students completed final copy. Students reflected on the what-why-how of the lesson. Students freewrote. Teachers during the composing time During student composition time, the teacher worked with individuals. During student composition time, the teacher circulated around the room. During student composition time, the teacher addressed whole
~
DEVELOPING A WRITING OBSERVATION PROTOCOL
43
Table 4 Features of Writing Instruction during the Composing Phase of the Lessons class. During student composition time, the teacher worked with small groups. During student composition time, the teacher did ‘other’ (explain).
13
13
12
19
2
0
4
0
97
98
97
95
94
93
95
91
91
93
90
91
90
88
91
90
34
45
30
43
17
18
17
19
12
10
12
19
1
2
0
5
0
0
0
0
0
2
0
0
74
67
75
82
68
62
68
77
Providing feedback on writing Teacher gave students feedback on their writing. If teacher gave feedback, was format individual? If teacher gave feedback, was the type verbal? If teacher gave feedback, was the type in comments? If teacher gave feedback, was format whole class? If teacher gave feedback, was the type in writing? If teacher gave feedback, was format small group? If teacher gave feedback, was the type in ‘other’ (explain)? If teacher gave feedback, was the type in grades? If teacher gave feedback, was the type in rubrics? Sharing of writing Teacher had students share their writing. Teacher had students share their writing with the whole class.
DEVELOPING A WRITING OBSERVATION PROTOCOL
44
Table 4 Features of Writing Instruction during the Composing Phase of the Lessons Teacher had students share their writing with a partner. Teacher had students share their writing with a small group.
8
8
9
5
3
0
4
4
81
72
81
95
43
48
43
25
68
56
69
81
50
60
50
0
31
40
30
0
8
12
7
4
Writing genres Students are working in imaginative/literary genre. Students are working in informational/expository genre.
~
End of lesson steps Teacher provided closure to lesson. If teacher assigned homework that involved writing, the purpose was to practice the focus of targeted instruction. If teacher assigned homework that involved writing, the purpose was to engage in other kinds of writing. Teacher assigned homework that involved writing.
~ p < .10. * p < .05, comparisons across the lessons rated somewhat effective, effective, and very effective.
~
Appendix Writing Instruction Observation Protocol Date ________________________ District ________________ School _________________ Teacher _____________________ Grade _________________ Observer _______________ Topic ______________________________ Length of Lesson: ___ Minutes
___ New Lesson
___ Continued Lesson
All observation questions refer to the writing instruction
Introduction Does the Teacher: 1. Explicitly state, verbally, the writing session’s objective?
Yes
No
2. Explicitly explain how previous lessons relate to today’s writing session?
Yes
No
3. Activate prior knowledge relevant to today’s writing session?
Yes
No
4. Provide direct skill/strategy instruction?
Yes
No
5. Provide indirect skill/strategy instruction?
Yes
No
6. Provide an example of writing related to the instruction and/or practice?
Yes
No
7. Model the process of using focus skills/strategies?
Yes
No
Skills/Strategies Instruction/Practice (___ minutes) __N/A Does the Teacher:
8. Explain how or why skills/strategies/processes will help students as writers (the why of instruction)?
Yes
No
DEVELOPING A WRITING OBSERVATION PROTOCOL
46
9. Ask open-ended and/or probing questions?
Yes
No
10. Engage the students in whole class discussion?
Yes
No
11. Engage the students in small group discussion?
Yes
No
12. Provide students with time, in class, to practice the focus skills/strategies?
Yes
No
13. Adjust whole class and/or small group skill/strategy instruction and/or practice based on observations of perceived student needs?
Yes
No
14. What are the students doing during skill/strategy instruction/practice? ____ writing single words or phrases ____ writing single sentences (not connected) ____ writing more than one sentence (connected text) ____ marking text ____ oral responses 15. As part of the skills/strategies instruction and/or practice, students are working: (Check all that apply) ___alone
___with a partner
___with a small group
___with the class
___ on paper
___ on computer
___ other (explain):________________________
16. What does the teacher do while students are working? (Check all that apply.) ___ circulates around the room ___ works with individuals ___ works with small groups ___ addresses whole class ___ other (explain): _____________________________________________________________
DEVELOPING A WRITING OBSERVATION PROTOCOL
47
17. Does the teacher give students feedback on their writing or oral responses as part of the skills/strategies instruction and/or practice?
Yes
No
If yes (check all that apply): Format
Types of Feedback
___ individually
___ comments
___ small group(s)
___ grades
___ whole class
___ rubrics
___ written
___ verbal
___ other (explain): ________________
Composing (____minutes) ___N/A Does the Teacher: 18. Provide an example of writing related to the lesson on composition?
Yes
No __ N/A
19. Model the relevant process of writing?
Yes
No __ N/A
20. Explain how or why skills/strategies/processes will help students as writers (the why of instruction)?
Yes
No __ N/A
21. Adjust whole class and/or small group instruction based on observations of perceived student needs?
Yes
No __ N/A
22. Have students working (Check all that apply) ___alone
___with a partner
___with small group
___with class
___ on paper
___ on computer
____other (explain): ______________________
23. In what part of the composing process are students engaged? (Check all that apply) ___planning/organizing
___drafting
___revising
___editing
___providing and/or receiving feedback
___ completing final copy
___ writing to a topic
___ reflecting on what-why-how of lesson
___ freewriting
DEVELOPING A WRITING OBSERVATION PROTOCOL
48
24. What does the teacher do while students are composing? ___ circulates around the room
___ works with individuals
___ works with small groups
___ addresses the whole class
___ other (explain): _____________________________________________
25. Does the teacher give students feedback on their writing?
Yes
No
Yes
No
If yes (check all that apply: Format
Types of Feedback
___ individually
___ comments
___ small group
___ grades
___ whole class
___ rubrics
___ written
___ verbal
___ other (explain):
__________________________________
26. Does the teacher have students share their writing? If yes (check all that apply): ____with a partner ____with small group ____whole class
27. What genre of writing are students working on today? ___ imaginative/literary
type: _______________________________________
___ informational/expository
type: _______________________________________
Miscellaneous 28. Does the teacher provide closure to the lesson?
Yes
No
DEVELOPING A WRITING OBSERVATION PROTOCOL 29. Does the teacher assign homework that involves writing?
49 Yes
No
If yes, the purpose of the homework is ___ to practice the focus of targeted instruction ___ to engage in other kinds of writing
30. What are other adults in the room besides the teacher doing during the lesson? ______________________ __________________________________________________________ Title ______________________ __________________________________________________________ Title ______________________ __________________________________________________________ Title
Additional comments: __________________________________________________________________________________ __________________________________________________________________________________ __________________________________________________________________________________ __________________________________________________________________________________ __________________________________________________________________________________
DEVELOPING A WRITING OBSERVATION PROTOCOL
50
31. Overall, how would you rate the effectiveness of this writing session?
1
2
3
0=Not Observed
1=Not Effective
2=Somewhat Effective
3=Effective
4=Very Effective
Lesson Objective
0
1
2
3
4
Activation/Use of Prior
0
1
2
3
4
0
1
2
3
4
0
1
2
3
4
Guided Practice
0
1
2
3
4
Feedback
0
1
2
3
4
Academic Challenge
0
1
2
3
4
Lesson Integrity
0
1
2
3
4
Classroom Climate
0
1
2
3
4
Categories*
4
N/A
Knowledge Direct Skill/Strategy Instruction Models and/or Modeling
* These categories were developed primarily from our observation protocol. Additional guidance was derived from:
Disston, J. (1997). Seeing teaching in videos: Using an interpretive video framework to broaden preservice teacher development. (Master’s project) MACSME Program, University of California, Berkeley.
DEVELOPING A WRITING OBSERVATION PROTOCOL
51
Howard, M., Shoemaker, B., Peach, H., & Tomlinson, D. (2007. PEP observation protocol. Appalachian Mathematics and Science Partnership, Partnership Enhancement Program. Retrieved from http://www2.research.uky.edu/amsp/pub/ Sharepoint%20Toolkit%20Documents/OTOP%20adapted%20for%20PEP%202.doc Sapien-Melchor, R. (2006, January). English language development observation tool. San Diego County Office of Education. Retrieved from http://www.sdcoe.net/lret2/eld/pdf/ ELDobservationtool.pdf Wainwright, C. L., Flick, L., & Morrel, P. (2003). The development of instruments for assessment of instructional practices in standards-based teaching. Journal of Mathematics and Science: Collaborative Explorations, 6, 21–46.
