THE DISAPPOINTMENT DISTRICTS: Additional Reasons Behind New ...

THE DISAPPOINTMENT DISTRICTS: Additional Reasons Behind New York’s Opt-Out Problem David Wakelyn Union Square Learning [email protected] November 2015

EMBARGOED COPY: NOT FOR RELEASE UNTIL FRIDAY NOVEMBER 20, 2015 12:00 A.M

!

Highlights Why did nearly 200,000 elementary and middle school students opt out of New York’s math and English tests last spring? Data released in August showed that students opting out were more likely to be white, were more likely to have scored at Partially Proficient or Well Below Proficient on the test in 2014, and were less likely to be disadvantaged. The analysis here1 takes a more detailed view, connecting changes in achievement over the last four years with measures of economic advantage. The opt-out movement is also about wealthy school districts struggling to meet the state’s new, challenging standards for college readiness. Opt-out rates were highest in districts that previously had been considered high performing under the old learning expectations, but are now “Disappointments” under Common Core. In spite of their economic advantages, their achievement has fallen to below average levels. The scale and severity of the performance decline is best visualized in Figures 1 & 2 on page 3. Comparisons to states that have also adopted higher standards but do not have high opt-out rates-- Kentucky and California—do not show a similar rise in the number of “Disappointment” districts. High-wealth suburban districts, accounting for 70% of all opt-outs, have been told by the state that 154,000 of their children who had once been considered on track to college readiness, no longer are. Because property valuations are so closely connected with student achievement, it appears parents-- and in some cases educators-- in many high-wealth districts are pushing back, opting out as means to protect their reputation for having “good” schools. A state does not have many tools to discourage parents and students from opting out of tests. However, there are three actions state and district policymakers can take to reinforce the credibility of the new tests and higher standards. 1.! Focus extra implementation support for the “Disappointment” districts and schools. 2.! Improve the amount of feedback to students and teachers. 3.! Get clear on what college readiness really means. These recommendations are apt not only for New York, but for states such as Connecticut, Colorado, New Jersey, Pennsylvania and Washington, which have sizable suburban populations and may see growing opt-out movements in the next few years.

!

What the Data Tells Us About Opt-Outs Initial reporting about who opted out of state tests centered around two explanations. First, parents did not want their children taking multiple-choice tests that are weak proxies for authentic learning. The technology for standardized tests has not evolved much since it was invented in the 1920s and New York educators and parents have complained for decades that the current test formats discourage high quality teaching. States are only in the early stages of adding electronic tools that promise better questions and speedier feedback. “These are highly educated parents who started the movement,” said Dolgeville superintendent Christine Reynolds. “Their rationale is solid. I can’t really argue with them.” Teachers-- upset about the governor’s attempt to increase the amount state test scores are weighted in their evaluations from 20 to 50% —also subtly encouraged families and students to keep their pencils down on test day. Some of their opposition is grounded in their belief that high stakes decisions should be put on hold while teachers figure out what it takes to meet the more challenging college readiness standards. What else does the data tell us? Seventy percent (140,000) of all opt-outs occurred in school districts that in 2012 had above average achievement and advantage (as measured by the percent of students who receive free and reduced lunch). That is, they were considered high achieving before testing of the Common Core State Standards went into effect in 20132. These are the districts shown in blue in Figure 1, all located in the upper right quadrant labeled Tier 4. (The size of each circle corresponds to the size of the district). Among all of these Tier 4 districts, the average opt-out rate in 2015 was 32%, considerably higher than the state average of 18.2%. Before the arrival of Common Core, 61% of the elementary and middle school students in Tier 4 districts were scoring at Proficient and Advanced (Levels 3 & 4) and were considered on track to being college and career ready in 2012. But in 2013, facing higher expectations, that number plummeted to 31.2% in these school districts. And then in the second year of Common Core testing, achievement plummeted again to 27.6%. The opt-out rates were highest among a subset of 109 Tier 4 school districts that struggled greatly in the transition to Common Core. These districts dropped from Tier 4 down into Tier 2 shown in orange, and located in the lower right quadrant. If high-achieving, highpoverty districts in Tier 3 are commonly known as “Beating the Odds”, we can refer to those in Tier 2 as “Disappointment” districts.

UNION SQUARE LEARNING

- 2 -!

“DISAPPOINTMENT DISTRICTS”

Figure 1. 2012 NY English Language Arts, Students Proficient & Advanced, By District

Figure 2. 2014 NY English Language Arts, Students Proficient & Advanced, By District

UNION SQUARE LEARNING

- 3 -!

“DISAPPOINTMENT DISTRICTS”

Comparing the two figures shows a large jump in the number of orange-colored districts. By 2014, the number of “Disappointment” districts had more than doubled, climbing from 79 to 183. The next year, the opt-out rate of the 109 districts that had moved into the “Disappointment” zone averaged 37.5%, a rate twice as large as the state average. While they have eight percent of all students in the state, they represented 15 percent of all opt-outs in 2015. One of the newly added “Disappointment” districts was Brookhaven-Comsewogue in Suffolk County on Long Island. The superintendent, Joseph Rella, was a vocal critic of the new tests, saying, “I’m not encouraging parents to refuse the test, but just to think and be informed.”3 But, he added, “If I had kids that age, they wouldn’t go near that test.” 79% of his students sat out the ELA exam. But in the preceding two years, the district had also seen the percentage scoring at Proficient and Advanced plummet 38%. Out of 1600 3rd to 8th grade students in suburban Comsewogue, 700 fewer were considered on track to being ready for college at the time opt-out fervor grew. What’s intriguing in matching up the 2015 opt-out numbers to prior achievement and poverty levels is the fact that opt-out rates were lowest in districts with below-average advantage, those in Tiers 1 and 3. As Table 1 shows, their opt-out rates were eight and five percent of all students respectively in those tiers, and they also had drops in achievement that were less than the average decline across the state from 2012 to 2014. Table 1. New York Opt-Out Rates & Decline in College Readiness, By Tier Percent Free & Reduced Lunch 2014

Opt-Out Rates 2015

TIER 4

25%

32%

TIER 4 SLIPPING TO TIER 2

38%

37.5%

TIER 3

61%

5%

TIER 2

43%

29%

TIER 1

74%

8%

TOTALS

52%

18%

Number & Percent of All OptOuts 109,895 (55%) 30,504 (15%) 7,369 (4%) 12,880 (7%) 39,358 (20%) 199,287

On Track to College Ready 2012

On Track to College Ready 2013

On Track to College Ready 2014

Decline in College Readiness 2012-2014

72%

46%

44%

-28%

61%

31%

28%

-33%

60%

38%

39%

-21%

51%

25%

24%

-27%

41%

19%

20%

-21%

55%

31%

30%

-25%

Even though 39 of the Beating the Odds districts slipped into Tier 1 by 2014, giving them below average achievement levels, they are still districts where teachers, parents and students have something to prove by performing well on state tests. The advent of Common Core has not represented as much of a dent to their reputation as it has in wealthier districts. In a UNION SQUARE LEARNING

- 4 -!

“DISAPPOINTMENT DISTRICTS”

regression analysis, the decline in the proficiency rate, combined with poverty rates, explains 20 percent of the variation in opt-out rates.4 Not all highly advantaged districts had high opt-out rates. There is a group of 55 Tier 4 districts that were in the bottom quartile of opt-out rates, with an average of nine percent of students not taking the ELA test. They also have very different characteristics than the rest of their Tier 4 peers. They began 2012 with a higher starting point (73.4% scoring Proficient and Advanced). Their drop in 2013, down to 50.6% was not as severe as other districts in Tier 4, and they remained well above state averages. Furthermore, their 2014 results, while down slightly to 49.4%, did not signal continued decline. Among all Tier 4 districts—those who started there in 2012 as well as those who slipped into Tier 2 in 2014-- the state has told parents that 154,000 fewer of their children in grades 3-8 are on the path to college and career readiness. These developments raise the question, how much is the opt-out movement about communities protecting their reputation and property values? In all of the original Tier 4 districts, property wealth was assessed most recently at $709 billion, about half of the state’s total. The 109 districts that moved into the “Disappointment” tier have property values of $503,000, above the state median of $457,000, and income that is slightly above the state median ($124,900 vs. 120,100). These are well-off, but not super-rich suburbs. They have considerably less wealth than the districts that stayed in Tier 4, whose median property value is $618,000. It is entirely possible that these Tier 4 districts, leading the way on opt-outs, are not just upset about test quality and the linking student scores to teacher evaluations; they may be trying to protect their property values and their reputation for having good schools. Research in Florida shows that negative changes in school performance affects housing markets and depresses real estate valuation. Economists David Figlio and Lawrence Kenny have shown that schools experiencing negative accountability information “shocks” also lose financial support from parents and community members, while those experiencing positive accountability information shocks gain financial support.5

States Where Opt-Out Isn’t a Problem: Kentucky and California It is worth comparing the opt-out and achievement trends in New York to changes in two states, Kentucky and California, where opt-out has not been a problem. Kentucky was the first to adopt Common Core and to give tests aligned to the new standards in 2012. California, because of its size and diversity, makes for an interesting comparison as well. Under the new standards, neither state has seen the same movement of highly advantaged districts dropping from Tier 4 in to Tier 2; that is, there’s no “explosion of orange” as there is in the New York graphs.

UNION SQUARE LEARNING

- 5 -!

“DISAPPOINTMENT DISTRICTS”

Most states didn’t start giving Common Core-aligned tests to all students until 2015, but Kentucky was first out of the gate in 2012 with its new tests known as K-PREP. Kentucky’s decline in the first year of tougher standards was sharp, with proficiency rates dropping 24.4 percentage points. Kentucky’s 2011 test scores are no longer available on the state’s website, but the 2010 scores are on the Kids Count website6 and serve as the baseline in Figure 3. (As in the other figures, the size of each circle is proportional to the size of the school district). Sixteen of the state’s 173 districts were in Tier 2. By 2015, that number had not increased but dropped by one as shown in Figure 4. Unlike New York, Kentucky’s “Disappointment” districts are more tightly packed near the state average. Kentucky has given itself time to breathe and adjust to Common Core. To combat the anticipated public backlash, Kentucky partnered with the State Chamber of Commerce on a public relations campaign to show why higher standards were necessary, a full year before the new tests were given. Kentucky has also seen achievement rise each year since the new tests were given. The percentage of students scoring at Proficient and Distinguished has risen from 47.6% in 2012 to 54.4% in 2015. Finally, Kentucky methodically piloted its teacher evaluation system but does not plan to use it to inform personnel decisions until this school year.

UNION SQUARE LEARNING

- 6 -!

“DISAPPOINTMENT DISTRICTS”

Figure 3. Kentucky, Pre-Common Core, 2010

Figure 4. Kentucky, Four Years Post-Common Core, 2015

UNION SQUARE LEARNING

- 7 -!

“DISAPPOINTMENT DISTRICTS”

Figure 5. California, Pre-Common Core, 2013

In August, California released the first year of results from the Smarter Balanced assessment consortium. Comparing where districts stand before and after taking tests aligned to the higher expectations, you see a pattern similar to Kentucky’s. While the bar has been raised, the number of districts in Tier 2 has increased only slightly, growing from 168 to 184. Suburban districts do not appear to be hit harder by the new standards than others in the state. Figure 6. California, One-Year Post-Common Core, 2015

UNION SQUARE LEARNING

- 8 -!

“DISAPPOINTMENT DISTRICTS”

Recommendations for Policy and Practice A state does not have many tools to discourage parents and students from opting out of tests. One suggested penalty, where the state would withhold funds to districts with high opt-out rates, seems too harsh. New York’s education commissioner MaryEllen Elia has started to address parents concerns by shortening the amount of time students spend on tests next year, as well as releasing all of the exam questions to the public.7 The latter is a smart move. In the face of more challenging learning expectations, secrecy does no one any good. Additionally, there are three actions state and district policymakers can take to reinforce the credibility of the new tests and higher standards. These recommendations are apt not just for New York, but for states such as Connecticut, Colorado, New Jersey, Pennsylvania and Washington, which also have sizable suburban populations and emerging opt-out movements. 1.! Focus extra implementation support for the “Disappointment” districts and schools The Common Core standards are built on the best evidence from international comparisons and previous state standards, and represent a massive shift in what states expect students to achieve. Some of the most recent analyses of Common Core show that schools across the country are struggling to implement the new standards well. Education Trust’s review of middle school assignments found that only 40% are aligned with the grade-appropriate standard and only 1 in 20 meet the criteria for a high-quality, Common Core-aligned assignment.8 Research on organizations under threat show that they tend to become rigid and rely on previously held ideas on what they already know how to do, escalating commitment to losing courses of action.9 However, a series of crises may spark flexibility and growth only if the organization is given appropriate support. New York’s EngageNY has received great praise for providing some of the best implementation support and materials anywhere in the country. But the 183 “Disappointment” districts are those that state officials tend to think are doing just fine and do not need help. Perhaps the state can begin a targeted project helping teachers in Disappointment districts to analyze student work and help them arrive at new insights about student misconceptions

UNION SQUARE LEARNING

- 9 -!

“DISAPPOINTMENT DISTRICTS”

2.! Improve the amount of feedback to teachers & students Imagine stepping on a scale in April and finding out how much you weigh in August. Our current testing structure hasn’t changed much since it was invented in the 1920s, and while new technology promises to deliver more timely results, those changes have yet to fully arrive. School districts have added interim tests that are sold as intending to help teachers guide their instruction.10 However, the rhetoric doesn’t match reality. Most school districts use the interim tests as predictors of how students are likely to perform on high-stakes state tests. As a result, teachers get the results, but they often can’t see the questions. Or they get a report, but it doesn’t tell them how each student performed on each standard, thereby making it impossible to know where students went wrong or what they should differently next time. It’s this excessive practice, while also being in the dark about the questions asked, that drives a lot of concern about over-testing. Superintendents should collectively pledge to increase the amount of feedback to shape teaching and learning. Start by allowing teachers and students to see the questions on the practice tests, so they can truly serve as a diagnostic guide for understanding what it takes to meet the new standards. 3.! Get clear on what college readiness really means New York adopted Common Core because of some very real problems in college readiness and completion: nearly 30 percent of undergraduates have to take remedial courses and the SUNY system’s four-year year graduation rate is 47 percent, with another 17 percent taking six years to earn a degree.11 Across the country, many states had compounded the problem by defining proficiency as a fairly low level of learning. In a 2010 speech, Secretary of Education Arne Duncan said, “…we are lying to our children and parents by telling kids they are ready for college, when in fact, they are far from ready.”12 Not too surprisingly then, three-fourths of parents believe their children have graduated high school very or extremely well prepared for college academics, while only half of students agree.13 That said, there is some concern among teachers, principals and superintendents that the state may have set the thresholds for college readiness too high. Figure 7 below shows how most states—with the clear exception of Texas who refused to adopt the Common Core—have redefined proficiency. For the most part, their scores are more in line with the results on the National Assessment of Educational Progress (NAEP). Georgia’s shift has been the most extreme: By their old standards, 94% of fourth-graders were considered proficient in reading in 2013. Under the higher standard, 37% were proficient in 2015, which is slightly above their

UNION SQUARE LEARNING

- 10 -!

“DISAPPOINTMENT DISTRICTS”

NAEP proficiency level of 34%. In contrast, New York is the only state where students’ fourth grade reading achievement on the NAEP now exceeds averages on state tests.14

Figure 7. 2015 NAEP 4th Grade Reading vs. Changes in State Proficiency Rates 33%

New York

59% 36%

Massachusetts

50%

New Jersey

54% 52%

40%

Florida

59%

41%

Texas

61%

54%

39%

Pennsylvania

60%

51%

43%

Kentucky

54% 57%

68%

74% 74%

31%

Georgia

94%

37% 34% 0%

10%

20%

30%

State Proficiency (2012/2013)

40%

50%

State Proficiency (2015)

60%

70%

80%

90%

100%

NAEP Proficiency 2015

The old research arm at the College Board did an analysis for New York in 2011, to determine what SAT scores predicted first year college success, and recommended benchmarks of 1100 on the critical reading and mathematics.15 But the average score for admission to the SUNY comprehensive colleges16 is 1089, and nationally, the SAT college readiness benchmark is in the range of 1030. It’s also important to keep in mind that of those students who do not meet the SAT readiness benchmark but enroll in college anyway, 53 percent graduate within six years.17

UNION SQUARE LEARNING

- 11 -!

“DISAPPOINTMENT DISTRICTS”

In the quest to get more students truly ready for and successful in college, it’s crucial to get the signaling right. State policymakers may want to revisit where they have set the benchmarks for proficiency. At a minimum, they should also consider setting a fifth performance level that sits below the current definition of Proficiency. This is what states in the PARCC testing consortium have done, to indicate that a student has “moderate” command of the standards. Adding a new performance level might give some breathing room to the “Disappointment” districts while also being honest with them, that when their students go to college, they may struggle to graduate.

UNION SQUARE LEARNING

- 12 -!

“DISAPPOINTMENT DISTRICTS”

Endnotes !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!

1

Testing experts discourage comparisons between tests based on different standards, but here we are concerned with the drop in the number of students considered college and career ready from 2012 to 2015, regardless of where proficiency cut scores are set. To do this, we combined Excel files for the last four years available on the NYSED website. District expenditure and revenue profiles can be accessed here and files on number of students who are in the economically disadvantaged subgroup can be accessed here. We used English Language Arts scores only, as this was the area of highest opt-out. To estimate the number of test refusals in each district, we multiplied the 2015 test refusal rate by the number of students tested in grades 3-8 in 2014. ! ! 2 !Common Core replaced New York’s “muddled and vague standards” with ones that defined for all students a clear staircase towards college and career readiness. For more, see Sheila Byrd Carmichael, et al., (2010). The State of State Standards—and the Common Core—in 2010. Thomas B. Fordham Institute. ! 3 !Elizabeth A. Harris and Ford Fessenden (2015, May 21). “’Opt Out’ Becomes Anti-Test Rallying Cry in New York State.” New York Times, A1.!!! ! 4 The regression equation is OPT-OUT RATE= .13 -.24 (ECONDIS) - .82 (14 TO 12 CHANGE) and is significant at the .05 level. Intercept 143155ECONDIS 145TO5125CHANGE%

Coefficients Standard.Error t.Stat P1value Lower.95% Upper.95% Lower.95.0% Upper.95.0% 0.1303371 0.03325832 3.91893263 9.7725E305 0.06503845 0.19563575 0.06503845 0.19563575 30.2444742 0.02989564 38.1775858 1.3724E315 30.3031706 30.1857777 30.3031706 30.1857777 30.828122 0.09770659 38.4755997 1.3923E316 31.019957 30.636287 31.019957 30.636287

5

!David Figlio & Lawrence Kenny, “Public Sector Performance Measurement and Stakeholder Support.” Journal of Public Economics. 93 (9–10), 1069–1077, 2009.! 6

2010 K-PREP scores can be found at http://datacenter.kidscount.org/data#KY/10/0

!

7

Geoff Decker (2015, September 16), “Responding to opt-out movement, New York officials say state tests will be shorter.” Chalkbeat New York. ! 8 Sonia Sanelises and Joan Dabrowski (2015). Checking In: Do Classroom Assignments Reflect Today’s Higher Standards? Education Trust. 9

!Barry M. Staw & Jerry Ross (1987). “Knowing When to Pull the Plug.” Harvard Business Review, March-April 1987, p. 1-7. See also: Barry Staw (1981), “Threat Rigidity Effects in Organizational Behavior: A Multilevel Analysis,” Administrative Science Quarterly, vol. 26, no. 4: 501–24.! !

UNION SQUARE LEARNING

- 13 -!

“DISAPPOINTMENT DISTRICTS”

!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! 10

Nationally, the amount spent on testing has grown from $1.6 to $2.5 billion in the last three years and most of that has been driven by the growth in interim testing. For more, see John Richards & Leslie Stebbins (2014). Behind the Data: PreK-12 Testing and Assessment Market. Software & Information Industry Association. 11

State University of New York (2014). SUNY Fast Facts. Most recently available data for the 2007 cohort as of October 2014. ! 12 Arne Duncan (2010). “The Link Between Standards and Innovation: Secretary Duncan’s Remarks to Innovate to Educate Symposium, Richmond, VA.” October, 27, 2010. ! 13 Achieve, Inc. (2015). “Rising to the Challenge: Are Recent High School Graduates Prepared for College and Work?” ! 14 !We used the most recent NAEP scores from 2015 and matched them with the most recent data available for all states. Florida’s proficiency rate is what the state commissioner has recommended but final cut scores will not be set until January 2016. We used the 2013-14 test scores, or in the case of New York, 2011-12, as the baseline. NAEP defines proficiency as “solid academic performance” for each grade assessed. Students reaching this level have demonstrated competency over challenging subject matter. For more on the variation within and across states, see U.S. Department of Education (2015). Mapping State Proficiency Standards Onto NAEP Scales: Results from the 2013 NAEP Reading and Mathematics Assessments. Institute of Education Sciences, National Center for Education Statistics, 2015-046. ! 15 New York State Education Department. “Methodological Summary The NYS Testing Program Grades 3-8 English Language Arts and Mathematics Common Core Tests. 2013 External Benchmark Studies Summary.” ! 16 Those campuses such as Buffalo State and Old Westbury, outside of the four flagship campuses of Albany, Binghamton, Buffalo and Stony Brook. ! 17 The College Board. 2013 SAT Report on College and Career Readiness. Author.

UNION SQUARE LEARNING

- 14 -!

“DISAPPOINTMENT DISTRICTS”