Massachusetts and Louisiana, which both implement systems that partially use .... Texas schools plan to release 100 perc
TRANSPARENCY IN TESTING RELEASE OF STATE ASSESSMENT QUESTIONS
This analysis by High Achievement New York (HANY) shows how New York State’s exam release practices for Grades 3 through 8 English Language Arts (ELA) and Math assessments compare with those of other states in order to provide an overview of the country’s leading assessment practices, consistency, and transparency. Releasing questions to the public requires taking those questions out of the pool for future exams and creates the need for additional funding and resources for the accelerated creation of new questions. To develop a fair comparison, and given the complexity and expense of releasing questions at such a large scale, the analysis compares the 25 most populous states – of which New York ranks third. The majority of the analyzed states release no labeled items from the prior year’s test. Only PARCC 1 states, New York, Texas, and Massachusetts demonstrate a consistent commitment to the timely release of test questions.
GRADING THE STATES
In May of 2016, New York released 75 percent of the items on the State’s 2015-2016 Grades 3 through 8 ELA and Math exams, as well as 100 percent of its Constructed Response questions. This analysis shows how this practice compares with other states in order to provide an overview and analyze various states’ test release practices, consistency, and transparency. New York, along with the 4 PARCC states in this analysis, scored an A. Texas is close behind with a B+. Massachusetts and Louisiana, which both implement systems that partially use PARCC questions, earned a B+ and C+, respectively. New York’s release of 75 percent of its total 2016 test items and 100% of its constructed response questions, compared to 33 percent for PARCC, makes the State the leader among large states in the nation in consistency, transparency, and commitment to improvement in the public release of annual state assessments for Grades 3 through 8. Each state was awarded a grade based on its education department’s commitment to test transparency by releasing test questions reliably, clearly, and consistently. A point was awarded for a positive response to each of the following factors: 1
PARCC: Partnership for Assessment of Readiness for College and Careers, a set standard of K-12 assessments currently used by a consortium of eight full- and partial-partner states.
High Achievement New York | July 2016
•
Release: Is the state releasing actual test questions from 2016? (Note: half-points given for 2015’s release.) Clarity: Is it clear that the released questions were administered test questions? Growth: Is the state improving by releasing more questions in 2016 than in 2015? Reliability: Are release practices reliable and consistent? And for states that do not release contemporary test questions, did the state update its test resources in 2016?
• • •
Table: Top 25 State Graded for Clarity, Growth, and Reliability on Assessment Release Practices
State
Release
Clarity
Growth
Reliability
Grade
NY
✔
✔
✔
✔
A
CO*
✔
✔
✔
✔
A
IL*
✔
✔
✔
✔
A
MD*
✔
✔
✔
✔
A
NJ*
✔
✔
✔
✔
A
TX
✔
✔
✔
½
B+
MA
✔
✔
½ ½
✔
B+
½ ½
C+
X
LA
✔
VA
½
½ ½
MO
X
X
X
✔
D
PA
X
X
X
D
C-
AL
X
X
X
✔ X
AZ
X
X
X
X
F
FL
X
X
X
X
F
GA
X
X
X
X
F
F
IN
X
X
X
X
F
MI
X
X
X
X
F
MN
X
X
X
X
F
NC
X
X
X
X
F
OH
X
X
X
X
F
SC
X
X
X
X
F
TN
X
X
X
X
F
WI
X
X
X
X
F
CA+ WA+
-
-
-
-
-
* PARCC States + SBAC States
STATE SPOTLIGHTS NEW YORK
High Achievement New York | July 2016
The New York State Education Department has improved upon a number of practices to boost transparency around the state assessments, and has vowed to build on this progress by releasing more information faster and expanding the number of staff reviewing test questions. Specifically, on May 31 of this year, New York released 75 percent of questions used in the state assessments, including 100 percent of constructed response questions. This represents a marked increase over the last three years, growing from 25 percent in 2013 and 50 percent in both 2014 and 2015. The State also released its questions at an earlier date, moving from August in 2015 to May in 2016. The State is also allowing parents and teachers to view their students’ individual constructed response answers for the first time. PARCC S T A T E S Eight states currently use the PARCC assessment system: Colorado; Illinois; Louisiana; Maryland; Massachusetts; New Mexico; Rhode Island; and Washington, DC. Four of these states are among the top 25 in population, and are therefore included in our analysis: Colorado, Illinois, Maryland and New Jersey (Massachusetts and Louisiana are partial partners – see below for more in-depth information on their practices). This year, PARCC released the equivalent of approximately one full test per grade. As three tests are administered for each grade in each consortium state, the released number amounts to 33 percent of total test questions, compared to 16.5 percent in 2015. The release also includes the learning standards associated with each test item, the scoring rubrics, and examples of scored student responses. According to a representative, PARCC will continue to release the same or an increasing percentage of administered questions. “PARCC is committed to releasing at least as many items in each subsequent year, which will demonstrate the diversity and breadth of items.” TEXAS Texas schools plan to release 100 percent of their test questions from the 2016 spring assessments. However, the release practices vary from year to year. For example, in 2015 the state released the full English Language Arts tests and a partial sample of the Math tests, while in 2014, the state released a small sampling of questions for both subjects. 2 The State Education Department has not yet indicated the percentage it plans to release next year. M A S S A CH U S E T T S Beginning in 2015, Massachusetts schools have been able to choose between administering the PARCC exam and the state-designed Massachusetts Comprehensive Assessment System (MCAS) exam. In 2015, 53 percent of MA students in Grades 3 through 8 took the PARCC tests. Both MCAS and PARCC have strong release policies (see above for more information on PARCC release policies). Massachusetts has consistently released 50 percent of the MCAS test items every year since 2011.
2
According to the Texas State Education Department, “a partial release is a subset of test questions that are eligible for the assessment but do not represent the entire test blueprint.”
High Achievement New York | July 2016
In 2017, Massachusetts will begin administering hybrid tests that use materials from both MCAS and PARCC systems. LOUISIANA Louisiana currently implements a hybrid system, using a composite test of which at least 50.1 percent of the test items must be constructed by the state, with the remaining 49.9 percent drawn from the PARCC pool. Louisiana does not release any of the questions from the state-designed portion, instead pointing the public to the 33 percent of questions released by PARCC. Thus, approximately 16.5 percent of Louisiana’s test questions are released via PARCC. VIRGINIA Virginia last released a full test form for English Language Arts in 2015 and for Math in the spring of 2014. This year, the state began a transition to computer-adaptive testing with 6th grade students taking adaptive tests in Math. The state has no plans to release even its non-adaptive test items from the 20152016 school year, and will not release full test forms going forward. S M A R T E R B A L A N CE D S T A T E S Fifteen states belong to the Smarter Balanced Assessment Consortium (SBAC), but the consortium does not currently release assessment questions to the public. Instead, the SBAC uses the adaptive testing model i.e., students are given test questions based on their answers to previous questions in order to gauge their level of understanding. As such, the test consists of a careful and complex fabric of over 30,000 possible questions, and test transparency becomes very expensive and difficult to manage. However, the Smarter Balanced group has announced plans to release 2016 items later this year. Due to the radically different framework of the SBAC tests, the consortium states are excluded from this analysis.
State AL AZ CA CO FL GA IL IN LA MA MD MI MN MO NC
Data: Top 25 State Assessment Release Procedures
Release Procedure
Item specifications and examples of representative, but not administered, test questions; last released in 2015 No resources available without login credentials Smarter Balanced PARCC Sample tests and blueprints available Study guides and blueprints available; last updated in 2015 PARCC Test blueprint available; last updated in 2015 Sample test available; updated yearly Release 50% PARCC Sample tests available; last updated in 2015 Item samples available but not updated with clear labels Release small sample; not for every grade Select sets of 2014 Social Studies and Science tests released
High Achievement New York | July 2016
NJ NY OH PA SC TN TX VA WA WI
PARCC Annual release with improvement Law prohibits release of questions; sample tests are available, but not updated Samples available; updated yearly Samples available; unclear when updated Blueprints available, but not grade-specific; last updated in 2015 Periodic release of full and partial tests Released full test form for Reading in 2015 and Math 2014 Smarter Balanced Online samples and practice tests available
METHODOLOGY
As there is no centralized database and a high degree of state-by-state variation in specific release practices, research was conducted using State Education Department online resources, testing consortia websites, recent news coverage, and phone calls to the Assessment Desks of various Education Departments. Given the lack of standardization across online resources, phone interviews with State Education Department Assessment Employees were conducted for Massachusetts, Texas, Louisiana, Tennessee, Virginia, PARCC, and Smarter Balanced. The decision to focus the largest 50 percent of states is also grounded in the fact that only large states (though not all large states) and consortia groups create and distribute multiple test forms for each grade each year. S T A T E G R A D I N G S CA L E The state grading scale is as follows: “Yes” Answers 0 1 2 3 4
Grade F D C B A
It must be noted that this analysis does not grade the resources provided for administering the tests; for instance, states get no grade for providing blueprints over practice tests. The aim is to provide a grade for the level of transparency following the administration of the tests. Furthermore, this report only analyzes states that release ELA and Math exam questions. States that release exclusively Science and Social Studies assessments are not awarded grades in the analysis.
High Achievement New York | July 2016
