Charter School Effects in an Urban School District - Vanderbilt University

Charter School Effects in an Urban School District: An Analysis of Student Achievement Growth

Mark Berends Maria Mendiburo Anna Nicotera Vanderbilt University

Not for Distribution March 2008

Paper Presented at the Annual Meeting of the American Educational Research Association New York, NY

Please direct all correspondence to Mark Berends, Vanderbilt University, Peabody Box #152, 230 Appleton Place, Nashville, TN 37203-5721 ([email protected]). This paper is supported by the National Center on School Choice, which is funded by a grant from the U.S. Department of Education's Institute of Education Sciences (IES) (R305A040043). Funding for Maria Mendiburo and Anna Nicotera was provided by an IES grant to Vanderbilt’s ExpERT program for doctoral training (R305B040110). All opinions expressed in this paper represent those of the authors and not necessarily the institutions with which they are affiliated or the U.S. Department of Education. All errors in this paper are solely the responsibility of the authors. For more information, please visit the Center website at www.vanderbilt.edu/schoolchoice/.

CHARTER ACHIEVEMENT GROWTH IN CROSSROADS

Abstract Charter schools are a popular and growing area of school choice in the United States, but current research about their effects on student achievement is mixed. In this paper, we analyze student achievement growth in an urban district called “Crossroads.” Over the 2002-2003 through the 2005-2006 school years, we compare the achievement levels and growth of charter school students with students enrolled in traditional public schools. We analyze data from the Northwest Evaluation Association (NWEA), which provides fall and spring assessments in reading and mathematics as well as student and school characteristics. Using propensity score matching to compare charter public school students to otherwise similar traditional public school students and cross-classified random effects models to estimate achievement growth, we find that charter students experience an initial drop in achievement when they make a move to a charter school. Charter public students experience higher growth rates than traditional public school students to make up for the initial loss, but this compensatory effect may take a couple of school years.

Not for Distribution

0


CHARTER SCHOOL EFFECTS IN AN URBAN SCHOOL DISTRICT: AN ANALYSIS OF STUDENT ACHIEVEMENT GROWTH Despite the growing popularity of charter schools, the current research about charter school effects on student achievement shows mixed results (see Berends, Springer & Walberg, 2008; Miron and Nelson, 2001; Hill & Christenson, 2006; Gill et al., 2007; Zimmer et al., 2003).1 What has become clear from this growing body of research, however, is that there is a tremendous amount of variability among charter public schools. On the one hand, this variability is encouraging because charter schools are intended to expand the number of educational choices available to parents. Unfortunately, this tremendous variability makes it difficult for researchers to systematically study the student achievement outcomes across a large number of charter

Not for Distribution schools. Although some charter schools are demonstrating higher levels of student achievement, others are performing well below comparable public schools. As new charter schools continue

to open across the country, researchers and policymakers are determined to better understand the

student achievement differences among charter public and traditional public schools in the hopes of understanding the conditions under which charter schools can be effective (Berends et al., 2008d; Betts and Loveless, 2005; Zimmer et al., 2003). In this paper, we examine the achievement effects of a unique charter school initiative located in an urban school district, which will be referred to as “Crossroads.” The charter

schools in Crossroads are unique because unlike other school districts in which multiple agencies have authorized charter schools, all but one of the charter schools in Crossroads were authorized under a special charter school initiative sponsored by the mayor’s office. Each of these schools

1

Charter public schools (CPS) are one of the fastest growing areas of school choice (Berends, Springer, & Walberg, 2008; Lake, 2007). There are now nearly 4,000 serving over a million students within the United Sates (Center for Education Reform, 2006).

1


underwent a rigorous and competitive application process and is held responsible to a comprehensive accountability system. The accountability system used by the mayor’s office uses a combination of standardized testing; site visits by an expert team; surveys of parents, students, and staff; and outside review of schools' finances to evaluate each of the charter schools on an annual basis. Over the last few years, the mayor’s charter school initiative has received national recognition for its innovative accountability system. The recent prestigious recognition of innovation within one level of operation naturally leads to questions of whether this innovation has had an effect on student achievement outcomes. This study takes the first steps towards understanding the charter school effects on student achievement in Crossroads by using propensity score matching and cross-classified

Not for Distribution hierarchical linear modeling to compare rates of growth between CPS and TPS students.

Specifically, we examine the following research question: What are the levels and rates of

achievement growth of Crossroads charter public school students compared to their traditional public school counterparts?

In the following sections, we take a closer look at the defining characteristics and theories about charter school effectiveness and consider how previous research influences our research approach. We then describe the data and the specific models used to test our research question before discussing the results of our analyses. Finally, we suggest directions for future research based on our findings. THEORETICAL PERSPECTIVES ABOUT CHARTER SCHOOLS Among the defining characteristics of charter schools are the following: they are funded by taxpayer dollars; they must accept all who enroll, free of charge; and they are held accountable by a school district or some public entity (Buechler, 1996). Charter schools are also

2


designed to be responsive to the needs of children and families by making fundamental changes in the form and substance of schools and schooling processes (Dressler, 2001). Like a private school, a charter school is “relatively autonomous, but it must attract and keep students, or it will fail” (Buechler, 1996, p. 4). The need to attract and retain students creates a type of consumerbased accountability for charter schools (see Murphy & Shiffman, 2002). Both charter schools and private schools are subject to consumer-based accountability, which is largely absent in the traditional public school system (Betts, 2005).2 There are two competing theories about the possible impact of charter schools on school structure, process, and outcomes—market theory and institutional theory. Many reformers argue that market style mechanisms of consumer choice and competition between autonomous schools

Not for Distribution will encourage diverse and innovative approaches (e.g., Finn &, Gau, 1998; Leonardi, 1998).

The assumption is that with efforts intended to undercut bureaucratic political control of public education, educators in charter schools are given the opportunity and motivation to experiment

with new instructional strategies for improving student achievement (Allen, 2001; Budde, 1988). However, institutional theory posits that actors in uncertain environments look for what has worked well in similar organizations facing uncertainty (Scott & Davis, 2007). Moreover, there are powerful institutional rules held by public opinion and important constituents about how schooling should occur (Meyer & Rowan, 1977; Scott & Davis, 2007). In addition, there are laws and regulations that contribute to conformity and congruency between schools of choice and regular public schools (DiMaggio & Powell, 1983). Thus, rather than predicting that charter schools would be different, unique, and innovative, institutional theory argues that charter

2

An argument could also be made that magnet schools, which are a type of public school, are also subject to consumer-based accountability. However, as schools of choice, magnet schools are not considered traditional public schools.

3


schools should look very similar to traditional public schools in terms of structure, process and outcomes. In what follows, we discuss each of these theoretical perspectives in more detail. Market Theory Reformers argue that allowing parents to choose their child's school will result in marketlike competition and the decline of bureaucratic structures, thus providing parents with greater opportunities for home-school interaction and a greater openness on the part of schools to parents' demands (Chubb & Moe, 1990). Supporters of de-bureaucratization claim that parents, especially low-income and minority parents, will be less intimidated by the school and more willing to make their needs known to school personnel (e.g., Cookson, 1993; Rinehart & Lee 1991), resulting in school processes that will lead to higher achievement. Based on the supply-

Not for Distribution and-demand supposition of market theory, we can imagine a situation in which school

administrators have almost complete control over the mix of services that they provide and the

approaches they use; and a situation in which parents have many choices of schools available for their children (Betts, 2005).

A consumer-based accountability system places heavy emphasis on parental satisfaction. To achieve parental satisfaction, charter schools must be sensitive to the changing needs of students in the school community. According to Murphy & Shiffman (2002), charter schools “represent a maturing of the infusion of market forces into public education over the last 15 years” (p.10). Charter school advocates see public schools as overly bureaucratic and resistant to change. In contrast, the “interplay of autonomy and market forces (e.g. consumer choice and competition) would theoretically make charter schools more innovative and of higher quality than traditional public schools” (Arsen et al., 1999; see also Goldring & Cravens, 2008). Given that charter schools are held to strict accountability requirements based largely on student test

4


scores it is implied that the interplay of autonomy and market forces should also lead to an improvement in student achievement outcomes. The stringent student achievement requirements imposed by No Child Left Behind also highlight the importance of student achievement outcomes for charter schools (Berends et al., 2008a). Institutional Theory Critics of the market model, however, raise questions about the empirical validity of its key assumptions about parent-consumers (demand-side), school (supply-side), and the products that a market in education would generate (Henig, 1999). An alternative theory about the consequences of school choice rests with institutional theory. Institutional theory predicts that choice will not result in innovation and the alteration of organizational conditions, curriculum

Not for Distribution and pedagogy (Goldring & Sullivan, 1995). Institutional theory emphasizes the “powerful

institutional rules” held by public opinion, important constituents, and the laws and regulations (Meyer & Rowan, 1977; Dimaggio & Powell, 1983) that contribute to conformity and

congruency between schools of choice and traditional public schools in terms of teaching and learning. From the institutional perspective, the structure of schools under recent reform

movements is a response to institutional processes rather than a response to technical needs for efficiency and change (Goldring & Sullivan, 1995). The institutional process tends to be “ritually defined meanings and categories” (Scott, 1992, p.279) that may include the rhetoric and legislation surrounding the ideas such as teacher empowerment and school site management, but often do not involve the next steps—e.g., implementing changes in the classroom through new knowledge of teaching and learning. Institutionalization is tied to legitimacy in that organizations facing uncertain environments and outcomes tend to adopt strategies and practices that others have used and are

5


seen as legitimate (Meyer &Rowan, 1977; Scott, 1992). Thus, schools that face uncertain environments and outcomes tend to adopt practices that are seen as legitimate; wide scale innovation is rare. The result is that schools and schooling processes look much more alike than different (Elmore, 2007). Thus, institutionalism provides an explanation for maintenance of a status quo and would predict that charter schools would not exhibit different in-school conditions from non-charter, traditional public schools. CHARTER SCHOOL EFFECTS ON ACHIEVEMENT When considering student achievement, research on charter schools reveals some positive, some negative, and some neutral effects. By conducting a review of 41 studies focusing on student test scores, Hill et al. (2006), found that research on charter schools indicates that the

Not for Distribution overall difference in student achievement outcomes between charter schools and public schools is null or mixed; they also report some studies show positive and others negative effects of

charter schools on student achievement. As in most efforts to examine school effects, this line of research faces the challenge of addressing issues of selection bias and the purported variation of missions across different charter schools (Loveless, 2003). An additional challenge is research design; in fact many of the studies that examine charter schools have weak designs. A few studies, however, have stronger research designs. For instance, several recent studies have investigated the effectiveness of charter schools vis-à-vis traditional public schools using value-added modeling—sharing methodological concerns about selection bias because students who enroll in charter schools are self selected. Such students may be atypical of the larger population of traditional public school students. Thus, comparisons of charter schools and traditional public schools may reflect pre-existing differences in the students rather than the quality of the schools—a major concern when examining school effects. This concern is partly

6


alleviated when schools are evaluated based on value-added scores because researchers can examine the progress students make while enrolled as opposed to looking at their achievement level at just one point in time. However, if students differ not only with respect to their initial level of achievement but also in their rate of growth, the use of value-added measures alone will not remove all selection bias (see Ballou et al., 2006; Bifulco & Ladd, 2006; Hanushek et al., 2005; Sass, 2006; Solomon, Paark & Garcia, 2001). A strong, but unfortunately rare, methodological design to assess achievement effects is examining students who win and lose the lotteries for charter schools that have a significant amount of oversubscription (i.e., some charter schools have more students who want to attend than the schools have open slots available). Three studies of charter school student achievement

Not for Distribution have used this type of research design. Hoxby & Rockhoff (2004) examined achievement effects of students in nine Chicago charter schools; Hoxby & Muraka (2007) are conducting an ongoing study of New York City’s charter schools; and McClure, Strick, Jacob-Almeida & Reicher (2005) examined one school in California, which limits its usefulness for generalization.

Both of the Hoxby studies found that charter schools positively affected the academic achievement of their students. In Chicago, Hoxby & Rockoff (2004) found that charter students in kindergarten through fifth grade, students improved 6 to 7 percentile points in math and 5 to 6 percentile points in reading. In New York City, Hoxby & Murarka (2007) found that charter school students had higher achievement in math and reading compared with their counterparts who lost the lottery. In math, charter students in kindergarten through fifth grade on average had test scores that were 0.09 standard deviations above the average other public school student. In reading, charter students in kindergarten through fifth grade, on average, had test scores that were 0.04 standard deviations above the lotteried-out public school students.

7


In general, the quality of many studies of charter school effects on achievement is highly variable with many studies using only school level test score averages or students’ test scores at one point in time as a measure of outcomes. In their review of charter school research, Betts & Hill et al. (2006) found that of the 55 charter public school achievement studies they reviewed, less than half used student value-added modeling or lottery-based methods to assess achievement effects. In our paper, we rely on student test scores over multiple time points to compare matched CPS and TPS students achievement growth over time. Our methods also allow us the unique opportunity to match students on background characteristics and prior achievement scores before making the move into a charter school. Because of conflicting research evidence and theories (see Berends et al., 2007a;

Not for Distribution Goldring & Cravens, 2008) about what to expect about charter school effectiveness, it is difficult to state a specific hypothesis about achievement effects of charter schools. Despite the

conflicting evidence and theories about expected charter school effects, Crossroads is a charter school initiative that may be producing greater achievement growth for its students than TPS. Because of the support that charter schools have received in Crossroads and because theories positing that charter school autonomy and flexibility leads to educational innovation, which seems to be taking place in Crossroads (Berends, Stein & Smithson, 2008c; Stein, Goldring, & Zottola, 2008), we hypothesize that CPS students are making greater gains in mathematics and

reading achievement than their TPS peers. Yet, when examining the initial move into a charter schools it is likely that this results in a decrease in student achievement for CPS students (see Mehana & Reynolds, 2004; Nicotera, Teasley & Berends, 2007). Thus, we hypothesize that although CPS students may experience an initial loss in achievement levels when transitioning to

8


a charter school, over time their achievement growth will be greater than the growth of TPS students. DATA & METHODS In this paper, we analyze data for CPS and TPS students from Northwest Evaluation Association (NWEA), which contracted with Crossroads to provide interim, or benchmark, assessment data between the 2002-2003 and 2005-2006 school years. Generally, NWEA contracts with states, districts, and schools to provide student assessments in mathematics, reading, and Language Arts. Currently, NWEA tests students in over 2,000 districts in 40 states across the nation. As a testing and research organization, NWEA has loaded in its Growth Research Database (GRD) over 4 million students, 36 million test records, from about 8,200

Not for Distribution schools in over 2,000 districts.

As a part of the National Center on School Choice’s (NCSC) quasi-experimental program

of research, we have entered into a partnership with NWEA for use of this database that to identify sample schools. Currently, we have files cleaned and analyzed for the 2002-2003

through the 2005-2006 school years (e.g., Ballou et al, 2006; Berends et al., 2007; Nicotera, Teasley, & Berends, 2007). In addition, data from the NCES Common Core of Data (CCD) has been merged with the NWEA data files to compile further information about schools. NWEA administers computerized adaptive assessments in the fall and spring of each academic year. All the NWEA subject scores in reading, Language Arts, and mathematics reference a single, cross-grade, and equal-interval scale developed using Item Response Theory methodology (see Hambleton, 1989; Ingebo, 1997; Lord, 1980). The mathematics RIT scale is based on strong measurement theory, and is designed to measure student growth in achievement

9


over time. NWEA research provides evidence that the scales have been extremely stable over twenty years (Kingsbury, 2003; Northwest Evaluation Association, 2002, 2003). When making comparisons between traditional public schools and charter schools, it is important to account for possible differences in the types of students that these schools serve. In the current study, we conduct propensity score analyses at the student level to subsequently examine the achievement growth of students in charter schools compared with matched students in regular public schools. To create this matched set of traditional public school students and charter school students we used a procedure of propensity score matching (Dehejia and Wahba, 2002; Luellen, Shadish, & Clark, 2005; Rosenbaum and Rubin, 1983). This procedure generates a propensity score for each student that is the probability that that student is a charter school

Not for Distribution student. Charter school students are then matched with a traditional public school student who has a similar propensity score (Zimmer and Buddin, 2005). Below we discuss the propensity score matching procedures in more detail, but before we do, we describe the variables in our analyses.

Dependent Variables—Mathematics and Reading Achievement We rely on the fall and spring NWEA test scores in mathematics and reading from the fall of 2002 through the spring of 2006, for a total of eight semesters of testing. Students who were students in Crossroads over this time period are included in the analyses; if students left Crossroads for another district, they were excluded from the analyses. All the NWEA subject scores in mathematics and reading reference a single, cross-grade, and equal-interval scale developed using Item Response Theory methodology (see Hambleton, 1989; Ingebo, 1997; Lord, 1980). The mathematics and reading RIT scales are based on strong measurement theory and are designed to measure student growth in achievement over time.

10


NWEA research provides evidence that the scales have demonstrated to be extremely stable over twenty years (Kingsbury, 2003; Northwest Evaluation Association, 2002, 2003). Level 1 Measures: Student-Level Time-Varying Covariates We include several measures at the student level within our models that can change over time. (See Appendix tables A.1-A.6 for descriptives on the measures used in our analysis.) For instance, we include testing time points in months (GROWTH RATE MONTH) in the analyses to control for the variation in the test administration timing. For each of the eight semesters, the month in which a student took the NWEA test is recorded as the date of the test. A student can have up to eight testing points. The month count is one continuous count throughout the eight semesters starting with September 2002 = 0, increasing by one for each month, ending with May

Not for Distribution 2006 = 43. The increment of time selected for the testing date (e.g., month) becomes the increment over which we are measuring achievement growth.

Students in the treatment group (CHARTER) who enter a charter school after being in a

traditional public school are coded with a 0 for the Charter move until the semester they make

the move from a traditional public school into a charter school, and then these students are coded 1 for all subsequent semesters in the first level of the model. This coefficient gives us the bump in achievement at the testing point that students make the policy relevant move. We measure the number of months after students make the move into a charter school with a measure similar to the month count (DURATION). The duration variable is designed to capture the amount of time that a student has spent in a charter school. The variable is coded as zero until a student enters a charter school; when the student changes schools, the duration begins counting the number of months in the charter school based on the testing date for the first

11


semester in the new school. The duration count will continue to grow with each testing season, provided the student remains in the NWEA database for Crossroads. In addition to these measures that varied over time for individual students, we also included grade level dummies (grades 2-9 with grade 3 as reference dummy). We did not examine students in high school grades 10-12 because of the small number of students tested in those grades in charter schools. Finally, we included a series of interaction terms between grade level and test date (GRADE*DATE) to examine whether growth in achievement is nonlinear across grade levels; between charter treatment and the grade level (GRADE*CHARTER) to see whether the treatment has the same effect across grade levels; and between duration of time in a charter school by

Not for Distribution grade level (GRADE*DURATION) to assess whether the effect of the charter treatment over time varies by grade level.

Level 2 Row Measures: Student-Level Time-Invariant Covariates

We also included several measures for students that do not vary over time, including

dummy variables for students’ race-ethnicity (black or African American, Hispanic, and “other” compared with non-Hispanic white) and gender (male = 1), and grade level (2-9).3 We also included a measure (CHARTER MOVE), indicating that a student moved from a Crossroads traditional public school to one of the charter schools at some point during the four years of testing. Level 2 Column Measures: School Level Variables Students in the NWEA database are linked to the school where they are tested. The NWEA school codes are linked to National Center for Education Statistics (NCES) school

3

The “other” racial-ethnic category includes students who say they are Native American, Asian, multiethnic, or do not report on race-ethnicity.

12


identification numbers that we use to merge with school-level Common Core of Data (CCD) for the four academic years of the study (National Center for Education Statistics, 2007). To control for school level demographic characteristics, we include variables obtained from the CCD for each school in our data: the school’s total enrollment, total percentage of each race/ethnicity category, student-teacher ratio, and the percent of the students in the school receiving free or reduced priced lunches. Propensity Score Matching To prepare the data for analysis, we constrained our student dataset to include students in the 2nd through 9th grades. In addition, if students left the Crossroads district during this time period, they were dropped from the analysis.

Not for Distribution The advantage of our matching procedures used our analysis is that we were able to

match CPS and TPS students on prior achievement before any of the students entered a charter school. With fall and spring testing for students in the sample between 2002-2003 and 20052006 school years, we created cohorts of students to follow over time, starting with fall 2002

through the fall of 2005. These cohorts were created by going back to the first traditional public school test score that students had before entering a charter school. We stratified our sample of students by the grade level to ensure that each student was matched to another student who resided in the same grade level with the Crossroads district. For example, we wanted to compare the typical 4th grader who entered a charter public school the next school year to a similar 4th grader in a traditional public school. This process created 9 ‘bins,’ to which each student was assigned.

13


In the next step of the process we generated a propensity score for all students within each grade level by estimating a probit regression model conditional on student observed characteristics. For example for each bin the following model was fit: Pr( S i ) = β 0 + X i β k + ε i where X is a vector of k observable characteristics of student i predicting a probability of student i being a charter school student S i (1=charter school student, 0= traditional public student [TPS]). The student observable characteristics that were included in this equation were: the prior achievement of the student,4 dummy variables for students’ race-ethnicity and gender. The predicted probability obtained from this estimation is then the propensity score p for each student in the data reflecting the conditional probability that the student is a charter school student.

Not for Distribution A distance score ( d ij = pi − p j ) was then computed for each charter student (i) with

every TPS student (j). Each charter student was matched to the TPS student who minimized d ij .

TPS students were allowed to match to multiple charter students. This was done to ensure that we retained as many charter school students as possible in our matched sample. This form of propensity score matching is known as nearest-neighbor matching with replacement. In some instances the nearest neighbor match to a charter school student may not be sufficiently close

such that one may question the validity of the match. To address this issue we applied a caliper to the distance scores so that only those matches whose distance score was less than .01 were retained in the matched sample. Tables 1 and 2 provide the results of t-tests of the difference in group means between traditional public and charter schools for each student characteristic used in the matching

4

For creating the mathematics analysis sample, we used prior reading score, and for creating the reading analysis sample, we used the prior mathematics score.

14


equation for our total matched samples in reading (Table 1) and mathematics (Table 2). In the unmatched samples (before propensity score matching), there are statistically significant differences between the group means of traditional public and charter school students with respect to black and Hispanic students, prior scores, and some grade level differences. In our reading propensity score model (Table 1), we match students on observables for the mathematics score available before students make the move to a charter school, grade level, gender, and race-ethnicity. Table 1 shows the results for the unmatched and matched sample for students who are “non-movers” (i.e., they never make the move to a charter school) and “charter school movers” (i.e., students who make the move to a charter school). Comparing non-movers to movers in the unmatched reading sample, we see that students significantly differ on the prior

Not for Distribution overall mathematics scores as well as test scores for certain grade levels (i.e., grade 7 and 9). In addition, the unmatched sample differs in terms of the proportion that is black or Hispanic

measure. After propensity score matching, only one of the matched sample measures differs; the mathematics scores for 9th graders who make the move to a charter compared with students who do not make the move. In mathematics (Table 2), a similar pattern emerges. In the unmatched mathematics sample, students who make the move differ from those that do not in terms of prior overall reading scores and the prior reading scores in grade 7. The unmatched sample also differs in terms of the proportion of students who are black or Hispanic. Propensity score matching reduces all of these differences, and none of the measures in the matched sample differs in a statistically meaningful manner. In the models that follow, even though students are matched, we include the observables to examine whether these measures have effects on achievement growth.

15


___________________________________________ Tables 1 & 2 ___________________________________________ Analytic Models We estimated several multi-level models, unconditional and conditional on other student and schools measures, to determine the best fit of the data (Singer & Willett, 2003). Here, we report on three models for each of the two dependent variables, mathematics and reading, based on other analyses that revealed that the full model presented here is the best fitting model.5 First, we modeled the dependent achievement variables as a function of the test time point indicator to determine that both the initial status and growth parameters are statistically significant. Second,

Not for Distribution we added the level-1 time varying covariates, including the charter and duration treatment

dummy variables. Third, we added the student- and school-level time-invariant covariates as regressors on the intercept and the growth rate in the second level of the model. All of the

variables included in the models are un-centered since the value of zero for every variable has

substantive meaning. The model is run using a cumulative specification (Raudenbush & Bryk, 2002). At level one of the final model, each student’s achievement is represented by an individual growth trajectory: Achievementjt = π0jt+ π1jt(Growth Rate Month) + π2jt(Charter)

(1)

+ π3jt(Duration) + π4jt(Grade) + π5jt(Grade*Date) + π6jt(Grade*Charter) + π7jt(Grade*Duration) + ejt, ejt ~ N(0, σ2), where Achievementjt is the expected mathematics or reading outcome and t denotes the time elapsed in months since the month student j was first tested. The initial status and growth

5

These models are available from the authors upon request.

16


trajectory parameters for student j are π0jt and π1jt. We can interpret π0jt as the initial status of achievement for student j and π1jt as that student’s average growth rate in achievement per month over four school years. Charter (π2jt) is interpreted at the bump in achievement that occurs when a student makes the move into a charter school after attending a traditional public school. Duration (π3jt)measures the number of months after a student makes the move into a charter school and can be interpreted as the average growth rate associated with attending a charter school. Duration must be interpreted in tandem with the overall average growth rate represented by Growth Rate Month. Lastly, ejt is a random within-subject residual assumed to be normally distributed with a mean of 0 and a variance of σ2. The level-2 equations of the model include time-invariant covariates for the row and

Not for Distribution column portions of the model. The row variables are student time-invariant covariates. The

column variables are school time-invariant covariates (though schools have unique identifiers for each semester which allows their characteristics to vary over time for students who remain in a school over multiple time points). We will add row and column predictors to the initial status and growth rate equations in level-2. We assume that only the initial status of achievement varies across individuals, represented by the following final level-2 equations:

+ γ01(Male)j + γ02(Black)j + γ03(Hispanic)j + γ04(Other Race)j + γ05(Charter Move)k + β01(School Size)k + β02(% Black)k + β03(% Hispanic)k + β04(% Other)k + β05(Student-Teacher Ratio)k + β06(% Free Lunch)k , + γ11(Male)j + γ12(Black)j + γ13(Hispanic)j + γ14(Other Race)k + γ15 (Charter Move)k + β11 (School Size)k + β12((% Black)k + β13((% Hispanic)k + β14((% Other)k + β14(Student-Teacher Ratio)k + β17(% Free Lunch)k

17


, , , , , , ), ),

where θ0 represents the mean initial achievement status and θ1 represents the mean achievement growth rate across students. After the student-level row components are controlled for, such as gender, race-ethnicity, and charter ever indicator, b00j is the residual random effect associated with student j on achievement. The random school effect for achievement is c00k. Dtjk = 1 if

Not for Distribution student j attends school k at time h and otherwise, Dtjk = 0 with the double summation cumulating the school effect c00k over time. The school-level residual random effect is the expected

deflection in the initial status associated with attending school k after school-level covariates are accounted for in the model. The row-level indicators include the dummy variables for male,

race-ethnicity, and ever attending a charter school over the four school years. The column-level measures include school-level variables for enrollment, minority composition, and the percentage of the school population that is eligible for free or reduced price lunch. RESULTS We first estimated unconditional models that examine whether initial status and growth rates in mathematics and reading achievement vary across students in our propensity score matched samples (see Table 3). Both the initial status and growth rate parameters for both subjects are statistically significant and the χ2 statistics for row level and column level variance

18


components lead us to reject the null hypotheses and conclude that students vary significantly in their initial levels of mathematics and reading achievement. ___________________________________________ Table 3 ___________________________________________ Although the pattern of results in reading and mathematics comparing CPS and TPS student end up to be quite similar, there are more coefficients that are statistically significant in reading than in mathematics, so our presentation of results focuses on reading. Students in the sample who make a move to a charter school experience an initial drop in achievement, based on the estimates in the best-fitting full model for reading controlling for

Not for Distribution student and school demographics as well as several interaction terms. In the full model in Table 3, we see that 3rd grade students who make a move to a charter school experience an average

decrease in achievement by 8.65 points (p < 0.001). Taking the average across grade levels, we find that students, upon entering a charter schools, experience an average decrease in achievement by about 3.87 points.

Generally, over the course of one school year charter school students do not make up the initial lost in achievement, but the longer a student stays in a charter school, the greater the reading achievement growth compared with TPS students. Calculating the average gains and growth is complicated, particularly due to the best-fitting model, which includes interaction terms, indicating that charter school effects are non-linear across grade levels, achievement growth differs across grade levels, and the effects of attending a charter school over time are not the same across grade levels.

19


Calculating the mean growth rate for a CPS student for each grade level requires adding the average growth rate, duration, and the interaction terms. For example, to calculate the average reading growth rate for a 4th grade CPS student, we take the sum of the average Growth Rate Month (1.04) to the interaction term Grade4*Date ( -0.02) to Duration (0.64) to the interaction term Grade4*Duration (-0.28), which results in an estimated monthly gain of 1.38. If we consider that a school year is nine months, we can calculate the gain over the school year as 12.42 points on the reading test (1.38 * 9). However, remember that CPS students experienced an initial drop in achievement when they made the move to a charter school, and the drop in reading achievement for 4th graders was 4.60 points (Gr4*Charter). Thus, on net over the school year, 4th grade CPS students made an estimated gain of 7.82 points. To compare whether this estimated gain for 4th graders is meaningful, we also need to

Not for Distribution calculate the gain for TPS students; this provides a counterfactual to compare CPS students’

gains to what their gains would have been in a TPS. To calculate the average growth rate for a

4th grade TPS student we also need to take into account the average growth rate and interaction

terms. (We do not consider the charter school move or duration in a charter school because that obviously does not apply to TPS students.) To compute the average reading growth rate for a 4th grade TPS student, we take the sum of the average growth rate (1.04) to the interaction term Grade4*growth (-0.02). Summing these terms results in an estimated monthly gain of 1.02 for 4th grade TPS students, and over the course of an academic year this results in an estimated 9.18 point gain on the reading test (1.02 * 9). Taking the difference of the CPS and TPS 4th grade students’ reading gains, we find that TPS 4th grade students outperform CPS 4th graders by 1.36 points (9.18 – 7.82). With a standard deviation for the 4th graders reading scores of 13.33, this 1.371 difference translates into an effect size (ES) of -0.10 (-1.36/13.33 = -0.10). (The negative

20


sign means that TPS students outperform CPS students; a positive sign would mean that CPS students score higher than their TPS peers.) Table 4 provides a summary of the comparisons between CPS and TPS students based on the final models in Table 3. Table 4 provides the results by grade level for the average gains and growth of CPS students compared with their TPS counterparts as well as the standard deviation (SD) of the subject area test by grade level. Effect sizes (ES) are calculated by dividing the CPSTPS Gain by the grade level SD. The table provides results for reading over one school year, but also provides showing what the differences would be over time as CPS students experience the positive effects of attending a charter school over 1.5, 2, and 2.5 years. ___________________________________________

Not for Distribution Table 4

___________________________________________

We see that over one school year, there are few grade levels in which CPS students

experience an overall positive estimated gain. Again, this is due primarily to the fact that students experience a significant loss in their initial reading achievement when they make the move into a charter school. Most of the ES in Table 4 for one school year are negative (range from -0.270 for grade 8 to -0.103 for grade 4. CPS students in three grade levels—grades 2, 6, and 9—reveal positive ES of 0.330, 0.103, and 0.104, respectively. If we look a the effects of attending a charter school over longer periods of time, we see that it takes from 2 to 2.5 school years for CPS students to see the benefits in reading compared with their TPS counterparts. For example, when looking at the estimated ES by grade level for students that have been in charter for 2.5 years, we see that in grade 2-6, the ES sizes are

21


substantively meaningful. The ES in these grade levels range from 0.107 in grade 5 to 1.276 in grade 2. In the 7th and 8th grades, however, the ES are negative, indicating that CPS students growth is outpaced by TPS students (Grade 7 ES = -0.077 and grade 8 ES = -0.149). In grade 9, the ES of CPS students is 0.224. Table 5 presents the results for mathematics. The results generally follow a very similar pattern to the reading results, but, again, since some of the key estimates do not reach standard levels of statistical significance, we focus on reading. ___________________________________________ Table 5 ___________________________________________

Not for Distribution DISCUSSION

Our analyses reveal that in Crossroads, students who make the move to a charter school

generally experience a drop in achievement scores, whether we consider reading or mathematics scores. The growth that CPS students experience over time is mixed, and certainly depends on

what grade levels and amount of time students attend a charter school. The initial lost in reading achievement that students experience when making a move to a charter school is not compensative for within a school year by the positive effect of attending a charter school over time (duration). Over the course of more than one school year, however, CPS students tend to outperform their TPS peers in the lower grade levels. The patterns for mathematics are very consistent with reading, but many of our estimates in mathematics fail to reach strong levels of statistical significance.

22


Some of the effect sizes are noteworthy, particularly in the context of recent research that examines student lottery winners and losers. For example, in New York, Hoxby and Murarka (2007) found that in elementary grades, students who one the lottery outperformed students who lost by about 0.09 standard deviations in reading. In our analyses, our effect sizes were small or negative over the course of one year, but because CPS students benefited from attending charter schools over time, over a two to three year period. Yet, it is important to be aware that even those we relay on quasi-experimental methods to estimate the effects of charters in Crossroads, we cannot be absolutely sure that we controlled for all selection bias. Despite our findings about achievement effects of charter schools, our analyses do not allow us to understand exactly why charter school students in Crossroads are outperforming

Not for Distribution public school students over time. We can hypothesize that the innovative accountability system developed by the mayor has contributed to the success of the charter schools in Crossroads, but at this point this is an untested hypothesis. Our analyses in this paper, like many other charter school studies, fail to get inside what others refer to as the “black box” of charter schools

(Berends et al., 2008a, 2008d; Betts and Loveless, 2005; Gill et al., 2007. Zimmer et al., 2003). By estimating cross-classified growth models with samples that rely on propensity score matching, we now have a better understanding of the outcomes of charter schools in Crossroads in terms of student achievement, but we have little understanding of the mechanisms at work inside of these charter schools. In our other research, we are analyzing parent and student surveys, an in-depth analysis of the enacted curriculum, and case studies to better understand some of the processes at work inside of the Crossroads charter schools that may explain our findings.

23


The next steps in our research are to re-estimate some of the models here to test the robustness of the results. For example, we will compare the cross-classified models that uses all the information available in the matched samples to fixed-effects models that compare students to themselves over time. That is, fixed effect models compare the student-level gains of students in the traditional public school context and then in the charter school context, using each student as their own comparison (see Hanushek et al., 2002; Buddin & Zimmer, 2003: Bilfulco & Ladd, 2006; Sass, 2006). A final direction for future research suggested by this study is a closer examination of the rate of growth for students from different racial-ethnic groups. Our findings suggest that black students, for example, have initial achievement levels in mathematics and reading that are lower

Not for Distribution than non-Hispanic white students, and understanding the effects of charter schools on African American students over time would advance the findings reported here. By examining the

processes at work inside of both the public schools and the private schools in Crossroads and in other school districts, we may eventually be able to make progress towards understanding this black-white achievement gap.

24


REFERENCES Allen, Jeanne. 2001. “Education by Charter: The New Neighborhood Schools”. In An Education Agenda: Let Parents Choose Their Children's School, J. C. Goodman, and F. F. Steiger. Dallas, TX: National Center for Policy Analysis. Arsen, D., Plank, D., & Sykes, G. (1999). School choice policies in Michigan: The rules matter. East Lansing: Michigan State University. Ballou, Dale, Bettie Teasley, and Tim Zeidner. 2006. “Comparing Student Academic Achievement in Charter and Traditional Public Schools.” Paper presented at the Annual Meeting of the American Educational Research Association, San Francisco, CA. Berends, M., Goldring, E., Stein, M., & Cravens, X. (2008a). Instructional conditions in charter schools and students’ mathematic achievement gains. Paper presented at the Society for Research on Educational Effectiveness, Washington, DC. Berends, M., Springer, M. G., & Walberg, H. J. (Eds.) (2008b). Charter school outcomes. Mahweh, NJ: Lawrence Erlbaum Associates/Taylor & Francis Group.

Not for Distribution Berends, M., Stein, M., & Smithson, J. (2008c). Differences between charter and traditional public school teachers’ instructional practices and curricular alignment to the mathematics standards and state assessment in Indiana. Paper presented at the Society for Research on Educational Effectiveness, Washington, DC.

Berends, M., Watral, C., Teasley, B., & Nicotera. (2008d). Charter school effects on achievement: Where we are and where we’re going. In M. Berends, M. G. Springer, & H. J. Walberg (Eds.), Charter school outcomes (pp. 243-267). New York: Taylor & Francis. Betts, J. (2005). “The Economic Theory of School Choice.” In Getting Choice Right. J. R. Betts and T. Loveless. Washington, DC: Bookings Institution Press

Betts, J. & Hill, P et al. (2006). “Charter School Achievement Consensus Panel.” In Key Issues in Studying Charter Schools and Achievement: A Review and Suggestions for National Guidelines. Seattle, WA: National Charter School Research Project, Center on Reinventing Public Education. Betts, Julian, and Tom Loveless. 2005. Getting Choice Right. Washington, DC: Bookings Institution Press Bifulco, Robert, and Helen Ladd. 2006. “The Impacts of Charter Schools on Student Achievement: Evidence from North Carolina.” Education Finance and Policy 11: 50-90.

25


Budde, Ray. 1988. Education by Charter: Restructuring School Districts: Key to Long Term Continuing Improvement in American Education. Andover, MA: Regional Laboratory for Educational Improvement of the Northeast and Islands. Buechler, M. (1996). Charter schools: Legislation and results after four years (PR-B13). Bloomington: Indiana Education Policy Center. The Center for Education Reform (2006). National Charter School Data: New school estimates 2006-2007. Washington, D.C.:Author. Retrieved on December 7, 2006 from:http://www.edreform.com/_upload/CER_charter_numbers.pdf. Chubb, John, and Terry Moe. 1990. Politics, Markets and American Schools. Washington, DC: Brookings Institute. Cookson, Peter. 1993. “School Choice and the Creation of Community.” Paper presented at a workshop entitled, “Theory and practice in school autonomy and choice: Brining the community and the school back in”. Tel Aviv University, Israel. Cuban, Larry, and David Tyack. 1995. Tinkering Toward Utopia. Cambridge: Harvard University Press.

Not for Distribution DiMaggio, Paul, and Walter Powell. 1983. “The Iron Cage Revisited: Institutional Isomorphism and Collective Rationality in Organizational Fields.” American Sociological Review 48, 2: 147-160.

Dehejia, Rajeev, and Sadek Wahba. 2002. “Propensity Score-Matching Methods for Nonexperimental Causal Studies.” Review of Economics and Statistics 841: 151-161. Dressler, B. (2001). Charter School Leadership. Education and Urban Society, 33 (22), pp. 170-185

Elmore, Richard. 2007. School Reform from the Inside Out: Policy, Practice, and Performance. Boston, MA: Harvard Education Press. Finn, Chester, and Rebecca Gau. 1998. “New Ways of Education.” The Public Interest 130: 7992. Gill, Brian, P. Michael Timpane, Karen Ross, and Dominic Brewer. 2007. Rhetoric Versus Reality: What We Know and What We Need to Know about Vouchers and Charter Schools, 2nd Edition. Santa Monica, CA: RAND. Goldring, E., & Cravens, X. (2008). Teachers’ academic focus on learning in charter and traditional public schools. In M. Berends, M. G. Springer, & H. J. Walberg (Eds.), Charter school outcomes (pp. 39-60). New York: Taylor & Francis. Goldring, E. B., & Sullivan, A. S. (1995). Privatization: integrating private services in

26


public schools In P. Cookson & B. Schneider (Eds.), Transforming Schools (pp. 533-556). New York: Garland. Hambleton, Ronald. 1989. Principles and Selected Applications of Item Response Theory. In Educational Measurement, 3rd Edition, R. L. Linn. New York: American Council on Education, Macmillan Publishing Company. Hanushek, Eric A., John Kain, Steven Rivkin, and Gregory Branch. 2005. “Charter School Quality and Parental Decision-Making with School Choice.” Cambridge, MA: NBER Working Paper 11252. Henig, Jeffrey. 1999. “School Choice Outcomes.” In School Choice and Social Controversy, S. Sugarman and F. Kemerer. Washington, DC: Brookings Institute Press Hill, P. Angel, L., & Christensen, J. (2006). Charter School Achievement Studies. Education Finance and Policy, 1 (1), 139-150. Hoxby, C.M. and Murarka, S. (2007). “New York City’s Charter Schools Overall Report,” New York City Charter Schools Evaluation Project. Cambridge, Massachusetts.

Not for Distribution Hoxby, C.M. and Rockoff, J.E. (2004). The impact of charter schools on student achievement. Unpublished manuscript, Harvard University and Columbia Business School. Ingebo, George. 1997. Probability in the Measure of Achievement. Chicago, IL: MESA Press.

Kingsbury, G. Gage. 2003. “A Long-Term Study of the Stability of Item Parameter Estimates.” Paper presented at the annual meeting of the American Educational Research Association, Chicago, IL.

Leonardi, Richard. 1998. “Charter schools in Ohio: The rush to mend them should not end them.” Perspective on Current Issues. Dayton, OH: Buckeye Institute for Public Policy Solutions. Lord, Frederic. 1980. Applications of Item Response Theory to Practical Testing Problems. Hillsdale, N.J.: Erlbaum. Loveless, Tom. 2003. Charter schools: Brown Center Report on American Education. Washington, DC: Brookings Institution Press. Luellen, Jason, William Shadish, and M. H. Clark. 2005. “Propensity Scores: An Introduction and Experimental Test.” Evaluation Review. 296: 530-558. Manno, Bruno, Chester Finn, Louann Berlein, and Gregg Vanourek. 1998. “How Charter Schools are Different: Lessons and Implications from a National Study.” Phi Delta Kappan 797: 488.

27


McClure, L, Strick, B., Jacob-Almeida, R., & Reicher, C. (2005). The Pereuss School at UCSD: School characteristics and students’ achievement. San Diego, CA: The Center for Research on Educational Equity, Assessment and Teaching Excellence, University of California, San Diego. Mehana, M., & Reynolds, A.J. (2004). School mobility and achievement: A meta-analysis. Children and Youth Services Review, 26, 93-119. Meyer, John. W., and Brian Rowan. 1977. “Institutionalized Organizations: Formal Structure as Myth and Ceremony.” American Journal of Sociology 83: 340-363. Miron, G. & Nelson, C. (2001). Student Achievement in charter schools: What we know and why we know so little (Occasional Paper No. 41): National Center for the Study of Privatization in Education, Teachers College, Columbia University. Murphy, J.& Shiffman, C. (2002). Understanding and Assessing the Charter School Movement. New York, NY: Teachers College Press. National Center for Education Statistics. (2007). Public Elementary/Secondary School Universe Survey Data, 2002-2003, 2003-2004, 2004-2005, 2005-2006 [Data file]. Available from National Center for Education Statistics Common Core of Data Web site, http://nces.ed.gov/ccd/ccddata.asp

Not for Distribution Nicotera, A. Teasley, B., & Berends, M. (2007). An empirical investigation of the No Child Left Behind school choice policy and academic achievement in a western state. Nashville, TN: National Center on School Choice, Vanderbilt University, Peabody College.

Northwest Evaluation Association 2002. RIT Scale Norm. Portland, OR.

Northwest Evaluation Association 2003. Technical Manual. Portland, OR. Raudenbush, S and Anthony S. Bryk (2002), Hierarchical Linear Models: Applications and Data Analysis Methods, Thousand Oaks, CA: Sage. Rinehart, James, and Jackson Lee. 1991. American Education and the Dynamics of Choice. New York: Praeger. Rosenbaum, Paul, and Donald Rubin. 1983. “The Central Role of the Propensity Score in Observational Studies for Causal Effects.” Biometrika, 701, 41-55. Sass, Tim. 2006. “Charter Schools and Student Achievement in Florida.” Education Finance and Policy, 11: 91-122. Solomon, Lewis, Kern Paark, and David Garcia. 2001. “Does Charter School Attendance Improve Test Scores? The Arizona Results.” Goldwater Institute.

28


Scott, W. Richard. 1992. Organizations: Rational, Natural and Open Systems. Englewood Cliffs: Prentice Hall. Scott, W. R., & Davis, G. F. (2007). Organizations and organizing: Rational, natural and open systems perspectives. Upper Saddle River, NJ: Pearson/Prentice Hall. Stein, M., Goldring, E., & Zottola, G. (2008). Student achievement gains and parents’ perceptions of invitations for involvement in urban charter schools. Paper presented at the Annual Meeting of the American Educational Research Association, New York, NY. Zimmer, R., Richard Buddin, Derrick Chau, Brian Gill, Cassandra Guarino, Laura Hamilton, Cathy Krop, Dan McCaffrey, Melinda Sandler, and Dominic Brewer. 2003. Charter School Operations and Performance: Evidence from California. Santa Monica, CA: RAND.

Not for Distribution

29


Table 1. Results for Differences in Group Means Between Students in the Reading Sample Who Do Not Move to a Crossroads Charter School and Students Who Do Move to a Crossroads Charter School

Non-Movers N Mean 27515 199.77 Prior Math (0.12) RIT Score Grade 2 7410 181.02 (0.14) Grade 3 3430 191.20 (0.21) Grade 4 3400 198.43 (0.22) Grade 5 3336 206.12 (0.23) Grade 6 3091 208.67 (0.27) Grade 7 2276 213.62 (0.32) Grade 8 1968 217.88 (0.38) Grade 9 2594 221.67 (0.36) Male 27515 0.50 (0.00) Black 27515 0.58 (0.00) Hispanic 27515 0.09 (0.00) Other 27515 0.04 (0.00) Non-Hispanic 27515 0.29 White (0.00) *p