Oct 6, 2005 - The technical ... secondary education), table 1 shows its different types of schools, classified by funding ... Number of students by type of school.
The Private-Public School Controversy: The Case of Chile Cristian Bellei Harvard Graduate School of Education PEPG 05-13
Preliminary draft Please do not cite without permission Prepared for the PEPG conference: "Mobilizing the Private Sector for Public Education" Co-sponsored by the World Bank Kennedy School of Government, Harvard University, October 5-6, 2005
Introduction. Marked-oriented strategies have increasingly been proposed as an effective and efficient way to increase both quality and equity in education. Academic and political discussions have attempted to predict the most probable consequences that market incentives could have on educational systems. A key issue on those analyses has been the comparative study of the public and private schools’ effectiveness in terms of students’ academic achievement. In this paper, I critically review the research about whether Chilean students attending private schools obtain greater learning outcomes than their peers studying at public schools. Chile constitutes a paradigmatic case to the public/private schools debate, and research on its experience might shed light on such a controversy. Its nationwide schoolchoice system finances both public and private subsidized schools under the same funding system, a particular type of voucher program. Compared to the small-scale of the majority of the U.S. voucher and school-choice programs, the Chilean situation is a particularly attractive case to study. Paradoxically, previous research on Chilean education has obtained very contrasting findings. The paper begins with (I) a brief description of the Chilean education; then, it reviews the research on both (II) systemic effects of school-choice and (III) private/public schools’ effect. Section (IV) analyzes some key methodological issues that account for the contrasting findings of previous research; and sections (V), (VI), (VII), (VIII) and (IX) provide empirical evidence about the consequences of the identified methodological limitations. A final section summarizes the main conclusions of the analysis, elaborates some interpretative hypothesis, and states some educational policy implications.
1
I. School choice and market oriented institutions in the Chilean Public Education. For more than two decades, the Chilean educational system has operated under an institutional design whose fundamental regulation and decision elements do not rely on national authorities, but on the combination of family preferences (which are expressed in their free choice of school) and (public and private) school competition for attracting such preferences. This system was created during the 1980s within the context of large national economic and institutional reforms, including the privatization of state companies, the restructuration of the social security and health systems, and the opening of the economy to international markets. The Chilean government of that time applied the neo-liberal canon in a paradigmatic way, trying to make education a self-regulated market. The main reforms were: i.) Creation of a single funding system for all state “subsidized” schools, be they public or private. The funding mechanism is a “voucher”, which consists of a monthly payment, to every school, of a fixed fee per each student who is enrolled and regularly attends his classes. ii.) Promotion of competition among public and private schools. Every school must compete to attract the families’ preferences, in order to guarantee its own funding. To obtain public funding, all schools must be free for families. Families do not have any restrictions to choose a school (be they public or private, near or distant from home, etc.). Private schools are not compelled to accept any applicant: They can select their students. Instead, public schools are compelled to admit any applicant. iii.) Deregulation of schools institutional management. Schools that receive state subsidy (be they public or private) must satisfy a number of minimum operational conditions, such as having basic facilities, hiring certified teachers, and fulfilling the 2
national curriculum objectives. With the exception of these “minimum standards”, schools are permitted to be managed with all the possible freedom, which means, for instance, the deregulation of teachers’ labor situation and wage, and the possibility for the school owner to profit. iv.)
Decentralization
of
public
schools
administration.
State
schools
administration was transferred from the Ministry of Education to the local governments (municipalities). The purpose of this measure was to establish the competition between public and private schools as local suppliers of education. v.) Creation of different institutional conditions for the competition among schools. The most important ones were: Creation of a decentralized Ministry of Education’s system of supervision; curriculum deregulation, in order for schools to create diverse “educational offers”; creation of a national evaluation system of students’ learning (SIMCE, Spanish acronysm for Measurement of Education Quality System), aimed at informing families about the quality of schools. Since 1990, the Chilean governments have driven a large-scale national educational reform, which attempts to combine the aforementioned market institutions with state regulation and intervention. Thus, the educational policies have been oriented to expand and deepen the market-oriented model, as well as to restrict it, through the promotion of social equity and educational quality. The main measures were: i.) Greater regulation of teacher labor market, through the creation of a “teacher labor statute”. This statute recovered a significant part of the historic teachers’ privileges, such as an ad-hoc minimum wage, rules for increasing teacher wage, wage bonus (by seniority and training, among others), and the rigidification of the firing mechanism.
3
Although all these regulations are compulsory only for municipal schools, many of them also rule subsidized private schools. ii.) Compensatory programs aimed at improving (public and private) schools that serve the poorest students, and/or those students who achieve the lowest learning outcomes. These “positive discrimination” programs consist, basically, in supplying students and teachers with teaching and learning materials, teaching training, external advisory, and enhancement of schools facilities. iii.) Full coverage policies to improve the quality of education. For instance, installation of computational laboratories, provision of school texts, teacher training, investment in school facilities, and the extension of the students’ school day. In addition, these policies included a curriculum reform much more prescriptive than that suggested by the flexible norms inherited from the 80s reform. iv.) Creation of a “price discrimination” system among subsidized private schools (in the case of municipal schools, this system only affects high schools), which allows schools to charge families for tuition, without loosing the state subvention, or, at most, reducing it to a minimum amount. v.) Strengthening of the national learning evaluation system. SIMCE results have been used to determine the target populations of compensatory programs, as well as those teachers who win a wage incentive, which operates as a merit-pay system. The technical characteristics of the test were also enhanced, while its coverage was expanded to rural schools (which had not been evaluated previously). Finally, and perhaps the most relevant issue, the knowledge and use of SIMCE results was spread among families (the results obtained by each schools are yearly published in the national press) as well as
4
among school communities (a school-level report is sent to the principal, teachers, and parents’ organizations). To understand the current structure of the Chilean school system (primary and secondary education), table 1 shows its different types of schools, classified by funding source and property. As seen, a few more than half of the students (53%) study in public schools (“municipal”), most of which are free. The remainder studies in private schools (47%). In most cases (34.2% of the national enrollment) private schools are co-financed by public funds and tuition charged to the families. Totally free private schools educate a marginal fraction of students (3.8%), while private schools totally paid by families represent approximately a tenth of Chilean preschool system (9%). Summing up, nearly half of the Chilean students pay for their education. In this paper, I will only distinguish between public, private subsidized (voucher), and private non-subsidized schools. Table 1. Chilean schools by funding source and property (percentage of total national enrollment). Funding source
Public Ownership and administration
Public (free)
Mixed (co-pay)
Private (tuition)
Municipal
Municipal with copay (some High Schools) (5.3%)
-----------
Subsidized Private with co-pay
Non-subsidized Private
(34.2%)
(9%)
39.5%
9%
(47.7%) Subsidized Private Private
(3.8%) 51.5%
53%
47%
Source: author elaboration, based on Ministry of Education 2002.
Additionally, graph 1 shows the evolution of enrollment between 1980 and 2000. Overall, the total enrollment remained quite stable during the first ten years of the new system. However, this stability did not imply that each type of school kept a steady level of enrollment: Private subsidized schools enrollment increased rapidly, especially during 5
the first 5 years (1981-1986), period in which it more than doubled. Simultaneously, municipal schools saw a systematic fall of their enrollment from 1981 and 1991. This “transfer” from public to private schools involved more than a half million of students (almost a fifth of the entire school system). The period that began in 1991 showed a different trend: national enrollment increased rapidly and systematically; thus, in 2000 there were 600,000 more students than in 1991. For the first time since the creation of the subvention system, municipal education stopped losing students, starting a slow expansion: In a decade, it increased in approximately 200,000 students. Private subsidized schools’ enrollment continued its expansionist trend, although less rapidly than in the 1980’s decade: it increased by 300,000 students during this period. The enrollment of non-subsidized private schools slowly increased throughout the entire period, remaining as a minor part of the Chilean school population, however. Finally, the testing system (SIMCE) has pointed out a systematic pattern: on average, private non-subsidized schools’ students score higher than private subsidized students, while private subsidized students’ score higher than public schools’ students. Since the early nineties, the raw achievement gap between non-subsidized private schools and public schools has been approximately 1.2 to 1.8 S.D. In turn, the raw test-score gap between subsidized private schools and public schools has been about 0.3 to 0.4 S.D. Whether or not this raw gap is produced by genuinely greater private school effectiveness has been one of the most controversial academic and political questions in the Chilean educational debate in the last fifteen years. Graph 1. Evolution of the Chilean primary and secondary education enrollment by type of schools.
6
Number of students by type of school. 1981 - 2000 3,750,000 3,500,000 3,250,000 3,000,000
Total Municipal
2,750,000
Private subsidized Private non-subsidized
2,500,000 2,250,000 2,000,000 1,750,000 1,500,000 1,250,000 1,000,000 750,000 500,000 250,000
2000
1999
1998
1997
1996
1995
1994
1993
1992
1991
1990
1989
1988
1987
1986
1985
1984
1983
1982
1981
0
Source : Ministry of Education 2002.
II. The research about systemic effects of school choice in Chile. There are two competing theories (both are simultaneously academic and policy theories) about the systemic effects of the Chilean voucher system. One states that subsidized private schools can help to improve public schools through a competition effect, predicting a global improvement of the Chilean education. The other theory proposes that the (expected) positive productivity effect on private and public schools may be canceled out by the (unexpected) negative effect of sorting (private schools “skim” the best public students) on public schools, predicting a kind of “zero-sum game”, with no systemic improvement. In spite of their contrasting points of view, both theories agree that, in order to evaluate the Chilean voucher system, it is necessary to assess its impact on both public and private schools. Unfortunately, there is very little research on this issue.
7
Gauri (1998), in a 1993-1994 survey based on a random sample of 726 households of the Santiago metropolitan area, studied the school choice process. He created a multinomial logit model to predict what kind of students attend the different types of schools, especially high performing publicly funded schools (both private and public). He found that the probability of attending a school situated in the top third of students’ learning outcomes significantly increased with the parents’ education, family income, and other family characteristics associated with higher socioeconomic status. He also found that the probability of studying in a high performing school significantly increased when the student was required to take a cognitive and/or academic test as a requirement to be admitted to the school. In other words, he found that top-performing schools systematically applied academic admission policies to select the most talented students. Gauri concluded that choice policies have increased both the social and the academic stratification of the Chilean educational system. Although this study provides valuable evidence about the current testing policies applied by schools with higher test scores, it is not clear the extent to which these processes are associated with the voucher system and the extent to which they have increased the stratification of the Chilean education. Hseih and Urquiola (2003) attempted to evaluate whether the introduction of school choice in Chile increased the educational and socioeconomic differences between private and public schools. The students’ outcomes they analyzed were 4th grade mathematics and language tests scores, repetition rates, and years of schooling among 1015 year olds, between 1982 and 1988, at schools and commune level. They controlled for several socioeconomic schools and commune factors. They found that communes with higher proportion of private enrollment tended to have higher public/private test-score 8
gap and repetition rate gap, as well as higher students’ SES public/private difference at commune level. The authors also found that commune enrollment in private schools rate was negatively associated with public test scores, after controlling for several commune and school factors. Hseih and Urquiola interpreted these findings as an evidence of a negative effect of private schools on public schools. Finally, at commune level, neither the level of 1990 private enrollment nor the 1982-1990 increase in private enrollment were associated with students’ outcomes (test scores, repetition rate, and schooling among 10-15 year olds). In other words, they did not find evidence that private subsidized schools yielded positive systemic effects. I think this study shows strong evidence of the association of both private enrollment and public/private gap, but fails to demonstrate a causal link between them. Finally, Gallegos (2002) also attempted to estimate the impact of market competition on public and private subsidized schools. The author used 4th grade (1994 and 1996) and 8th grade (1995 and 1997) test scores at school level (school mean of Language and Mathematics) as the outcome variables, and controlled for the schools’ SES composition. Gallegos defined each commune as a different school-market, and controlled for commune variables (level of urbanization, size). The study found that the level of market competition (as measured by the proportion of private enrollment at commune level) positively affected the school performance (statistically significant estimates for 1994, 1995, and 1997), and that this “competition effect” was stronger for private schools. The key limitation of this approach is that the level of private enrollment is not an exogenous variable to students’ performance; on the contrary, there is strong evidence that private schools tend to serve students, families, and geographical areas with characteristics positively associated with students’ learning outcomes. In a further study, 9
Gallegos (2004) attempted to overcome this limitation. He used “priest per capita” as an instrumental variable to identify exogenous variation in private subsidized enrollment at commune level. The paper analyzed 4th grade (2002) test scores, and controlled for students’ mother education and school resources. The author estimated that an increase in private enrollment by one S.D. (which is about 20 percentage points) was associated with a 0.2 S.D. increase in students’ test scores. Catholic schools account for about 10% of the Chilean enrollment (only a third of the total private subsidized enrollment), and Catholic schools are precisely those private schools that existed in Chile prior to the introduction of the market oriented model. Consequently, it is not clear that priest per capita might be a valid instrument for private schools in Chile. Overall, the available evidence is not sufficient to evaluate the abovementioned theories about the systemic effects of school choice in Chile. A more productive approach should include analyses of longitudinal educational data, analyses of institutional and educational policy contexts, and a deeper understanding of the parents’ choice and schools’ selection processes. Undoubtedly, this is a very difficult task. Instead, most of the research has been focused on test-scores comparisons between private and public schools. The rest of the paper analyses this line of inquiry. III. Private versus Public schools’ effectiveness in Chile. In general terms, the research on the comparison between public and private schools’ effectiveness in Chile has evolved following three stages. Rodriguez (1988), Aedo and Larrañaga (1994), Aedo (1997), are the best examples of the first phase. All of them studied small (not representative) samples of schools, and analyzed exclusively school-level data (obtained during the eighties or early nineties), and focused on urban primary schools. These three studies concluded that -after 10
controlling for school characteristics- private schools scored higher than public schools, and that this difference was statistically significant. Unfortunately, it is not possible to generalize these findings to the Chilean school population. Because of their lack of representativeness, I do not include the research of this phase in my analysis. All the available studies of the second and third phases are summarized in table 2. The first six listed studies also analyzed exclusively school-level data, but they studied very large, nationally representative samples; in fact, most of them used the entire, nation-level database of schools’ test scores1. This research is also focused on primary education (mainly 4th grade), and all of them applied Ordinary Least Squares estimates. These six studies constitute the second phase of this kind of research. Finally, the last four studies included in table 2 are part of a third, more sophisticated stage. These four studies analyzed student-level test scores as the outcome variable, and also included student-level predictors. They used the entire nation-level database, and included both primary and secondary education. As shown, studies of the third phase applied more sophisticated research methods: in addition to OLS estimated, they applied Hierarchical Linear Models, and probabilistic models of choice. Although the first three studies included in table 2 analyzed more than one year of students’ test scores, none of them is a longitudinal analysis (these studies are only a series of cross-sectional estimates). The studies on Chilean education have analyzed Mathematics and/or Language test scores as the outcome variable (two of them used the Mathematics-Language average as the outcome measure). As reported in the last column of table 2, studies using schoollevel data explain a greater proportion of the test-scores variation (about 40% to 60%) 1
Vegas (2002) is an exception: her sample is only representative of the Santiago Metropolitan Area. I included this study because it is very recent (analyzed 1999 test-scores), and used a unique database on teachers’ characteristics.
11
than studies using student-level information (about 10% to 20%). As known, this is a consequence of the loss of variation caused by the aggregation of individual test-scores at school level. Most studies compare public schools with two categories of private schools: voucher and non-subsidized schools, although some of them also distinguish between Catholic voucher schools and non-religious voucher schools. As shown in table 2, there are noticeable differences in the estimates of the private/public test score gap: while some authors have found private school advantage (0.05 S.D. to 0.27 S.D.), others have found public school advantage (0.06 S.D. to 0.26 S.D.), and some others have found no statistically significant difference between them. Moreover, these differences can be found not only between studies, but also within studies. An additional puzzling fact is that several studies have differed in their findings even if they analyzed the same database. In the next section, I will provide some hypothesis and new evidence to explain those contrasting conclusions.
12
Table 2. Summary of the studies about private/public test score gap in Chilean schools. Analyzed data, controlled variables and main results. Study Bravo et al., 1999
Level/type of analysis School (OLS)
Control variables School SES (school average of parents’ education and family educational spending) / geographical location [ This paper estimated 12 models for every grade and subject matter for years 1992, 1994, and 1996. In the last column of this table I only reported equivalent models that are available for all years (all of them only controlled for the mentioned variables). The additional non-reported models introduced more family characteristics (opinion about the school), and school variables (for example student/teacher ratio, number of teachers, school SES Index). In general terms, when these additional controls are included the difference between private subsidized and public schools is not statistically significant neither for language nor for mathematics, neither for 4th nor for 8th grade. In turn, the difference between private non-subsidized and public schools although reduced, remains statistically significant for both Mathematics and Language in 4th grade, but it is not statistically significant for 8th grade in both subject matters. ]
Gallegos, 2002
School (OLS)
School SES level (1-4) / % private subsidized enrollment at commune level
13
Year/ Grade 1982/4th 1983/4th 1984/4th 1988/4th 1990/4th 1992/4th 1994/4th 1996/4th
1982/8th 1983/8th 1984/8th 1989/8th 1991/8th 1993/8th 1995/8th 1997/8th
1994/4th 1996/4th 1995/8th 1997/8th
Private/Public difference in S.D. by Subject matter* Language: Subsidized: +* 1982, ‘83, ‘84, ‘88, ‘92, ’94, ‘96 + 1990 Non-subsidized: +* ‘82, ‘84, ‘88, ‘92, ’94, ‘96 + 1983, 1990 Mathematics: Subsidized: +* 1982, ‘83, ‘84, ‘88, ‘94 + 1990, ‘92, ‘96 Non-subsidized: +* ‘82, ‘83, ‘84, ‘88, ’94, ‘96 + 1990, ‘92 Language: Subsidized: +* ‘82, ‘83, ‘84, ’89, ‘93, ‘95, ’97 + 1991 Non-subsidized: +* ‘82, ‘83, ‘84,’91, ‘95, ’97 + ’89, ‘93 Mathematics: Subsidized: +* ‘82, ‘83, ‘84, ‘93, ‘95, ’97 + 1989 - 1991 Non-subsidized: +* ‘82, ’83, ‘84, ‘97 + ‘91, ‘95 - ’89, ‘93 Lang/Math average: 0.08* Subsidized 0.05* Subsidized 0.14* Subsidized 0.11* Subsidized
R2 0.420.64
0.420.57
0.420.59
0.430.54
≈0.26 -0.31
Carnoy and McEwan, 2000
School (OLS)
Average schooling of parents (levels 1-5) / % of mothers with less/more than 8 years of schooling / School SES Index (%) / geographical location (rural/urban)/ size of the city (levels 1-5)
1990/4th 1992/4th 1994/4th 1996/4th 1990/4th
1992/4th 1994/4th 1996/4th Mizala and Romaguera, 1999
Sapelli, 2003 Vegas, 20022
School (OLS)
School (OLS) School2 (OLS)
School SES level (1-4) / School SES Index (%) School SES level (1-4) / School SES Index (%) / geographical location / male-female school / student/teacher ratio / school size / teacher experience / preschool level in the school Mothers’ education (school mean and school S.D.) / geographical location School SES Index (%) / Teacher characteristics (education, years experience, high school grade, teachers’ salary) / School management (decentralization of decision making, teacher absenteeism, teachers autonomy, teachers’ satisfaction)
14
1996/4th
1999/4th 1999/4th
Language: -0.05* Non-religious Sub. 0.31* Catholic Subsidized 0.63* Non-Subsidized -0.10* Non-religious Sub. 0.23* Catholic Subsidized 0.61* Non-Subsidized -0.07* Non-religious Sub. 0.25* Catholic Subsidized 0.66* Non-Subsidized -0.07* Non-religious Sub. 0.27* Catholic Subsidized 0.38* Non-Subsidized Mathematics: -0.04* Non-religious Sub. 0.28* Catholic Subsidized 0.67* Non-Subsidized -0.10* Non-religious Sub. 0.19* Catholic Subsidized 0.58* Non-Subsidized -0.08* Non-religious Sub. 0.17* Catholic Subsidized 0.65* Non-Subsidized -0.08* Non-religious Sub. 0.24* Catholic Subsidized 0.40* Non-Subsidized Lang/Math average: -0.02 Subsidized 0.18* Non-Subsidized 0.03 Subsidized 0.45* Non-subsidized Mathematics: 0.79* Subsidized Mathematics: 0.01 Non-religious Sub. 0.30* Catholic Subsidized 1.04* Non-Subsidized
0.60 0.63 0.64 0.54
0.55 0.55 0.56
0.47
0.42 0.17 0.69
McEwan, 2001
Student (OLS)
(OLS+ model of choice)
Mizala and Romaguera, 2003
Student (OLS) (HML between school)
Sapelli and Vial, 2002
Mizala et al., 2004
Student (OLS)
Student (HLM between school)
Gender / indigenous mother / mother and father education / family income / books at home / geographical location / % indigenous student in the classroom / mean of classroom mother education / mean of classroom father education / mean of classroom family income
1997/8th
Gender / indigenous mother / mother and father education / family income / books at home / geographical location / % indigenous student in the classroom / mean of classroom mother education / mean of classroom father education / mean of classroom family income / selectivity variable (multinomial logit model of choice)
School SES (weighted mean of mother and father education, and family income) / School SES Index (%) / gender / school curriculum / length of school day / school size / teacher experience / student/teacher ratio. School SES (weighted mean of mother and father education, and family income) / male-female school / school size / teacher experience / student/teacher ratio / students with similar achievement (%) School SES*type of school interaction / students with similar achievement (%)*type of school interaction / male-female school / school size / teacher experience / student/teacher ratio Family income / mother education / father education / indigenous family Family income / mother education / father education / indigenous family / self-selection model Family income / mother education / father education / indigenous family / self-selection model School SES (weighted mean of mother and father education, and family income)*School type interaction / geographical location / male-female school / length of school day / school size / teacher experience / student/teacher ratio / students with similar achievement (%)
Source: author elaboration. * Key: * = p < .05; + = positive private school effect; - = negative private school effect. 1 Interaction effect was added to the main effect, by using the total population mean of the respective variable. 2 Sample: 171 Santiago schools.
15
1998/10th
1998/10th
1999/4th
Language: -0.07* Non-religious Sub. 0.09* Catholic Subsidized 0.46* Non-Subsidized Mathematics: -0.12* Non-religious Sub. 0.12* Catholic Subsidized 0.47* Non-Subsidized Language: -0.12 Non-religious Sub. -0.06 Catholic Subsidized 0.12 Non-Subsidized Mathematics: -0.26 Non-religious Sub. -0.11 Catholic Subsidized 0.03 Non-Subsidized Language: 0.27* Subsidized 0.35* Non-Subsidized Language (school mean): 0.27* Subsidized -0.06 Non-Subsidized Language (school mean) 1: 0.28* Subsidized 0.36* Non-Subsidized Language (OLS): 0.19* Subsidized -0.05 Subsidized (ATE) 0.14* Subsidized (TTE) Mathematics (school mean)1: 0.04* Subsidized 0.39* Non-subsidized
0.080.19 0.080.17 0.090.19 0.080.17
0.26
0.14
In spite of those divergences among the summarized studies, it is possible to draw some general trends about the private/public test score gap. Table 3 synthesizes the main conclusion of every study included in table1. To elaborate this synthesis, I have taken into account the most complete model included in the research, the most precise estimate, or the most general findings according to the author (note: table 2 does not include all estimates of every study, but the most comparable or consistent). As shown, five out of the ten studies concluded that private subsidized schools score higher than public schools; four studies concluded that there is no statistically significant difference between both kinds of schools; and one study concluded that private subsidized schools score lower than public schools. In addition, two out of the three studies that made that distinction estimated that Catholic subsidized schools score higher than public schools, while one study found no statistically significant difference between them. Finally, six out of the seven studies that included comparisons with private non-subsidized schools found that this type of schools score higher than public schools, while one of them found no statistically significant difference between public and private non-subsidized schools. Table 3. Main conclusion about private school effect on test-scores in ten studies about Chilean schools. Study Bravo et al., 1999 Gallegos, 2002 Carnoy and McEwan, 2000 Mizala and Romaguera, 1999 Sapelli, 2003 Vegas, 2002 McEwan, 2001 Mizala and Romaguera, 2003 Sapelli and Vial, 2002 Mizala et al., 2004
Private subsidized
= + = + = = + + +
Catholic subsidized
Private nonsubsidized
+ +
+ +
+ =
+ = + +
Source: author elaboration.
Key: + : positive effect; - : negative effect; = : no statistically significant difference. Reference category: public schools.
16
As pointed out, the size of the estimated differences varies markedly; nevertheless, it is also possible to identify some general trends to this respect. The estimated effect size of the private subsidized schools on students’ test-scores seems to be extremely small in 4th grade (about 0.05 S.D.). It is important to note that, when studies have found public schools’ advantage, the estimated effect size has had similar magnitude. The very large sample size used in those studies is a key factor to explain why so small test-score differences are found statistically significant. Unfortunately, the available evidence on 8th and 10th grades is not sufficient to conclude about a general pattern on these grades. On the other hand, the estimated effect size of private non-subsidized schools on students’ achievement seems to be larger than that of private subsidized schools (about 0.4 S.D.), although the parameter estimates are, in this case, less accurate. Whether or not these general findings are valid conclusions depends on the methodological strength of the available research. In the next section, I will analyze some key limitations of the aforesaid studies. IV. Methodological issues in the research on Chilean public/private schools comparisons. As described, Chilean students are not randomly assigned to their schools. As a consequence, there are at least three factors that complicate the comparisons between public and private schools’ performance in Chile. Firstly, the supply of private schools is not evenly distributed among geographical areas or among social classes: private schools tend to be situated in urban areas, as well as to serve middle and middle-high (voucher schools) and high (nonsubsidized schools) social-class students. Some studies have attempted to use instrumental variables to overcome this limitation, but their results are arguable, because of the difficulty to find a valid instrument for the supply of private schools (see for example Gallegos 2004).
17
Secondly, each type of schools operates with different levels of resources: non-subsidized schools charge families very high tuitions; most voucher schools charge families with variable tuitions; and, although most public schools do not charge tuition, they receive other kinds of public transfers. Therefore, it is difficult to control for “school resources”. This is a key challenge for those studies focused on the cost-effectiveness of private and public schools. Nevertheless, for this analysis, oriented to determine whether there is a private school advantage, not why this would be the case, this is not a relevant methodological problem. Finally –and most importantly-, the selection processes by which Chilean students are enrolled in schools is highly complex and there is little information about them. There are no formal restrictions for parents to choose, and so, they can select a school based on their preferences and/or their capacity to pay the tuition. However, private schools may select their students. The process by which Chilean schools select the best students has not been studied indepth, even though its existence has been reliably documented. Gauri (1988) found that, in Santiago, 82% of non-subsidized private school students, 37% of voucher school students, and 18% municipal school students had been compelled to take a selection test in order to be admitted to their respective schools. As stated, Gauri also found that the probability for a given student to study in a publicly funded school (be they private or public) situated in the upper third of the student outcome distribution (as measured by SIMCE 1992) significantly increased when they took an admission test. CIDE–La Tercera, in 2002, surveyed the principals of the schools that obtained the highest SIMCE-2000 scores at national level; they found that 88% of nonsubsidized private schools, 66% of voucher schools and 22% of public schools systematically used those compulsory admission tests. Those tests -focused on basic language, reasoning, psychomotor and social skills- were applied even to pre-school applicants. In a 2003 nationwide
18
full-coverage survey to 10th graders’ parents2, 85% of the private non-subsidized, 73% of the voucher, and 59% of the public schools’ respondents stated that their child was selected by the school through an admission process that included some kind of examination or minimum academic requirement. Finally, student selection is not limited to the school admission, but it is a continuing process, which can operate at any time of the students’ schooling. In fact, many private schools expel those students who have low academic achievement or behavioral problems. In these cases, the students’ selection is not based on predicted but on demonstrated student capacities. Consequently, selection bias affects the estimates of the public/private test-scores gap. This has been the most difficult challenge that researchers on the Chilean case have faced. Selection bias is a crucial problem because unobserved students’ characteristics related to students’ performance are highly correlated to the probability of attending a private/public school. Thus, cognitive skills, motivation, and discipline are probably the most relevant unobserved students’ characteristics affecting the private/public school effects estimates. This implies an additional methodological problem: controlling for family characteristics is not sufficient to control for selection bias. Unfortunately, there is no information about students’ initial characteristics or previous test scores. Researchers have tried to control this bias by introducing different student-level (e.g. parents’ education), and school-level controls (e.g. students’ SES), as well as by applying different methodological tools (e.g. instrumental variables, statistical models for selection). The findings have been highly sensitive to the type of approach used. Additionally, it is relevant to take into account that selection bias is not only an individual issue, but also a collective factor affecting students’ performance. In fact, the literature 2
Author’s calculation based on SIMCE 2003 data base.
19
about the Chilean case has increasingly recognized the potential role of peer-effects on students’ learning outcomes. Peer-effect is always present in school settings, but it is possible that, in highly segregated environments, peer-effect might play an even more influential role on students’ performance. Researchers also vary in the way they measure and model peer-effects (compare, for example, Sapelli 2003 with McEwan 2001). The appropriate level of data aggregation has also been a source of divergence among authors; while some of them apply commune-level analysis, others prefer school-, classroom-, and student-level analysis. The recent availability of student-level data permitted to create more complex methodological models (phase two versus phase three studies). As known, models estimated by using student-level outcomes are more rigorous, but less accurate in predicting testscores. Data aggregation is also an issue linked to the control variables. While some authors think that controlling for student-level variables (if available) suffices, others think that school compositional effects are relevant as well, so that they should be simultaneously included. Finally, little attention has been given to the multilevel nature of the educational data (only Mizala and Romaguera 2003, and Mizala et al. 2004 have applied multilevel analysis). To this respect, there is little information about between-schools and within-schools test scores variation, and how this issue is linked to the different types of schools and the segregation patterns of students’ distribution in Chile. The control variables introduced into the analysis is another source of disagreement. As shown in table 1, studies differ considerably in both the quantity and the quality of the control variables they use. In addition, researchers assess the same phenomenon in very different ways and scales. For example, students’ characteristics at school level has been measured as the percentage of students whose parents have attained primary education, the school mean of
20
parents’ education, or the standard deviation of parents’ education. Finally, structural variables (geographical location or family income) have frequently been present in the studies, but cultural variables (like peer-effects or books at home) have been almost absent. Lastly, little attention has been given to the exploration of possible differential effectiveness between private and public schools. For example, Mizala and Romaguera (2004) found an interaction effect between students SES and the type of school they attend. Other researchers have suggested that private school effect differs according to the level of urbanity of the city. Unfortunately, there is little research on the interaction between type of school and grade level, student’s initial ability, subject matters, etc. In the next five sections, I will empirically demonstrate the consequences of some of the mentioned methodological issues. Based on this analysis, I will conclude that even the answer to the most basic question on this topic (what are the most effective types of schools) is extremely sensitive to methodological decisions, and –consequently- the current literature does not provide a defendable conclusion for the public/private debate on the Chilean case. The data I analyzed was SIMCE 2003, 10th grade evaluation. This database contains 243,151 students, who are the 95% of the total 2003 Chilean student population of the corresponding grade. The data includes 2,117 high schools, which are the 96% of the Chilean high schools. Individual test scores on Mathematics and Language were analyzed, and several student-level and school level control variables were included. Table 4 provides a description of every variable used in the analysis. In order to illustrate some of the aforementioned methodological disagreements and to analyze whether these choices significantly affect the estimated private school effect, I conducted several regression analyses.
21
Table 4. Descriptive statistics and variable definitions. Variable Student-level variables Mathematics Language Mother’s education Father education Family income Books Gender Repetition Selection School-level variables Mean mothers’ education S.D. mothers’ education Mean parents’ education Mean books at home Selected students School SES level
Quintile income LOG income S.D. families’ income Type of school
Definition Standardized IRT test score (S.D.=50; mean=250) Standardized IRT test score (S.D.=50; mean=250) Years of education of the student’s mother Years of education of student’s father Natural LOG of student’s family income Number of books at student’s home, scale ranging from 0 (no books) to 5 (more than 200 books) Dummy variable for student’s gender (omitted category: woman) Dummy variable indicating whether the student has repeated a grade Dummy variable indicating whether the student was selected by the school through an admission process (e.g. tests, grades requirements) School average of years of education of students’ mothers School standard deviation of years of education of students’ mothers School average of years of education of both students’ parents School average of the individual variable “books at home” Percentage (divided by 100) of students who sere selected by the school through an admission process Series of 5 dummy variables that classifies schools in Low/MiddleLow/Middle/Middle-High/High students’ socioeconomic status (the classification is based on mother’s and father’s years of education, family income, and proportion of at-risk students in the school) Quintile classification of schools based on the school average of family income Natural LOG of the school average of student’s family income School standard deviation of the students’ family income Series of three dummy variables indicating whether the school is public (omitted category), private voucher, or private non-subsidized
V. How to control for parents’ education? There is a wide academic agreement on the relevance of parents’ education as a predictor of students’ test scores. Thus, to the extent that public and private schools serve students with markedly different levels of parents’ education, research focused on the private/public test score gap needs to introduce some kind of control for this variable. This consensus is also present in the research on the Chilean private/public school gap: all studies synthesized in table 1 included control information on parents’ education. Nevertheless, those studies vary noticeably in the way
22
this control is introduced into the regression models. In first place, some studies use a specific measure of parents’ education (e.g. Sapelli and Vial 2002), while others use indexes and other kinds of composite variables as general measures of school socio-economic status (e.g. Mizala and Romaguera 1999). Below, I will explore some of the consequences of using these indexes. In second place, studies also differ in the level of aggregation of parents’ education variables: while some of them measure parents’ education at student level (e.g. McEwan 2001), others introduce school-level measures into this aspect (e.g. Carnoy and McEwan 2000). Finally, studies diverge in the specific parents’ education variables that are introduced. In order to show how those differences may affect the estimates of the private/public test score gap, table 5 shows several regression models with different options, all of them present in the research on the Chilean private/public gap. The six models were estimated by using the same student population. Model 1 is a baseline model: private voucher schools score about 21 test-scores higher than public schools (0.42 S.D.), and private non-subsidized schools score about 88 points higher than public schools (1.76 S.D.). Models 2 and 3 use student-level parents’ education variables (mother’s and father’s years of schooling). As expected, controlling for mother’s education –model 2- reduce the private/public test score gap (to 0.27 S.D. and 1.17 S.D. respectively), nevertheless this gap remains statistically significant. Moreover, when father’s education is added –model 3- the private school advantage is reduced only slightly and remains statistically significant for both kinds of private schools. Models 4, 5, and 6 also control for parents’ education, but measured at school level. Model 4 estimates the private/public gap by controlling for the school average of mothers’ years of education. The results are very different from those obtained by models 2 and 3: students in
23
both private voucher schools and private non-subsidized schools obtain lower test-scores than students in public schools do. As shown, although small (0.04 S.D. in both cases), the size of the gap is statistically significant. More recently, some researchers have introduced the heterogeneity of the student population as a different control variable for parents’ education. Model 5 uses the school standard deviation of the mothers’ years of education as the only control variable: interestingly, this variable per se has almost no effect on the private/public school gap. In fact, neither the estimates nor the R2 of model 1 are different from model 5, and this control variable has a very little effect on students’ test scores (statistically significant only at 10% level). Table 5. How to control for parents’ education? Regression models that describe the relationship between school type and students’ Mathematics achievement, controlling for different parents’ education variables. Omitted category: Public schools. Dependent variable: 10th grade students’ Mathematics test score, SIMCE 2003 MODEL 1
MODEL 2
MODEL 3
MODEL 4
MODEL 5
MODEL 6
Private voucher
20.79***
13.36***
11.67***
-2.03***
20.84***
-2.34***
Private non-subsidized
87.91***
58.66***
50.33***
-1.83**
88.16***
-3.20***
4.53***
3.03***
Mother’s education Father’s education
2.50***
School mean mothers’ education
13.91***
School SD mothers’ education Constant R2 N (students)
13.94*** 0.68~
-3.21***
230.9***
189.6***
179.8***
104.4***
228.8***
114.4***
0.14
0.21
0.23
0.30
0.14
0.30
180, 388
180, 388
180, 388
180, 388
180, 388
180, 388
Source: author elaboration.
Key: ~p
