1
Linking international surveys of student achievement
Maciej Jakubowski Directorate of Education, OECD Faculty of Economic Sciences, University of Warsaw
Outline 2
International surveys of student achievement Why to link them to other surveys? combining data from two student surveys PISA PIRLS or TIMSS PISA 2000 PISA 2006
linking data from student and adult survey
PISA EU-SILC
summary
international surveys of student achievement 3
PISA 2000, 2003, 2006, 2009 reading, mathematics, science 15-year-olds
TIMSS 1995, 1999, 2003, 2007 mathematics and science 4th and 8th graders
PIRLS - reading CivED – civic education IALS/ALL/PIAAC – adult literacy
Why do we need to combine surveys? 4
Cross-sectional surveys measure achievement LEVEL which heavily depends on student or country background
Combining surveys which measure student achievement at different ages allows estimating student achievement GROWTH
achievement GROWTH could be related to policy variables to estimate their causal effects
empirical example: estimating causal impact of tracking
Example: institutional tracking 5
• Early tracking countries segregate students between schools with different study programmes • Tracking countries: Germany, Netherlands, Czech Republic … • Non-tracking countries: Sweden, England, Poland …
• typically one cannot assess the impact of tracking policy within a country, because of self-selection • some countries extended comprehensive schooling which allows estimating the effects of tracking, however, this is rare and could be confounded with cohort effects
6
Combining primary and secondary school surveys to obtain difference-in-differences causal estimate
DD = (post – pre | treated) – (post – pre | control)
Control group: non-tracking countries
Treatment group: tracking countries
Pre: primary school (PIRLS, TIMSS)
Post: secondary school (PISA)
DDTRACKING= [E(PISA) –E(PIRLS/TIMSS)|tracking] - [E(PISA) – E(PIRLS/TIMSS)|non-track]
ways to estimate DD with international surveys of student achievement 7
I. Country-level regression (Hanushek, Woessmann, 2006)
estimating simple regression on country means or other statistics
YPISA 0 1YPIRLS d track
assumption of linear effect needed
low number of observations
II. pooled-data regression (Jakubowski, forthcoming)
estimating regression on the pooled data from two surveys
Y 0 Xβ 1t 2 d 3dt
III. reweighting/matching estimator (Jakubowski, forthcoming)
estimating probability of being sampled for one of the surveys
using predicted probability as a propensity score to match/reweight data
PIRLS 2001 and PISA 2003 Turkey 40
Norway
8
0
20
New Zealand Scotland Iceland Hong Kong Canada
-20
France
-40
Sweden Latvia
United States England Slovak Rep. Netherlands Germany Czech Rep. Greece
-60
Hungary
Italy
Russian Federation
55
60 65 70 difference in average age (in months) tracking countries
non-tracking countries
75
Linking PIRLS/TIMSS data to PISA data 9
differently defined target populations
i.
different distributions of grade/age across surveys
different distributions of grade/age across countries
ii.
background variables differently coded
iii.
some data collected from students while other from parents
iv.
number of missings differ across countries and surveys
These problems could be at least partially resolved through interaction terms between all variables and time/treatment
similarly, by estimating propensity score on interaction terms
results 10
country-level approach is not robust difficult to account for differences across surveys and countries controlling for mean age difference alone changes results importantly estimates of the impact of tracking on student score dispersion heavily depend on the adjustments made to the sample pooled regression and matching methods give similar results smaller achievement growth in tracking countries however, this effect is similar for student with high and low SES tracking is confounded with smaller growth in eastern European countries
Poland: PISA 2000 and 2006 11
Polish school system was reformed in 1999/2000 8-years primary school was replaced by 6-years primary school and 3-years comprehensive lower secondary school In PISA 2000 15-year-olds were in different secondary school types (comprehensive, technical, vocational) in PISA 2006 all15-year-olds were in the same type of comprehensive lower secondary school Poland has the highest PISA score improvement in Europe Is that the effect of the reform? Was the reform similarly beneficial for all students? What aspects of the reform were the most helpful?
Poland: PISA 2000 and 2006 12
M. Jakubowski, H. Patrinos, E. Porta, J. Wiśniewski „The Impact of the 1999 Education Reform in Poland”, World Bank, forthcoming
decomposition of changes in distributions using OaxacaBlinder method propensity score matching to compare scores across PISA 2000 and 2006 similar background questionnaire in 2000 and 2006 however, some questions omitted, differently asked or coded
PISA 2006 matched counterfactual score
PISA 2000 factual weighted mean score
PISA 2006 factual weighted mean score
(no of obs)
(no of obs)
Kernel matching
1-1 matching
(1)
(5)
(6)
(7)
All schools
480.0 (3654)
513.8 (5233)
517.3 (5229)
514.6 (3056)
ISCED 3C schools
357.8 (983)
-
472.4 (5141)
476.0 (1090)
ISCED 3B schools
480.4 (1491)
-
503.8 (5163)
504.5 (1823)
ISCED 3A schools
543.7 (1180)
-
545.8 (5221)
553.2 (1376)
ISCED 3A and 3B schools
514.6 (2671)
-
527.5 (5233)
525.9 (2609)
achievement 9th grade 1st plausible value individual scores
13
(no of matched obs)
Propensity score matching estimates of score change for students in different upper secondary school tracks Score change:
achievement 9th grade 1st plausible value individual scores All schools ISCED 3C schools ISCED 3B schools
ISCED 3A schools ISCED 3A and 3B schools
14
PISA 2006 – PISA 2000 Kernel matching 1-1 matching (1) - (6)
(1) - (7)
37.3*** (6.4) 114.6*** (6.9) 23.4*** (5.9) 2.1 (5.8) 12.9*** (4.1)
34.6*** (3.1) 118.3*** (5.1) 24.1*** (3.4) 9.5*** (3.8) 11.3*** (2.9)
Relative score change (difference-in-differences) for students in vocational schools from PISA 2000 to PISA 2006 Relative score change
15
Kernel matching
1-1 matching
ISCED 3C versus ISCED 3A+3B
101.7
107.0
ISCED 3C versus ISCED 3A
112.5
108.8
ISCED 3C versus ISCED 3B
91.2
94.2
summary - linking student surveys 16
even different surveys have similar background questions (e.g. gender, age, grade, migrant status, number of books at home…) one can analyze them by pooling the data and controlling for background variables or reweighting with propensity score matching how to calculate standard errors accounting for: complex survey design which differs across surveys linking error in 1-1 matching
linking student and labor market surveys 17
one of the major questions in labor economics is what are the returns to skills/education? returns to education are usually calculated with countryspecific data international comparisons are based on attainment levels, not on actual level of skills and knowledge international student or adult surveys measure achievement but do not contain income data there are many income surveys which contain income information as well as other labor market indicators linking both sources of information could provide with internationally comparable rates of returns to education could be also used to analyze impact of skills on different labor market outcomes
linking student and labor market surveys 18
while in student surveys similar information is collected from students, in labor market surveys information from adults is collected we do not know a final level of education or occupation of a student we only know what is the level of education or occupation of her parents however, in labor markets information about parent education/occupation is rarely collected
PISA 2000 and EU-SILC 2005 19
PISA 2000
43 countries reading (main domain), mathematics, science these students were 20-year-olds in 2005 however, it could be argued that achievement distribution is similar for older cohorts, e.g. 25- or 30-year-olds
EU-SILC 2005 all EU countries detailed income and labor market information special „social mobility” module „What was the occupation/education of your parents when you were teenager?”
PISA 2000 and EU-SILC 2005 20
common questions mother/father
education mother/father occupation family structure number of siblings
+ gender, migrants
PISA 2000 and EU-SILC 2005 21
estimate simple statistics by categories of parents education/occupation, e.g. comparing across countries skills and income levels by parent education linking PISA data to EU-SILC using common questions as linking covariates for propensity score matching estimating
possible skill level for each individual running any kind of model on the linked dataset
PISA 2000 and EU-SILC 2005 22
potential problems missing
data differently coded/distributed answers low number of observations by country/category
how to account for linking error? should we account for survey design in PISA? use income surveys for non-European countries? use other linking methods?
summary 23
survey of student achievement could be successfully linked using information from the background questionnaires linking the same cross-sectional survey across different cycles is quite straightforward as typically background questions and survey design are similar linking different surveys could be more problematic; many adjustments are needed here and estimation of standard errors could be difficult it seems possible to link student survey with labor market studies, however, in this case labor market survey has to include questions about family/parents which is rare framework for linking PISA to EU-SILC was proposed
Thank you!
[email protected] www.wne.uw.edu.pl/mjakubowski
