NSRCG - Mathematica Policy Research

Analyzing the Effects of a Data Collection Mode Change in the 2003 National Survey of Recent College Graduates (NSRCG) 2009 American Association for Public Opinion Research Annual Conference Hollywood, FL May 15, 2009

Donsig Jang and Zhanyun Zhao Mathematica Policy Research, Inc. Kelly H. Kang National Science Foundation (NSF)

Disclaimer The opinions and assertions in this presentation are those of the authors and do not reflect the views or policies of the National Science Foundation (NSF).

Outline      

Overview of NSRCG Data collection mode change in 2003

Issues related to mode change/differences Random experiment Results

Summary

NSRCG  Sponsored by NSF  Conducted every two or three years since 1974  Researchers collect information about education, employment, and demographics from recent graduates with bachelor’s or master’s degrees in science, engineering, or health fields

 Two-stage sample design: school sample at the first stage and graduate sample at the second stage

 For more information on NSRCG: www.nsf.gov/statistics/srvyrecentgrads

Data Collection Mode Change in 2003  Prior to 2003, the primary means of data collection was interviewer-administered computer-assisted telephone interviewing (CATI) with a small mail followup.

 In 2003, the data collection protocol changed to multimode administration (mail and Web with CATI followup). – Self-administration (mail and Web) became the primary mode of data collection.

Why Multimode Collection?  Declining response rates in telephone surveys  Seven percent of the total completed surveys in 2001 NSRCG were via mail (not CATI)

 Characteristics of NSRCG sample persons



– Frequent Internet users – Highly mobile Providing additional response options helps increase the likelihood of participation

 Since Web completes are less costly, offering Web mode to everyone increases the potential savings

Issues Associated with Data Collection Protocol Change  Time series analysis  Data quality  Mode effect

Experimental Samples  Study to assess the effects of the data collection protocol changes in the 2003 NSRCG estimates

 Randomly split the entire sample of 18,000 into two groups: – 16,000 had multimode survey administration (main mode of data collection) – 2,000 had CATI-only survey administration (same as the previous mode)

Outcome Measures 

Unit response rates – A measure of the quality of the survey and the potential nonresponse bias



Item nonresponse rates – To evaluate the quality of the survey data



Survey estimates

Unit Response Rates (Unweighted)

Response Rate for 2003 NSRCG %

Response Rates by Mode and Gender %

Response Rates by Mode and Race/Ethnicity %

Response Rates by Mode and Field of Major %

Response Rates by Mode and Degree Level %

Item Nonresponse Rates

Item Nonresponse Rates for Race and Salary Data Items by Mode Data Item

CATI Only

Multimode Overall

CATI

Mail

Web

Race*

11.2%

9.4%

12.5%

5.6%

8.7%

Salary

4.2%

4.0%

5.2%

2.3%

4.3%

* Indicates the statistical significance at =0.1

Survey Estimates  Treated the CATI-only and multimode groups as two independent samples and reweighted each group so that they represent the same population

 Constructed replicate weights for each sample group to calculate standard errors

Survey Estimates (cont’d.)  Compared survey estimates in the following five categories: – Degree levels and field of major – Demographic characteristics such as age, gender, and race/ethnicity – Academic outcomes such as undergraduate GPA, major field of highest degree major, and whether an associate’s degree was received – Financial information such as type/amount of loans and other financial support – Employment outcomes such as employment field, labor force status, and primary work activity

Percentages of Graduates in Each Bachelor’s Degree Field of Major (Before and After Nonresponse Adjustments) Before Nonresponse Adjustments Field of Major

CATI

Multi

Sciences

77.9

76.8

Life and related sciences

15.7

Computer and information sciences

CATI

Multi

1.4%

72.6

72.7

-0.1%

17.7

-12.7%

16.1

16.1

0.0%

12.2

9.6

21.3%

8.6

9.1

-5.8%

5.9

4.7

20.3%

3.9

3.8

2.6%

Psychology

14.7

16.5

-12.2%

16.4

16.4

0.0%

Social and related sciences

26.7

25.3

5.2%

24.8

24.6

0.8%

Engineering

12.4

13.2

-6.5%

11.8

12

-1.7%

Physical and related sciences

RD

After Nonresponse Adjustments RD

RD stands for relative difference between multimode group estimate and CATI group estimate.

Percentages of Graduates by Labor Force Status and Degree Level (Before and After Nonresponse Adjustments) Before Nonresponse Adjustments Labor Force Status Bachelor’s In labor force** In labor force and employed In labor force and unemployed Not in labor force** Full-time employed, including all jobs Part-time employed, including all jobs Master’s In labor force** In labor force and employed Not in labor force** Full-time employed, including all jobs Part-time employed, including all jobs

CATI

Multi

RD

After Nonresponse Adjustments CATI Multi

RD

88.8 82.8 6.0 11.2 68.8 14.0

86.8 2.3% 82.0 0.9% 4.7 21.7% 13.2 -18.3% 68.7 0.1% 13.3 4.8%

89.2 83.4 5.8 10.8 69.4 14.0

86.6 82.0 4.6 13.4 68.8 13.2

2.9% 1.7% 20.7% -24.1% 0.9% 5.7%

87.2 85.7 12.8 75.5 10.2

92.0 -5.5% 87.5 -2.0% 8.0 37.8% 76.0 -0.7% 11.5 -12.1%

87.2 85.8 12.8 74.9 10.8

92.0 87.1 8.0 75.6 11.5

-5.5% -1.5% 37.5% -0.9% -6.5%

** Indicates the statistical significance at =0.05; RD stands for relative difference between multimode group estimate and CATI group estimate.

Summary  Unit response rates – The self-administered data collection mode (mail and web) helped improve the response rate. – The non-Hispanic Asian group was less likely to respond to the survey by CATI only compared to other race/ethnic groups.

 Item nonresponse rates – In general, the data collection mode did not greatly affect overall data quality. – Specifically, we found no statistically significant differences on a variety of survey items, including demographic, degree, financial, and employment information.

Summary (cont’d.)  Survey estimates The CATI-only sample and multimode sample provided significantly different survey estimates for proportions of graduates in each field of major before nonresponse adjustments. – Correlation with response propensity (certain major groups are more likely to respond to certain modes of data collection) – Such differences became negligible with nonresponse adjusted weights (weighting cells include field of major)

Summary (cont’d.)  Differences between CATI-only and multimode samples existed even after nonresponse adjustments for other employment-related survey estimates. – Implies that respondents answer the same questions differently when interviewed in different data collection modes – Must be cautious when comparing the results from multiple rounds of NSRCG data collection that involved different data collection methods