Comparing Interviewer Variance Between Telephone and ... - mapor

Comparing Interviewer Variance Components Between Telephone and Text Modes Huiying Yanna Yan*, Brady T. West*, Frederick G. Conrad*, Michael F. Schober†

* University of Michigan, Ann Arbor, MI NY

† The New School for Social Research, New York, 1

Introduction • What is the interviewer variance component?

𝒚=𝝁

Introduction • What is the interviewer variance component?

𝒚=𝝁

𝒚𝒊 = 𝝁 + 𝜺𝒊 𝜺𝒊 ∼ 𝑵(𝟎, 𝝈𝟐 )

Introduction • What is the interviewer variance component? Iwer A

Iwer C

Iwer B

𝒚=𝝁

𝒚𝒊 = 𝝁 + 𝜺𝒊 𝜺𝒊 ∼ 𝑵(𝟎, 𝝈𝟐 )

𝒚𝒊𝒋 = 𝝁 + 𝜶𝒋 + 𝜺𝒊𝒋 𝜶𝒋 ∼ 𝑵(𝟎, 𝝉𝟐 ) 4 𝜺𝒊𝒋 ∼ 𝑵(𝟎, 𝝈𝟐 )

Introduction • Previous Studies  Magnitude of the interviewer variance component 2

𝜏 ,

𝜌=

𝜏2 𝜏2 +𝜎 2

Mahalanobis 1946, Franzen and Williams 1956, Feldman et al , 1951 Stock and Hochstim 1951, Hansen et al 1961, Kish 1962 , U.S. Census Bureau, 1985, Hill 1991,Bailar et al1977, Bailey et al 1978, Freeman and Butler 1976, Tucker 1983, Groves and Magilavy 1986, Tourangeau et al 1997, O‘Muircheartaigh and Campanelli 1998, Groves 2004, West and Olsen 2010 , West and Elliott 2013

 Explanation for the interviewer variance component Question type: attitudinal, ambiguity of the questions, open-ended questions Interviewer interactions: probing and feedback Hansen et al 1961, Feather 1973, Fellegi 1964, Feather 1973, O’Muircheartaigh 1976, Fowler and Mangione 1985, Groves 1989, Mangione, Fowler, and Louis 1992, O'Muircheartaigh and Campanelli 1998

5

Introduction • Emerging interviewer-administered mode  Text Mode: Rs disclose more and round less (Schober et al 2012)

6

Introduction • Research Questions  RQ1: Can we see interviewer variance components in text mode? Are they smaller than those in telephone mode? • •

•

Data quality differences were found between automated mode and intervieweradministered mode. (Schober et al 2012) Suggests social presence in the text mode. Could be due to the fact that… o Interviewer introduced themselves by name in the beginning of the interview o Human-like interviewer-respondent interactions, such as probing Human-like interactions are more frequently observed on telephone

 RQ2: Might sensitivity of the questions and the interviewer-respondent interactions produce more interviewer variances in telephone than text mode? • •

Voice conveys social cues, which may create more variation in social presence Telephone is synchronous, interviewers have to fill the silence

7

Introduction • Research Questions  RQ1: Can we see interviewer variance components in text mode? Are they smaller than those in telephone mode? • •

•

Data quality differences were found between automated mode and intervieweradministered mode. (Schober et al 2012) Suggests social presence in the text mode. Could be due to the fact that… o Interviewer introduced themselves by name in the beginning of the interview o Human-like interviewer-respondent interactions, such as probing Human-like interactions are more frequently observed on telephone

 RQ2: Might sensitivity of the questions and the interviewer-respondent interactions produce more interviewer variances in telephone than text mode? • •

Voice conveys social cues, which may create more variation in social presence Telephone is synchronous, interviewers have to fill the silence

8

Data • iPhone Experimental Study  Data Collection o Recruiting: 318 Rs from various online sources (Antoun et al 2013) o Interviewers: 8 administered both; 6 retained for each mode  

About 20 Rs per interviewer, 288 Rs in total Standardized interviewing, but could provide definition on demand

o Randomization: across telephone and text modes o Randomization: across interviewers? 

Not by design, but in fact…  R’s attributes did not differ across interviewers: age, gender, race, income, education.  Administering telephone interviews and text interviews are independent

o Questions: 14 factual numeric questions (sensitive vs. non-sensitive) 9

Data • iPhone Experimental Study  Data Collection o Recruiting: 318 Rs from various online sources (Antoun et al 2013) o Interviewers: 8 administered both; 6 retained for each mode  

About 20 Rs per interviewer, 288 Rs in total Standardized interviewing, but could provide definition on demand

o Randomization: across telephone and text modes o Randomization: across interviewers? 

Not by design, but in fact…  R’s attributes did not differ across interviewers: age, gender, race, income, education.  Administering telephone interviews and text interviews are independent

o Questions: 14 factual numeric questions (sensitive vs. non-sensitive) 10

Data • iPhone Experimental Study  Behavioral Coding  Interviews were transcribed verbatim  3 independent coders coded all cases. (agreement%>90%)  Codes: including interviewer’s behaviors (probing, providing definition etc.)  Coding software: sequence viewer 5.1 (Mac App)

11

Method • Model Specification (Hierarchical Linear Model)  𝑦𝑖𝑗 ∼ 𝑃𝑜𝑖𝑠𝑠𝑜𝑛(𝜆𝑖𝑗 )

 log(𝜆𝑖𝑗 ) = 𝛽0 + 𝛽1 𝐼𝑡𝑒𝑥𝑡 + 𝑢𝑗(𝑡𝑒𝑥𝑡) 𝐼𝑡𝑒𝑥𝑡 + 𝑢𝑗(𝑝ℎ𝑜𝑛𝑒) 𝐼𝑝ℎ𝑜𝑛𝑒 2  𝑢𝑗(𝑡𝑒𝑥𝑡) ∼ 𝑁(0, 𝜏𝑡𝑒𝑥𝑡 ) 2  𝑢𝑗(𝑝ℎ𝑜𝑛𝑒) ∼ 𝑁(0, 𝜏𝑝ℎ𝑜𝑛𝑒 )

• Statistical tools (SAS): proc glimmix, covtest homogeneity

12

There is Interviewer variance in text mode RQ1: Can we see interviewer variance components in text mode? Are they smaller than those in telephone mode?

Most of them are significantly greater than zero!

Q23. On the average day, about how many hours do you personally watch television? Q26. During the last month, how many times did you shop in a grocery store? 13

Less interviewer variance in text than telephone RQ1: Can we see interviewer variance components in text mode? Are they smaller than those in telephone mode?

Less in text on 12 out of 14 Qs ! Not statistically significant though

14

Question sensitivity may affect interviewer variance RQ2: Might sensitivity of the questions and the interviewer-respondent interactions produce more interviewer variances in telephone than text mode? Sensitive Qs

non-sensitive Qs

Topic sensitivity ?

Interviewer variance

Q7. How many sex partners have you had in the last 12 months? Q28. How many songs do you currently have in your iPhone? Response scale 0--20000 15

Extreme small interviewer variance may raise a flag RQ2: Might sensitivity of the questions and the interviewer-respondent interactions produce more interviewer variances in telephone than text mode? Sensitive Qs

non-sensitive Qs

Topic sensitivity ?

Interviewer variance

Q7. How many sex partners have you had in the last 12 months? Sensitive Q Fewer disclosure in telephone mode. Most of people reported 0, 1, or 2 sex partners, whereas in text mode we saw more larger reported values.

16

Interviewer behaviors differ in telephone and text mode • RQ2: Might sensitivity of the questions and the interviewerrespondent interactions produce more interviewer variances in Telephone than Text mode?  Interviewer behaviors among Q5,7,10,11,28:    

More probing in Telephone mode? Yes (6.0%>2.2%) More turns in Telephone mode? Yes (3.61 turns>2.27 turns) More definitions provided in Telephone mode? Yes (1.1%>0.9%) More probing/interactions for interviews in which interviewers have extreme random effects? No, non-paradigmatic (i.e. any interaction patterns that are not simple question-answer sequence) interactions are quite few.

17

Take-home points • • •

Interviewer variance components are observed in text mode Smaller interviewer variance components in text than telephone mode for 12 out of 14 Qs Topic sensitivity might affect interviewer variance. o Might be higher discrepancies of interviewer variance between two modes on sensitive questions o Extremely low interviewer variance in sensitive questions may raise a flag about socially desirable responding

• •

More turns, more probing, more definitions given in Telephone than Text mode; however, these behaviors are too rare for conclusive finding. The differences between modes may go beyond what we measured 18

Take-home points • • •

Interviewer variance components are observed in text mode Smaller interviewer variance components in text than telephone mode for 12 out of 14 Qs Topic sensitivity might affect interviewer variance. o Might be higher discrepancies of interviewer variance between two modes on sensitive questions o Extremely low interviewer variance in sensitive questions may raise a flag about socially desirable responding

• •

More turns, more probing, more definitions given in Telephone than Text mode; however, these behaviors are too rare for conclusive finding. The differences between modes may go beyond what we measured 19

In the future • Alternative estimation: Bayesian approach (West and Elliott 2013) • Negative binomial models • Comparing intra-class correlation coefficients (i.e. roh) • Design an ad hoc study:  better randomization, interpenetrated design  larger sample size (more interviewers)

20

Thank you • Contact info: [email protected]

21