The nonrandom selection of don't knows in binary and ordinal ...

Quality & Quantity 29: 87-110, 1995. 9 1995 Kluwer Academic Publishers. Printed in the Netherlands.

87

The nonrandom selection of don't knows in binary and ordinal responses: Corrections with the bivariate probit model with sample selection TIM FUTING LIAO University of Illinois-Urbana, Department of Sociology, 326 Lincoln Hall, 702 S. Wright St. Urbana, Illinois 61801, U.S.A.

Abstract. On social surveys don't knows are a common answer to attitudinal questions, which often have binary or ordinal response categories. Don't knows can be nonrandomly selected

according to certain demographic or socioeconomic characteristics of the respondent. To model the sample selection and correct for its bias, this paper discusses two types of bivariate models - binary probit and the ordinal probit model with sample selection. The difference between parameter estimates and predicted probabilities from the analysis modelling the sample selection bias of don't knows and those from the analysis not modelling don't knows is emphasized. Two empirical examples using the 1989 General Social Survey data demonstrate the necessity to correct for the bias in the nonrandom selection of don't knows for binary and ordinal attitudinal response variables. A replication of the analyses using the 1990 and 1991 General Social Survey data helps demonstrate the reliability of the sample selection bias of don't knows.

Introduction

Social scientists c o n d u c t m a n y surveys o n which s u b j e c t i v e , a t t i t u d i n a l questions a r e a s k e d . A c h a r a c t e r i s t i c o f t h e s e q u e s t i o n s is t h a t t h e i r r e s p o n s e c a t e g o r i e s m o s t o f t e n a r e d i s c r e t e o r n o n c o n t i n u o u s . A n o t h e r f e a t u r e is t h a t t h e r e a r e o f t e n m a n y d o n ' t k n o w s ( D K ) , p o s s i b l y b e c a u s e a t t i t u d i n a l questions a r e less c l e a r - c u t t h a n r e g u l a r d e m o g r a p h i c a n d b e h a v i o r a l q u e s t i o n s o r b e c a u s e the r e s p o n d e n t s i m p l y d o e s n o t h a v e an i d e a a b o u t o r a t t i t u d e t o w a r d t h e issue. A c o m m o n p r a c t i c e in t h e social sciences is to t r e a t t h e s e D K s as missing d a t a a n d o m i t t h e m just as with o t h e r missing data. B u t missing d a t a like t h e s e c a n n o t b e i g n o r e d ; if so t r e a t e d , e i t h e r i m p o r t a n t i n f o r m a t i o n is lost, o r t h e s a m p l e s e l e c t i o n is n o n r a n d o m thus b i a s e d , o r both. T h e p u r p o s e o f this p a p e r is to t r e a t t h e D K r e s p o n s e as a s e l e c t i o n c r i t e r i o n , a n d use a b i v a r i a t e p r o b i t m o d e l with s a m p l e s e l e c t i o n to investig a t e t h e effect o f s e l e c t i o n bias o n t h e e s t i m a t e s f r o m b o t h b i n a r y a n d o r d e r e d - r e s p o n s e p r o b i t m o d e l s o f a t t i t u d i n a l o u t c o m e s . B y so d o i n g we a s s u m e t h e D K r e s p o n d e n t s m a y o r m a y n o t h a v e an o p i n i o n b u t in e i t h e r case t h e r e l e v a n t i n f o r m a t i o n is c e n s o r e d . N o o p i n i o n can b e s e e n as o n e

88

Tim Futing Liao

type of opinion, and respondents who have no opinion may arguably form only a portion of the DK respondents including others who have an opinion but do not want to, or are not able to, express it. In this paper, attitudinal outcomes are modelled by taking into account the censored DK responses. First, the issue of don't knows and possible ways dealing with it are discussed. Next, the topic of selection bias models is reviewed, then the binary probit model with sample selection is given. This is followed by an empirical example using 1989 General Social Survey (GSS) data. In the second half of the paper the ordinal probit model with sample selection is introduced, and illustrated with an example using the GSS data. The results in either case demonstrate that sample selection bias significantly affects the binary or the ordinal probit model of the outcome equation in the parameter estimates and hence in the predicted probabilities of the response categories in the attitudinal dichotomous-response or ordered-response variable.

Where do DKs come from and how to deal with them?

Broadly speaking, DK responses are given on opinion surveys to reflect respondents' attitude or lack of attitude toward the question. In that regard, a study of DK responses can start from the study of attitude formation and change. Some scholars (e.g., Smith, Bruner, and White, 1956; Katz, 1960) have researched on the understanding of the motivational base for holding and changing attitudes. The issue of DK responses is also related to the study of belief systems. For instance, the belief system of the elite and that of the masses are found to be different (Converse, 1964). By extension, asking the same question to the elite and the masses may arouse a different proportion of DK responses in them. Converse (1964) studied the sources of constraint on idea elements: These include logical, psychological, and social sources of constraint. I consider these sources of constraint on belief systems as applicable to the study of DK responses. If a question appears illogical to a respondent, a DK response may be in order. A respondent's socioeconomic background or the niche in the social structure the person occupies may set a constraint on the attitude formation and help generate a DK response. The issue of DK responses can also be approached by studying nonattitudes. Converse's (1970) pioneering research on nonattitudes showed that low consistency in attitudes in panel surveys are due to nonattitudes, though this has been criticized as caused by instrument error (Achen, 1975; Pierce and Rose, 1974). Are DKs nonattitudes? The answer is a qualified yes because DKs are not merely self-confessed nonattitudes, but contain atti-

Nonrandom selection of don't knows

89

tudes, such as ambivalent responses (Smith, 1984, p. 230). Ideally all survey questions should include all necessary categories so that ambivalence can be separated from nonattitudes. On the other hand, it is important to study nonattitudes because sometimes people may feel obliged to give an opinion on attitude surveys, thus leaving nonattitudes concealed with hastily-fabricated affective judgement (Converse, 1970). This has been shown by Schuman and Presser's (1978, 1981) experimental work that using DK filters would increase "no opinion" responses by 10 to 30 percent and might significantly change the substantive proportions as well. Although the effects of possible background factors were not as clear-cut, Schuman and Presser (1980, 1981) found that volunteered DK responses tended to rise with the level of education on the standard form. The effect of education on filtered form was less clear. In short, DK filters appear to elicit concealed unsure responses, and are widely used in attitude surveys, especially the quasi-filter, questions with which offer "no opinion" or "don't know" as one of the alternatives to respondents. If such alternative is not provided, many respondents will not attempt to assert no opinion, even though most of whom would do so if the option was offered (Schuman and Presser, 1981; Turner and Martin, 1984). Because of the lack of "crystallization" (nonattitude) on the part of respondents (Converse, 1970; Turner and Martin, 1984), DK responses are bound to appear on attitude surveys even if alternatives for ambivalent answers are included. How do we as researchers deal with these DK responses when analyzing attitudinal data? A common, though dangerous, practice treats these responses similar to other types of missing data and removing them from the analysis. So doing may not present any problem if the proportion of DKs is rather small, but it can bias the analysis if the proportion is nonignorable. Thus, in principle the practice should be avoided. A feasible approach is also to treat these responses as regular missing data and use Little and Rubin's (1987, 1989/1990) multiple imputation to get at the responses. EM can be applied to the models of missing data to provide ML estimates from incomplete data. This is the most recent of a long tradition of research on missing data: Statistical models for nonignorable missing data including nonresponse have been studied by way of prior odds of response for categories of the table that modify the likelihood (Little, 1982; Nordheim, 1984; Pregibon, 1977); these models have also been examined within the framework of the contingency table and log-linear models (Fay, 1986; Little, 1985; Little and Rubin, 1987). Recently, Little and Rubin (1989/1990) in an excellent article discussed and reviewed the use of maximum likelihood estimation, with the EM algorithm, and multiple imputation in missing data problems for both continuous and discrete data.

90

Tim Futing Liao

Clogg (1982, 1984) proposed an ingenious method using association models to treat DKs as a meaningful category in the attitudinal variable of interest and estimate the location of the DK category with respect to other response categories. This approach should be considered and used to model DKs if at least one nondisputable instrumental variable is available. In the following I consider an alternative approach to modelling attitudinal data with DK responses. I use sample selection bias models to analyze attitudinal responses. In these models DK responses are regarded as cases for which valid information is censored because of nonattitudes or ambivalence, and the effect of censoring is corrected. How do we determine that we may have a problem with sample selection of DKs? I suggest using two criteria: one substantive and one statistical. Statistically, there ought to be enough DKs cases to warrant attention. Two or three DKs out of a sample of 500 observations probably will not have any consequence for the analysis even though these cases may have attributes different from other observations. If a sample has about 5% or more DKs, we should consider sample selection bias as a possibility. Substantively, the knowledge of whether certain type of respondent tends to give a DK answer to a particular question will help discern a possible sample selection bias. Often we do not have such knowledge; we may need to react on suspicion. If there is a sizable (about 5% or more) DKs in a sample and we are not sure that sample selection is not a problem, then we should exercise caution and consider modelling DKs with sample selection models.

Sample selection bias models Because of the quasi-experimental nature of sociological research, the detection and correction of sample selection bias is of great importance. Following Heckman's (1976, 1979) seminal work, sociologists have shown a strong interest in censored sampling and sample selection models (e.g., Berk, 1983; Sorensen, 1977; Stolzenberg and Relles, 1990; Tuma and Hannan, 1979, 1984). There has also been much sociological research applying Heckman's model, in which the equation of primary interest, often called the outcome equation, has a continuous dependent variable (for a list of applications, see Stolzenberg and Relles, 1990). A shortcoming with the type of selection bias models is the normality assumption. The most recent econometric research has been on semiparametric and nonparametric estimation of sample selection models with a continuous outcome variable (see Heckman, 1990; Manski, 1990; Newey, Powell, and Walker, 1990; and the references therein). While the research discussed above is focused on the sample selection

Nonrandom selection of don't knows

91

model with a continuous outcome variable, recent research has examined selection bias models with a dichotomous outcome variable by using bivariate probit and logit models (Dubin and Rivers, 1989/1990; for an earlier sociological application of a probit model with sample selection, see Mare and Winship, 1984). In the following sections I study the issue of D K responses via sample selection bias models with a binary outcome variable and such models with an ordinal outcome variable. The latter case - selection bias models with an ordinal outcome variable - is a natural extension of the binary case but has not seen empirical applications.

The binary probit model with sample selection Instead of a classical regression outcome equation with a continuous dependent variable, now let us have a usual binary probit model (see Maddala, 1983) as the outcome equation when sample selection bias is suspected. A natural extension of the usual probit model is to allow for more than one equation, with correlated error terms. Following Greene (1990), the general specification of a two-equation bivariate probit model is as follows:

y~ =/3~x~ + E~, Y1=1

if y * > 0 ,

0otherwise,

(1)

and y* = fl~x2 + e2, Y2=1

if y * > 0 ,

(2) 0otherwise.

If we think in terms of sample selection bias, Equation (1) is similar to the primary equation of interest in Heckman's model (1976, 1979), except that the dependent variable is binary. Equation (2) is identical to the selection equation in Heckman's formulation. Here yl is observed only when Y2 = 1. That is, when observations satisfy the selection criterion (or giving a valid answer rather than DK) in Y2, responses are not censored for y~. In this model, the expectation of the error terms is normalized to zero, E[Ea] = E[c2] = 0, and their variances are normalized to one, Var[el] = Var[E2] = 1. Their covariance is therefore equal to their correlation, Cov[E~, e2] = P. Another way to express the expected value of y] is as a function of/1~xl and /3~x2 instead of/3~xl alone, such that E(y*) = f(~ixl) + g(/3~x2), where f(.) and g(.) are functions to be specified.

92

Tim Futing Liao

The bivariate normal cumulative distribution function (cdf) for the relation in Equations (1) and (2), denoted by q)z(xl, x2, p), is given below:

Prob[X~ 5%) DKs using the bivariate probit model. The questions are about whether sexual materials provide information about sex, lead people to commit rape, and provide an outlet for bottled-up impulses. None of the converged models studying sample selection bias of DKs in these questions was substantially changed in their parameter estimates due to the nonrandom DKs. Since I first conducted the analysis, the 1990 and 1991 GSS data have subsequently become available. To get an idea about the reliability of the selection bias of DKs, I replicated the analysis (reported in Table 2) with a

98

Tim Futing Liao Answering

"Yes ....

Binary

Probit

0.90

0.85

0,lIB

0.75 .Q 0.70 ~a o

0.65

0.60

0.55

0.50

I

I_

I

20

I

I__

40

I

60

80

Age []

Model

1

+

Model

2



Model

3

Fig. 1. Predictedprobabilities.

combined sample of the 1990 and 1991 GSS data. The results are presented in Appendix 1. With the GSS data from these two years, the sample selection of DKs is still a problem. The estimated p is large and significant. Besides, there are some sizable changes in the estimates. The change in the size of the estimate for the religion variable is over 15% and that for the education variable is over 30%.

The ordinal probit model with sample selection Often attitudinal questions have ordered scales. Examples are many, such as the varying-degree Likert-type scales of agreement and importance. A statistical model appropriate for analyzing dependent variables with ordered categories is the ordinal (or the ordered-response) probit model (and its logit counterpart). This model was first considered by Aitchison and Silvey (1957) and Ashford (1959). It was later extended, refined and popularizcd by McKelvey and Zavoina (1975).

Nonrandom selection of don't knows

99

Similar to the binary or binomial probit model, the ordinal probit model is built around a latent regression. That is, y* is not observed while y is: Y { = ][~{X1 -t- 151,

y,=0 =1 =2 =J

if y* ~< 0, if0