Working Paper series of the Program in Survey Research and Methodology University of Nebraska, Lincoln Gallup Research Center http://sram.unl.edu
Origins and Developments of the Cognitive Models of Answering Questions in Survey Research
Mario Callegaro Program in Survey Research and Methodology, University of Nebraska, Lincoln
Working paper no 14 September 2005
This paper was presented at the 1st Conference of the European Association of Survey Research http://www.easr.upf.edu/ The full citation is: Callegaro, M. (2005, 18-22 July). Origins and developments of the cognitive models of answering questions in survey research. Paper presented at the First annual meeting of the European Association for Survey Research (EASR), Barcelona. This version has been submitted for publication Corresponding author: Mario Callegaro email:
[email protected] __________________________
Running head: COGNITIVE MODELS OF ANSWERING QUESTIONS
Origins and Development of the Cognitive Models of Answering Questions in Survey Research
Mario Callegaro University of Nebraska, Lincoln Survey Research and Methodology Program
Cognitive models of answering questions Abstract During the last thirty years, several models have been proposed to explain how respondents process information in a survey situation. They refer to a standardized interview situation where the respondent is generally asked close-ended questions and has to provide an answer by selecting one the response options provided. The movement called Cognitive Aspects of Survey Methodology (CASM) sought to study how respondents process a standardized survey questions. It brought together cognitive psychologist and survey researchers to study cognitive factors that influence survey responses and the ways in which survey questions might be designed to take these factors into account. This paper presents thirteen models, three precursors, and two spin-off models showing how a convergence was reached over the last years toward a four steps model that became the paradigm of the survey response process. Now the “four steps model” is taught in survey research handbooks. A comparison of the models is conducted to highlight commonalities and differences and to show how some ideas introduced in the early stages of survey research and later forgotten about are subsequently reintroduced again. The paper concludes with an evaluation of the cognitive modeling idea and its empirical support.
1
Cognitive models of answering questions Origins and Development of the Cognitive Models of Answering Questions in Survey Research During the 2005 annual conference of the American Association for Public Opinion Research (AAPOR) the Innovator Award was given to the founders of the Cognitive Aspects of Survey Methodology (CASM) movement1. The research and funding stimulated by the movement led to the extension and refinement of a set of theories explaining response errors in surveys. This paper documents the history of the theories that led to the development of cognitive models of answering questions, now main components of the survey research and questionnaire design literature. After an introduction describing the beginning of survey research, three precursors of question answering models will be presented. Then the discussion will concentrate on the task-analysis models of the survey response process, two spin-off models, and a comparison of the theories delineated in the paper. A final evaluation with some open question will conclude the paper. The beginning of survey research in the forties was distinguished by a strong interest in the “wording of questions”. Cantril (1944) reported a number of split ballot studies comparing different forms of a question. Several years later, Payne (1951) concentrated on the different types of questions asked in a survey, concluding with a checklist of 100 “considerations” to take into account before and during the question writing process. In the sixties, the researchers moved from the characteristics of the questions to the characteristics and influence of the interviewer and the reaction of the respondent to the survey interview (Cannel & Kahn, 1968; Hyman, 1954; Kahn & Cannell, 1957). However, the results of the accumulation of research showed that the influence of the interviewer, and the characteristics and motivation of the respondent had been overrated. In fact, the evidence suggested that “the characteristics of the task are the major source of response effects and are, in general, much larger than effects due to the interviewer or respondent characteristics” (Bradburn, 1983 p. 291 bold added). For this 1
AAPOR annual conference, May 12-15, 2005. The 2005 Innovators Awards goes to Thomas B. Jabine, Miron L. Straf, Judith M. Tanur, and Roger Tourangeau.
2
Cognitive models of answering questions reason the field, led by the work of Sudman and Bradburn (1974) and Schuman and Presser (1981), returned to study the characteristics of the question. Cognitive psychologists began to shed light inside the “black box” of the survey process the mind of the respondent. The earliest instance of the collaboration between cognitive psychologists and survey researchers goes back to a seminar organized by the British Social Science Research Council and the Royal Statistical Society in 1978 (Moss & Goldstein, 1979). In the U.S., survey researchers, statisticians and cognitive scientists were brought together to explore the cognitive and memory issues raised by the National Crime Survey (Biderman, 1980). “Unfortunately, some of the most intriguing findings from the resulting experiments were never published and are buried in hard-to-find memoranda” (Tourangeau & McNeeley, 2003, p. 11). The Advanced Seminar on the Cognitive Aspects of Survey Methodology (CASM I, June 1983), organized by the National Academy of Sciences’ Committee on National Statistics built a bridge between cognitive science and survey research by starting a new research program named CASM (Jabine, Tanur, Straf, & Tourangeau, 1984), which is still a major component of the survey literature (Tanur, 1999). This seminar has been advocated as the official beginning of the CASM movement2. Under the CASM umbrella a series of workshops called QUEST (QUestionnaire Evaluation STandards) were organized, beginning in 1995 (Esposito, 2001). The QUEST group took the lead to organize a main conference sponsored by the American Statistical Association called “Questionnaire Design, Evaluation, and Testing”; it took place in North Carolina in 2002. A book containing a set of selected papers from the conference was published early in 2004 (Presser et al., 2004). One of the main contributions of the CASM movement to the field of survey research was to investigate the cognitive tasks respondents perform in order to complete a standardized questionnaire. The cognitive paradigm can be summarized in a three-stage survey response 2
For the history, prologue, and development of the CASM movement see Jobe & Mingay (1991) and Tanur (1999)
3
Cognitive models of answering questions process: (1) the question is administered to the respondent, (2) the respondent performs some cognitive tasks, and (3) the respondent answers the question (Sirken & Schechter, 1999). The present paper focuses on the second and third stage of survey process and presents different models of respondent cognitive tasks in a comparative overview. Models of information processes that include the interviewer’s cognitive processes will not be presented. Furthermore the discussion will regard household surveys only, and will not address cognitive models of answering questions for establishment surveys (Edwards & Cantor, 1991; Sudman, Willimack, Nichols, & Mesenbourg, 2000; Willimack & Nichols, 2001). A past review was written by Jobe and Hermann (1996), where the focus is on seven models of response answering process, selecting the ones that deal with autobiographical events. This paper focuses on venue in the history and development of the CASM movement, specifically on the different models of the question answer process that converged over the years to the well established model, the four steps model (comprehension, retrieval, judgment, and reporting) that is now the paradigm taught in the most recent textbooks (e.g. Biemer & Lyberg, 2003; Groves et al., 2004; Presser et al., 2004). Early precursors of question answering models The very first models of the response process were developed by psychometricians. They were particularly interested in measuring attitudinal strength about an issue. Thurstone (1927) developed the first model to measure the psychological processes that lead to expressing an attitude or opinion. The law of comparative judgment formalizes mathematically how people compare across several stimuli, or, in survey terms, how people choose among response options. However, psychometricians were not really interested in how respondents construct the response. This is because the response is assumed to be constructed by two components: the true value and the error, considered to be randomly distributed. Since the error is randomly distributed with the increasing number of questions, it is not worth trying to understand how it is
4
Cognitive models of answering questions generated. “Thus the psychometrician is not primarily interested in how a question is understood and how a answer is generated” (Strack & Schwarz, 1992, p. 176). In the survey field, two interesting contributions regarding the understanding of response processes can be traced back to the sixties. Lansing, Ginsburg and Braaten (1961) delineated the response error theory reporting the outcomes of three field experiments. They measured the magnitude and the direction of response error in financial surveys on saving accounts, loans, and car debt. For each study, external validation data was collected. The investigation delineated three major causes of response error: (a) motivational factors, (b) failure of communication, and (c) inaccessibility of the information to the respondent. Motivational factors are not only the level of interest in the survey, but also the willingness to provide information due to social desirability and self-presentation factors. Failure to communicate arises mainly because the respondent does not understand what information is required, but also because the interviewer can fail to understand and record the answer provided. Inaccessibility can be due to different degrees of availability of the information requested from the respondent. In one case, the information does not exist in the mind of the respondent, or it is difficult to remember. The book concludes by advocating for a “psychology of survey error” (p. 204). Lansing and colleagues identified a task very similar to the comprehension stage discussed later by every task analysis model of the survey response process. The respondent “must understand what information is required” (p. 188). “Words may be used in a question which some people do not understand or understand in a sense different from that which is intended” (p.189). Inaccessibility of the information to the respondent can be seen as a memory encoding problem. “The information required may not be available at all to the respondent, or may be available to him only with varying degrees of difficulty” (p.188). The difficulty of recall resembles a retrieval process. “There will be psychological forces at work which will influence the rate at which different items are forgotten and may lead to distortions in what is remembered” (p. 188). Motivational factors are defined as willingness of reporting the information in a precise way. “He 5
Cognitive models of answering questions may wish to conceal or distort the facts” (p. 188). This last quotation can be read as an editing issue as discussed later by Strack and Martin (1987). In his classic work Questionnaire design and attitude measurement, Oppenheim (1966) devotes one subchapter to the process of responding. He starts with the assumption that the respondent has understood the question as intended and has the information required to respond. “The question has alerted him in a particular direction: he has searched that corner of his mind and has found what we are looking for” (p. 50). At this stage the process of responding begins. It is influenced by the way and the strength the information is organized: … some process of bringing into awareness has to take place, coupled with a degree of self-analysis, feedback, conceptualization of ideas, generalization from specific points and so on. The respondent is trying to form a percept of his own ideas. (Oppenheim, 1966, p. 50) During the process recall problems may arise. At this point the respondent has “some kind of inner picture of his response, though he has not yet communicated it” (p. 50). The communication step requires some ability to “put things into words adequately” (p. 50), and the willingness to do so. In the second edition of his book written 26 years later Oppenheim (1992) does not change the process of responding model much, nor does he acknowledge any of the models developed during those years. Task-analysis models of the survey response process The common denominator of the models presented in this section is the four steps model (Tourangeau, 1984). For this reason, it is useful to present this model at the beginning of this section to help the reader follow the chronological exposition of the other models and make more evident the communalities and differences of this model with the others. Tourangeau’s four steps process model (Tourangeau, 1984; Tourangeau & Rasinski, 1988) is the most cited in the literature. Four processes are delineated: comprehension of the question; retrieval from memory of the relevant information; use of the information to make a judgment, if the question 6
Cognitive models of answering questions calls for one; and finally selection and reporting of the answer. Comprehension is a crucial stage in the question answering process. The respondent has to understand the meaning of the question in order to answer it. Two processes work together at this stage. Top-down processing emphasizes that the text is understood by imposing a pre-existing structure on it - first we identify a general outline and then we recognize how each piece fits in the pattern. The respondent recognizes the interrogative form of the question and the response alternatives. Bottom-up processing emphasizes the role of prior knowledge in comprehension. The context is relevant to both approaches because it allows respondents to determine what the relevant information is and helps in disambiguating the meaning. This is especially important for attitude questions. Retrieval is the step where the respondent searches his/her memory to find information in order to come up with an answer. With factual questions events are recalled from the long-term memory. With attitudes the process is slightly different. Attitudes are seen as structures residing in the long-term memory too. Respondents retrieve a sample of their beliefs that are most accessible at that moment to form the attitude. But some events or attitudes cannot be retrieved, or are poorly retrieved. In the first case, the reason is that the event has never been transferred to the long-term memory, or encoded. In the second case, sources of forgetting are due to the time elapsed from the original event, the time of recall, a different context, similar events that become blurred into the mind, and interferences and intrusions that contaminate the original event. Judgment is a process that depends on the question and on the particular answer of the respondent. Some questions ask to combine pieces of information and this process requires some cognitive effort. Other times the answer does not need the judgment process because it is already stored in the mind of the respondent (ex. never did or experienced that) and does not require any judgment. For attitude questions the situation is more complex. Unless the respondent has a well-defined attitude or its complete absence (a non-attitude), the judgment process involves a scaling of beliefs, an assessment of their relative importance or weighting, and the combination into an overall judgment. After rendering a judgment, the 7
Cognitive models of answering questions respondent generally has to select an answer from the response options provided. In the case of open-ended questions, the process is more complicated because it requires a formulation of the answer. Selecting an answer means making a choice. This can entail different strategies, such as eliminating answers in order to narrow down the options or selecting the first option that seems satisfactory, and ignoring the others. For attitude questions the answer generally follows a scale or rating. There is also another process that may take place, called editing. Editing is done if the answer does not fit with the preceding answers, for consistency, or for social desirability issues. The four step process takes into account the fact that the order of the steps can be different or incomplete: for example when the respondent already selects an answer before the interviewer even finishes reading the question, or when the respondent refuses to answer, stopping the process at the first stage. This model focuses only on the respondent’s task, leaving out the interviewer’s cognitive processes. Tourangeau did not display the model in a pictorial format until very recently (Groves et al., 2004) as shown in Figure 1. ------------------------------[Figure 1 about here] -------------------------------
The first model of information processing in an interview situation, the question answering model was developed by Marquis in 1977 (Cannel, Marquis, & Laurent, 1977). The two actors are the respondent and the researcher. They interact throughout the questionnaire, where an event experienced by the respondent is the object of investigation of the researcher. ------------------------------[Figure 2 about here] ------------------------------In order to have proper recall, the question should interact adequately with the respondent’s cognitive organization. The model takes into account all the steps prior to the actual interview 8
Cognitive models of answering questions from the time the event is experienced; the integration of the event into the cognitive organization of the moment. The ability of the researcher is to match the question wording with the respondent’s memory trace of that event. In order to properly understand the focus of the model, one must remember that the authors were working on the Health Interview Survey data; i.e. mainly with factual data and for most of them they were also able to collect validation records. The model was subsequently improved and updated (Cannel, Miller, & Oksenberg, 1981). It is a two track model where some processes lead to an inadequate answer (boxes at the top of figure 3) and other processes lead to a careful answer (boxes at the bottom). ------------------------------[Figure 3 about here] ------------------------------In order to produce an adequate answer, the respondent must comprehend the question at a vocabulary level and at a conceptual level. Then an assessment of the information is needed in order to answer the question. This stage involves searching the memory for relevant information. The potential response is then evaluated for accuracy and if it is not adequate, the respondent can loop back to the preceding activities. At stage four the respondent evaluates the potential answer with a personal goal extraneous to the survey like self esteem or selfpresentation. At this point if the answer is judged non-threatening, it is reported. The second track (upper part of the diagram) highlights what can interfere with giving an accurate answer. Three factors can have an impact: the interviewer, the preceding questions, and the respondent’s beliefs and goals. Step six and seven are called situational cues. In a way they deviate from the original and “candid” answer, leading to three kinds of bias: conformity, social desirability, and acquiescence. The model allows for only one loop, from step three to step two; the other paths are one-way. Delamater (1982) noted that two aspects of stage two, the retrieval of the information and the decision concerning what information is necessary to respond, are similar to what Oppenheim (1966) called “bringing into awareness” (p. 50) of the information, 9
Cognitive models of answering questions and the assessment of its appropriateness. One weakness of this model is that it focuses mainly on what happens after the potential response is formulated and it does not go into much detail regarding the actual formulation of the answer - everything happens in stage two. The second problem is that it lacks specific detail, and the exposition is condensed in three pages. Nevertheless, it is able to identify key aspects of the response answering process and also focuses on the interviewer effect, an aspect that other models do not take into account Martin’s three steps model (Martin, 1983) has been forgotten in the literature (.i.e. it has never been cited by any of the other models) and was recently rediscovered by Willis (2004a). In the first step: “In order to formulate a response the respondent must impute meaning to the question” (p. 712). The second task is searching for relevant information on the basis of the meaning that was imputed by the question. Lastly, the respondent must integrate the information in order to formulate a response. Interestingly, this model is very similar to four steps model presented one year later by Tourangeau (1984). The information processing model (Strack & Martin, 1987) is geared towards attitudinal questions and accounts for context effects in surveys. The model is depicted in Figure 4. ------------------------------[Figure 4 about here] ------------------------------The respondent must understand the meaning of the question. The first level is the semantic meaning of the words. But this is not sufficient - there is a need to understand the intended meaning of the question as well. The intended meaning is understood by the content of the question itself, by the response options used as a basis for inferences, and by the preceding questions (context effect). The second step is generating an opinion on the issue. Here there are two possibilities, the respondent already has a judgment stored in memory and has to access it, or he/she has to come up with a judgment at the time of the interview. In the author’s opinion, the latter is the most frequent case. In order to do that, the respondent has to 10
Cognitive models of answering questions access the relevant information on the issues, decide how to use this information and compute a judgment. Since the judgment has to be provided in the format chosen by the researcher, the answer has to be formatted too. The respondent has to select one of the provided options. The original response may be edited for social desirability or self-presentation issues. Strack and Martin’s (1984) model is a two-track model because responses to attitude questions can be based on previously stored judgments, or on new judgment created on the spot during the interview. The latter is more prone to context effects. The following two models focus on autobiographical questions. The survey memory model (Strube, 1987) was developed for the discussion of the veridicality in recall and distinguishes various sources of possible influence as shown in Figure 5. The emphasis is on autobiographical questions. ------------------------------[Figure 5 about here] ------------------------------It is not the focus of this section to discuss the sources of influence for each cognitive step. What is important is that the model substantially accepts Tourangeau’s four steps process (1984). The highlight on the encoding stage however is new and reminds of a similar concept of encoding in the response error theory by Lansing, Ginsburg and Braaten (1961). The encoding of an event at the time the respondent experiences it determines the future likelihood of retrieval. Strube (1987) reminds survey researchers that we do not have any control on the encoding process; what we can do, instead, is to study and test better retrieval cues that match the encoding processes of the respondent. The rest of his paper discusses some of them. A very similar model, encoding plus four steps, was presented four years later by Eisenhower, Mathiowetz and Morgenstein (1991). Schwarz’s (1990) model is strictly related to answering quantitative autobiographical questions. This kind of question asks how often one did something or how much of something 11
Cognitive models of answering questions one has consumed in a specified reference period. Schwarz’s model adds one stage to the four step model decomposing the judgment stage in two phases. If the question asks for a reference period, the respondent has to determine if the event(s) that they just retrieved from memory occurred during the reference period or not. If the question calls for a “usual behavior” the respondent has to determine if the instances he/she recalled are a reasonable representation of the usual behavior, i.e. if they are typical. Next the respondent uses different strategies, such as decomposition strategies, availability heuristics, implicit theories of stability and change to infer a reasonable answer. “Respondents are likely to begin with some fragmented recall of the behavior under study and to apply various inference rules to arrive at a reasonable estimate” (p.107). The QUEST model (Graesser & Franklin, 1990) was originally developed to address answering general questions such as “Why?,” “How?,” “When?,” “Where?,” “What are the consequences of X?,” and “Is X true or false?” (Graesser, Roberts, & Hackett-Renner, 1990). Six years later the authors explained how to use the model for survey questions. QUEST is different from all the other models described in this paper because it is a computational model. A computational model seeks to specify in very fine detail the information and the processing components in a way that can be implemented on a computer. QUEST was designed to simulate the process of answering questions in people. The QUEST model specifies the cognitive strategies that people use when they answer nineteen different categories of questions. Examples of survey questions can be verification questions (Is X true?, “Are you citizen of the U.S.?”), concept completion (Who?, What? “Who is your physician?”), quantifications (How many? “How many children do you have?”), instrumental/procedural questions (What instrument or plan allows an agent to accomplish a goal? “How did you move to Memphis?”), or judgmental questions (What value does the answer place on an entity “What do you think about the new tax plan?”). Four major components of the QUEST model generate an answer to a question and are further specified in Table 1: 12
Cognitive models of answering questions •
Question interpretation. This component translates the question into a logical form that a computer can process. The question is then assigned to one or more of the nineteen categories of questions.
•
Access to relevant information sources. These sources provide the content that is needed to formulate the answer. QUEST activates the relevant generic knowledge sources and the specific ones.
•
Pragmatics. QUEST identifies the goals of the questioner and the respondent and the shared knowledge or common ground that they both share.
•
Convergence to relevant answers. After searching the vast background or relevant knowledge, QUEST produces a small subset of the information that constitutes a good answer to the question. ------------------------------[Table 1 about here] -------------------------------
The authors identified twelve potential problems with survey questions that can be anticipated by the QUEST model, five of them are currently handled by a software called QUAID3 - Question Understanding Aid (Graesser, Wiemer-Hastings, Kreuz, Wiemer-Hastings, & Marquis, 2000). QUAID evaluates each question on potential comprehension difficulties at various levels: unfamiliar technical terms, vague or imprecise predicate or relative terms, vague or ambiguous noun-phrases, complex syntaxes and working memory overload. The first two components of QUEST closely correspond to the comprehension and retrieval stage of preceding models, e.g. the question answering model (Cannel, Marquis and Laurent, 1981), the four step process (Tourangeau, 1984) and the information processing model (Strack and Martin, 1987). But the judgment process is part of component one, three and four in QUEST. Stage four
3
QUAID is currently available at the following website: http://mnemosyne.csl.psyc.memphis.edu/quaid/quaidindex.html
13
Cognitive models of answering questions in Cannel’s model is part of the convergence to relevant answers stage in QUEST. QUEST specifies the question answering strategies and the sources of information in a more detailed (computational) way than the other models because the final goal is the implementation on a computer. The encoding plus four steps process model by Eisenhower, Mathiowetz and Morgenstein (1991) was introduced to address response error associated with the recall of autobiographical memories. For this reason the authors introduced the encoding process as a potential source of error. The encoding process refers to the formation of memory. Schema theory models have highlighted the incompleteness and distortion of memory. What is stored in memory, or encoded, is determined by a knowledge framework that chooses and modifies experiences in order to arrive to a consistent representation of an experience. Memory is also abstractive - a verbatim record is very rarely stored into memory, instead the meaning of it is more likely to be stored. Another characteristic, according to the schema theory, is that memory has an interpretative characteristic such as filling in missing details or distorting the original event in order to make it consistent with the schema. If the possibility of a correct recall is determined by what schema is used by the respondent and by the matching of the question wording and response option to this schema, the second factor is whether the question asks about an event that the respondent experienced (self) or about an event someone else experienced (proxy). Memories about ourselves are likely to be organized differently than memories about others. For example, we can have a visual image about other’s experiences but not visual images about our own experiences. Social desirability effects may be stronger in self-responses than proxyresponses. Eisenhower, Mathiowetz and Morgenstein largely accept Tourangeau’s model (1984), adding the encoding step that precedes the other processes but is responsible for the accuracy of autobiographical report. Their model resembles closely Strube’s model (1987), even if the emphasis here is on recall error instead of retrieval cues.
14
Cognitive models of answering questions The flexible processing model (Willis & Bercini, 1991) was originally developed to explain how people answer health survey questions and is organized in four stages: encoding, comprehension of the question, decision/judgment, retrieval of information and response. Each stage is seen as an input-output phase, and the model allows going through each stage, without a predetermined sequence. ------------------------------[Figure 6 about here] ------------------------------During the encoding process, the respondent forms an internal representation of the question that serves as an output for the decision stage. In this phase the respondent judges if it is reasonable to search his/her memory for the information requested and also if it is worth doing so. The first loop (upper left of the diagram) can occur if the respondent determines that he/she has not adequately understood the question and asks for a re-phrasing of it. In this context the meaning of encoding is comprehension. The authors do not refer to the actual stage of encoding as explained by Strube (1987). The third stage is retrieval of the event from memory using a direct manner or estimation techniques based on the frequency and distinctiveness of the event (Blair & Burton, 1987). The response step is described as the reverse of the encoding process: the answer has to be verbalized by the respondent and re-coded into verbal or written form depending on the way the question is asked - oral versus written. The flexible processing model specifies the fact that judgment can happen both before and after memory retrieval: before is when the respondent determines if it is feasible and worth doing a memory search; after is when what is recalled is accurate or more retrieval is necessary. This model differs from the previous ones because “retrieval of information, once the question has been encoded, is not necessarily automatic” (Willis, Royston and Bercini, 1991, p. 252). The sequence is flexible because there are no sets of stages to go through. The meaning of the response stage is also
15
Cognitive models of answering questions different. In Tourangeau’s (1984) model the response is a selection among the response alternatives presented; in this case the selection is covered by the decision and judgment stage. The information exchange theory model (Sander, Conrad, Mullin, & Herrmann, 1992) assumes that the only way to explain the respondent’s answers is to take into account two other models: the interviewer model and the interviewer interaction model. The first one expresses how the interviewer formulates and reads the questions, and the second model explains the rules of the interaction between the interviewer and the respondent. Thus the authors present three models: the interviewer model of question generation, the respondent model of question answering and the interviewer-respondent interaction model. The focus of this section will be on the second model, exemplified in Figure 7. ------------------------------[Figure 7 about here] ------------------------------The model assumes a strict succession of steps in order to arrive at the answer. The successful steps (YES path) are depicted on the left of the diagram. The respondent recognizes the wording of the question, interprets its meaning, understands the question, feels to know the answer, and is willing to report it. In case one of these steps is not successfully completed (NO path), the respondent asks for clarification or makes a statement. The bottom part of the diagram describes the retrieval strategies, the judgment process and the response formulation by the respondent. An important stage is the “try direct retrieval” junction. If the respondent attempts this path, he/she uses some retrieval strategies that may or may not have a successful outcome. In the affirmative case the next steps are calculation and judgment. In case of an unsuccessful direct retrieval, some inferences are formulated in order to come up with an answer. If the inferences are plausible, the information goes to the judgment stage, otherwise the respondent asks for clarification or expresses a statement. The answer parsing follows the judgment stage without any other process or loop in the middle. The model is fairly complex and 16
Cognitive models of answering questions contains more stages than any other model presented in this review. Nevertheless, the information exchange theory offers many guidelines for survey researchers. The form appraisal model (Forsyth & Hubbard, 1992) was developed in order to build a coding system for appraising questionnaires, and was fully developed fours years later by Lessler and Forsyth (Lessler & Forsyth, 1996a). It is an adaptation of the original question answering model (Cannel, Marquis and Laurent, 1977). Forsyth and Hubbard add a fifth stage to Tourangeau’s model (1984), inserting an interpretative stage between the comprehension stage and the retrieval phase. The interpretative processes are used by the respondent to build a general representation of the question-task demands. They guide the other three processes, specifying what has to be retrieved from the memory or specifying the goals of the judgment process. After the form appraisal model published in 1992, no substantially new models were developed. The scholars seemed to agree on the four step process and the emphasis of the researchers was on using those models more than developing them. In the meantime two books were published in 1996 organizing the findings of the CASM movement. In the first one, ‘Thinking about answers” (Sudman, Bradburn, & Schwarz, 1996), the knowledge about survey response was summarized and systematized in the first comprehensive handbook. In the second one, “Answering questions” (Schwarz & Sudman, 1996), several techniques for determining and measuring cognitive and communicative processes in the survey interview were presented. Four years later other leading scholars (Tourangeau, Rips, & Rasinski, 2000) published “The psychology of survey response”, a book that reviewed and integrated the current research on survey responding. In that book the four step process model was updated, extended, and refined. I will refer to it later as the revised four steps process. The authors specified the mental processes that might be used by respondents in each of the four cognitive stages. This does not mean that each respondent is performing all of them, but it does depend on the content of the question. The level of accuracy and the time allowed for a response 17
Cognitive models of answering questions determines which set of processes the respondent will carry out. Table 2 highlights the main specific processes that can occur in each cognitive stage. Each of the components can cause a response effect. The authors substantially kept the original model and detailed the specific processes of each component. ------------------------------[Table 2 about here] ------------------------------If we take into account the years of publication of the different models we can see that after a slow start (Cannel et al., 1977; Cannel et al., 1981; Martin, 1983) the models were developed primarily in a short period of time, 1984-1992. This period coincides with an exponential increase in applied studies under the CASM umbrella reflected also by a special issue of the journal “Applied Cognitive Psychology” published in 1991. Two spin-off models The following two models accept the basic model by Tourangeau (1984) and apply it to solve a particular problem. They are presented in a separate section because they discuss mainly how the cognitive steps are carried out instead of concentrating on what cognitive steps are carried out. The satisficing model concentrates on “how” the respondents can carry out the cognitive steps in order to answer a question. The response decision model decomposes into explicit parts the process leading to a “don’t know” answer. The satisficing model (Krosnick, 1991; Krosnick & Alwin, 1987) does not explicitly identify the cognitive steps in answering a survey question, instead it focuses on the strategies respondents can implement in order to answer a question (see also Krosnick, Narayan and Smith (1996) and Krosnick (1999). The satisficing principle was first introduced to explain the response order effect, especially the fact that people with low cognitive sophistication are more prone to primacy effects when items are presented visually. The concept of satisficing is borrowed from Simon (1957). He offered the notion of satisficing as an alternative to the 18
Cognitive models of answering questions economic assumption that people use any effort to maximize the profit gathered from their decisions. People often spend only the necessary effort to make a satisfactory decision when faced with cognitively demanding decision tasks. When faced with survey questions, people can use a satisficing approach in order to provide a plausible answer. Motivation and cognitive ability are two key factors in the notion of satisficing. Krosnick uses the four step process model to explain the concept of satisficing and its opposite, optimizing. People who optimize carry on all the stages with every effort in order to come up with a proper answer. Satisficing, on the other hand, entails compromising at one or more of the four steps. We can see optimizing and satisficing as two extremes on the same scale. ------------------------------[Figure 8 about here] ------------------------------Satisficing has two levels; strong and weak. Strong satisficing implicates ignoring the retrieval and judgment steps. In other words the respondent interprets the question superficially and provides an answer that appears reasonable to the interviewer. They do very little or almost no personal introspection to search for an answer. They use cues in the answer to select an easily defensible response. Four strategies are examples of strong satisficing: (1) selecting a status quo option, when offered, because it is easy to justify keeping things as they are now; (2) selecting a no-opinion option when offered, because it is easy to claim ignorance on a topic; (3) selecting at random from the response alternatives, and (4) giving responses without much differentiation on a battery of items. Weak satisficing implicates carrying out all four steps but less carefully and thoroughly. The first answer that seems acceptable is reported. The two strategies that are examples of weak satisficing are: (1) selecting the first acceptable alternative in a close ended question, and (2) acquiescence where the respondent agrees with assertions offered in agree/disagree, true/false, and yes/no questions. The satisficing model provides a theory to explain many response effects already noted in the literature (see, for example, the 19
Cognitive models of answering questions classic work of Schuman and Presser (1981). It shares the assumption of a dual path to a survey response with the Cannel model. The concept of satisficing was already brought into the survey literature by Tourangeau (1984, p. 90) as an overall approach to answering questions, but was never fully developed as Krosnick did. The response decision model (Beatty & Herrmann, 1995, 2002; Beatty, Herrmann, Puskar, & Kerwin, 1998) was developed to address the issue of the “don’t know” (DK) answers in survey questions. Traditionally, researchers have assumed that DK answers reflect a lack of memory relevant to the question. This vision is judged as an oversimplification by the authors. The decision to respond or not to respond is driven by three factors: •
Cognitive state: the availability of the information requested.
•
Adequacy judgments; the respondent’s perception of the level of accuracy required by the questioner.
•
Communicative intent: the respondent’s motivation to provide the information requested.
Four cognitive recall states can classify the information through which the respondents draw their answer: 1. Available: The requested information can be retrieved with minimal effort. 2. Accessible: The requested information can be retrieved with effort or prompts. 3. Generatable: The requested information is not exactly known, but may be estimated using other information in memory. 4. Inestimable: The requested information is not known, and there is virtually no basis for estimation. (Beatty and Hermann, 2002, pp. 72-73) In the first state the respondent knows the answer by definition, in the fourth state the respondent does not really know or at best he/she can make a wild guess. In the two intermediate stages the respondent has some kind of information that can be used to generate the answer. The cognitive state is the basis to provide a response, but adequacy judgments and 20
Cognitive models of answering questions communicative intent also determine the final response. Adequacy judgment is needed when the potential response contains uncertainty, guessing or estimation, or in other words, when the respondent’s knowledge falls into the two intermediate stages: accessible or generatable. Communicative intent refers to the willingness of the respondent to report the answer. Social desirability and motivation are two important factors of influence on the communicative intent. Figure 9 shows all possible paths that lead to a response outcome from the four cognitive states. ------------------------------[Figure 9 and 10 about here, they can also go side by side] ------------------------------The odd numbered paths lead to a substantive response, the even numbered paths lead to a Don’t Know. T is a potentially truthful response; C is an error of commission - the respondent provides untruthful information; O is an error of omission - the respondent does not provide information that he/she can provide. Figure 10 summarizes the role of cognitive states, adequacy judgment, and communicative intent when answering a survey question. After the question is posed, the respondent must come up with a viable interpretation. If the respondent does not understand the question task, we have the first opportunity of item nonresponse (path a). In the next stage, the respondent may attempt to answer depending on his/her motivation. A very low motivation can lead again to item nonresponse (path b). For path c, the respondent decides to withhold information, even if the information is available and meets the objective of the question (cognitive state 1: available). The last explanation for item nonresponse is path d. In this case the information is inestimable (4) and the respondent believes that the inadequacy of the answer calls for a non response. The response decision model is a tool in understanding how respondents can decide to provide a substantive answer or just saying that he/she does not know the answer. The model can be seen as a decision strategy paradigm respondents use after having a potential response in mind. In order to have that, they must have gone through 21
Cognitive models of answering questions the comprehension phase at a minimum (viable interpretation). The errors of omission and commission are part of the editing process described by Tourangeau (1984) and Strack and Martin (1987). Pleading ignorance is a form of strong satisficing, as described by Krosnick (1991), or may be due to the respondent’s decision not to share that information. One of the most common reasons for such a decision is the respondent’s belief that the information, either behavior or attitude, is socially undesirable (DeMaio, 1984), or revealing the information puts the respondent at risk, for example in case of illegal activities (Turner, 1982). Some parallels can be found between this model and the response error theory. Lansing and colleagues (1961) discussed the inaccessibility of information, which can be equivalent to cognitive state two; motivational factor in answering the question, similar to path c, and failure of communication, close to path a. Comparison of response models In this section the models are compared according to three points of view: appropriateness for factual versus attitude questions, degree of commonalities of steps across the models, and degree of flexibility and succession among the steps. Table 3 illustrates the emphasis of the models in explaining the cognitive processes, depending on the kind of question asked. The majority of the models were developed to take into account cognitive processes related to answering behavioral questions. The models in italics are the most tailored in each of the categories: factual or attitude questions. Tourangeau’s model (1984; Tourangeau and Rasinski, 1988) is the most versatile, as it can be fully applied for both factual and attitude questions. ------------------------------[Table 3 about here] ------------------------------When comparing the models according to their degree of flexibility we notice that almost all models share a strict sequence of cognitive processes. Some, more than others, allow for backtracking; for example, the four step model, the question answering model, the flexible 22
Cognitive models of answering questions processing and the information exchange model. The revised four step process, question answering model, flexible processing and the information exchange models anticipate feedback loops in the processes. The most elastic models are the flexible processing model and the response decision model, where multiple loops and several paths can be taken. In the latest revision of the four step models Groves et al. (2004) finally acknowledge that “some degree of backtracking and overlap between these processes is probably the rule rather than the exception” (p. 203 bold added). The respondent may already have a partial answer while the interviewer reads the question but, after listening to the response alternatives, can backtrack to the comprehension stage because the response options have reshaped the meaning of the question stem. Is it reasonable to assume parallel processing of some processes? Tourangeau’s last revision of the model (Tourangeau, Rips & Rasinski, 2000) considers some possible parallel processing; for example, judgment can be parallel to retrieval, rather than following it. Some overlapping is possible between steps when, for instance, retrieval starts at the same time as comprehension. Overlapping can be seen yet as another form of parallel processing. The flexible processing model allows for parallel processing between the stage of recall and estimate. This view that the survey response process cannot be fully accounted by a simple serial processing model has been recently recognized by other authors (Ahola, 2004; Collins & Becher, 2001). Table 4 compares the models according to the cognitive steps. A first look reveals that there is a lot of agreement among models, and the common denominator is the four step process: comprehension, retrieval, judgment, and response. Only three models take into account the encoding process happening at the time of the experience. Another way to interpret the table is to look for the debate of what happens between the memory retrieval and the judgment stage. Once the information is brought from the long-term memory to the short-term memory, different models emphasize several processes that occur before the judgment stage. It is between the retrieval and the judgment stage that the models differ the most. 23
Cognitive models of answering questions ------------------------------[Table 4 about here] ------------------------------Final evaluation The literature of question answering processes is dense with several models; thirteen are presented in the present paper. The final goal of each model is to understand how respondents process information in a survey setting, trying to break down the different steps carried out in order to come up with the answer. In this historical overview, the reader can notice how the first models lumped many ideas together; Oppenheim, for example, had great insight into the “responding” process that was made more explicit by Cannel and subsequently by Tourangeau. The four step model is the common denominator for many other models that further specify some particular step (Sander et al., 1992), or try to apply the findings to a certain type of question (Schwarz, 1990). Over the years, the process of responding to a survey question has been decomposed into more and more steps. We can see a linear progression of the models, the latest model by Tourangeau et al. (2000) identifies thirteen possible mental processes that can be activated during an interview. We can see how the majority of models share a multicomponent approach while only a few (question answering, information processing model, and satisficing) share a two tracks or dual path idea. An implication for a cognitive theory of survey questions is to seek empirical evidence of the cognitive tasks performed by the respondents that are described by the authors of the different models. Sudman, Bradburn, and Schwarz (1996) present several methodologies to determine them: verbal protocols, behavioral coding, cognitive experts, response latency methods, sorting, focus groups and experiments. Table 5 offers a summary. ------------------------------[Table 5 about here]
24
Cognitive models of answering questions ------------------------------To update the list, three other promising methods can be added: eye tracking, usability studies, and usage of timing and paradata in computer assisted data collection, such as web surveys. Eye tracking studies can be useful in understanding the comprehension process, especially for self-administered questionnaires, via paper or computer screen. They can shed light on the way respondents read the questions, what they read, what catches their attention, and what they skip (Redline & Lankford, 2001). Usability studies generally videotape the respondent answering a questionnaire on a computer and create a recording of the computer screenshots, and a file log of the events with time stamps. One of the advantages of this methodology is the possibility of tracking editing processes and feedback loops in answering a question (Murphy et al., 1999). Collecting “paradata” (Couper, 2002) such as time latencies, keystroke files, changes of answers, and movement across the questionnaire can shed light on the different steps the respondents perform when answering a questionnaire. Table 5 shows that only one method can determine all four cognitive tasks directly from the respondent - the use of retrospective talkaloud techniques used in cognitive interviewing (DeMaio & Rothgeb, 1996; Willis, 2004b; Willis, DeMaio, & Harris-Kojetin, 1999). While several techniques can uncover the comprehension and retrieval steps, many of them are weak or inappropriate for the judgment and the editing stage. Retrospective thinkaloud produces some explanations for the judgment stage and the editing stage but depends mostly on the ability of the respondent to verbalize. Another critique is that a retrospective thinkaloud is always an expost justification and rationalization of the processes; just the act of reporting on the processes changes them. The latter is described as the “Heisenberg effect” by Sudman, Bradburn, and Schwarz (1996, p.16). Is it possible to measure cognitive activities without changing them? The answer to this question leads to an evaluation of the level of reactivity of each process. Retrospective protocols are based on spontaneous thinkaloud but mostly on direct probing, and demand a great deal of cognitive effort from the respondent (Groves, 1996). It is definitely a 25
Cognitive models of answering questions reactive and “intrusive” method. Other researchers are more extreme and argue that respondents are unaware of the judgment and some editing processes that happen in their mind. Cognitive interviews cannot expose directly nonconscious processing, such as judgment. Any verbalization is only a tentative reconstruction. The last implication is more philosophical in nature. In a way these models filter our view, directing the researcher to look for instances of the processes; the final result can be a selffulfilling prophecy. Comprehension, retrieval, judgment and response problems are found in cognitive interviews because the researcher is looking for them. But this does not necessarily mean that other processes are happening in the respondent’s mind. The process of responding is very complex and an answer is generally obtained in a matter of few seconds (Bassili, 1996). During this time, many processes are triggered in the mind of the respondent, who can use various strategies to answer a question. The entire picture is complicated by the presence of the interviewer and by the context in which the respondent is taking the survey. A careful study of the interplay of these three factors can better determine how a response is generated and thus confirm its validity. The history of the different models of the question answering process shows how a convergence of the scholars was reached around a model that comprises four steps; comprehension of the question, retrieval of information, judgment and estimation, and lastly, reporting an answer. This model is the backbone of modern questionnaire design, testing, and evaluation (Presser et al., 2004), especially in its methods of cognitive interviewing (Willis, 2004b), questionnaire appraisal (Lessler & Forsyth, 1996b) and behavioral coding (Fowler Jr. & Cannell, 1996). Finally, the four step model has reached general acceptance and is presented as “the main model” by review papers such as Bradburn (2004), and Schaeffer and Presser (2003).
26
Cognitive models of answering questions References Ahola, A. (2004). Cognitive model of the question-answering process and development of pretesting. In P. Prüfer, M. Rexroth & F. J. Fowler Jr. (Eds.), QUEST 2003. Proceedings of the 4th conference on questionnaire evaluation standards (pp. 26-33). Mannheim: Zentrum fur Umfrage, Methoden und Analysen (ZUMA). Bassili, J. H. (1996). The how and why of response latency measurement in telephone surveys. In N. Schwarz & S. Seymour (Eds.), Answering questions. Methodologies for determining cognitive and communicative processes in survey research (pp. 319-346). San Francisco: Jossey Bass. Beatty, P., & Herrmann, D. (1995). A Framework for evaluating "Don't know" responses in surveys. In American Statistical Association (Ed.), Proceedings of the section on Survey Research Methods (pp. 1005-1010). Washington DC: American Statistical Association. Beatty, P., & Herrmann, D. (2002). To answer or not to answer: Decision processes related to survey item nonresponse. In R. M. Groves, D. A. Dillman, J. L. Eltinge & J. A. L. Roderick (Eds.), Survey Nonresponse (pp. 71-85). New York: Wiley. Beatty, P., Herrmann, D., Puskar, C., & Kerwin, J. (1998). "Don't know" responses in surveys: Is what I know what you want to know and do I want to know it? Memory, 6(4), 407-426. Biderman, A. (1980). Report on a workshop on applying cognitive psychology to recall problems of the National Crime Survey. Washington, D.C.: Bureau of Social Science Research. Biemer, P. P., & Lyberg, L. E. (2003). Introduction to survey quality. Hoboken, NJ: Wiley. Blair, E., & Burton, S. (1987). Cognitive processes used by survey respondents to answer behavioral frequency questions. Journal of Consumer Research, 14(2), 280-288. Bradburn, N. M. (1983). Response effects. In P. H. Rossi, J. D. Wright & A. B. Anderson (Eds.), Handbook of survey research (pp. 289-328). San Diego: Academic Press. Bradburn, N. M. (2004). Understanding the question-answer process. Survey Methodology, 30(1), 5-15. 27
Cognitive models of answering questions Cannel, C. F., & Kahn, R. L. (1968). Interviewing. In E. Aronson (Ed.), The handbook of social psychology (2 ed., Vol. II Research methods, pp. 526-595). Reading, MA: AddisonWesley. Cannel, C. F., Marquis, K. H., & Laurent, A. (1977). A summary of studies of interviewing methodology. Vital and Health Statistics, Series 2(69), i-68. Cannel, C. F., Miller, P. V., & Oksenberg, L. (1981). Research on interviewing techniques. Sociological Methodology, 12, 389-437. Cantril, H. (1944). Gauging public opinion. Princeton: Princeton University Press. Collins, D., & Becher, H. (2001). Question testing and archiving of results to make best use of limited resources. In Proceedings of the QUEST 2001 conference (pp. 36-44). Washington D.C.: U.S. Census Bureau. Couper, M. P. (2002, 25-28 August). New technologies and survey data collection: Challenges and opportunities. Paper presented at the International Conference On Improving Surveys, Cophenagen. DeLamater, J. (1982). Response-effects of question content. In W. Dijkstra & J. Van der Zouwen (Eds.), Response behaviour in the survey-interview (pp. 13-48). London: Academic Press. DeMaio, T. J. (1984). Social desirability and survey measurement. A review. In C. T. Turner & E. Martin (Eds.), Surveying subjective phenomena (Vol. 2, pp. 257-282). New York: Russel Sage. DeMaio, T. J., & Rothgeb, J. M. (1996). Cognitive Interviewing techniques. In the lab and in the field. In N. Schwarz & S. Sudman (Eds.), Answering questions. methodology for determining cognitive and communicative processes in survey research (pp. 177-196). San Francisco: Jossey-Bass.
28
Cognitive models of answering questions Edwards, W. S., & Cantor, D. (1991). Toward a response model in establishment surveys. In P. P. Biemer, R. M. Groves, L. E. Lyberg, N. Mathiowetz & S. Sudman (Eds.), Measurement errors in surveys (Vol. 211-33). New York: Wiley. Eisenhower, D., Mathiowetz, N., & Morganstein, D. (1991). Recall error: Sources and bias reduction techniques. In P. P. Biemer, R. M. Groves, L. E. Lyberg, N. Mathiowetz & S. Sudman (Eds.), Measurement errors in surveys (pp. 127-144). New York: Wiley. Esposito, J. L. (2001). Brief history of the QUEST and Objective for QUEST2001. In Proceedings of the QUEST 2001 (pp. 1-5). Washington D.C.: U.S. Census Bureau. Forsyth, B. H., & Hubbard, M. (1992). A method for identifying cognitive properties of survey items. In American Statistical Association (Ed.), Proceedings of the section on Survey Research Methods (pp. 470-475). Washington DC: American Statistical Association. Fowler Jr., F. J., & Cannell, C. F. (1996). Using behavioral coding to identify cognitive problems with survey questions. In N. Schwarz & S. Sudman (Eds.), Answering questions. Methodology for determining cognitive and communicative processes in survey research (pp. 15-36). San Franscisco: Jossey-Bass. Graesser, A. C., & Franklin, S. (1990). QUEST: A cognitive model of question answering. Discourse Process, 13(3), 279-303. Graesser, A. C., Roberts, R. M., & Hackett-Renner, C. (1990). Question answering in the context of telephone surveys, business interactions, and interviews. Discourse Process, 13(3), 327-348. Graesser, A. C., Wiemer-Hastings, K., Kreuz, R., Wiemer-Hastings, P., & Marquis, K. H. (2000). QUAID: A questionnaire evaluation aid for survey methodologists. Behavior Research Methods, Instruments & Computers, 32(2), 254-262. Groves, R. M., Fowler, F. J. J., Couper, M. P., Lepkowski, J. M., Singer, E., & Tourangeau, R. (2004). Survey methodology. New York: Wiley. Hyman, H. H. (1954). Interviewing in social research. Chicago: The University of Chicago Press. 29
Cognitive models of answering questions Jabine, T. B., Tanur, J. M., Straf, M. L., & Tourangeau, R. (1984). Cognitive aspects of survey methodology: Building a bridge between disciplines. Washington DC: National Academy Press. Jobe, J. B., & Herrmann, D. (1996). Implications of models of survey cognition for memory theory. In D. Herrmann, C. McEvoy, C. Herzgog, P. Herter & M. K. Johnson (Eds.), Basic and applied memory research (Vol. 2 Practical applications, pp. 193-205). Hillsdale, NJ: Erlbaum. Jobe, J. B., & Mingay, D. J. (1991). Cogniton and survey measurement: History and overview. Applied Cognitive Psychology, 5(3), 175-192. Kahn, R. L., & Cannell, C. F. (1957). The dynamics of interviewing. Theory, techniques and cases. New York: Wiley. Krosnick, J. A. (1991). Response strategies for coping with the cognitive demands of attitude measures in surveys. Applied Cognitive Psychology, 5(3), 213-236. Krosnick, J. A. (1999). Survey Research. Annual Review of Psychology, 50, 537-567. Krosnick, J. A., & Alwin, D. F. (1987). An evaluation of a cognitive theory of response-order effects in survey measurement. Public Opinion Quarterly, 51(2), 201-219. Krosnick, J. A., Narayan, S., & Smith, W. R. (1996). Satisficing in surveys: Initial evidence. New Directions for Evaluation, 70, 29-44. Lansing, J. B., Ginsburg, G. P., & Braaten, K. (1961). An investigation of response error. Urbana. Champaign, IL: Bureau of Economic and Business Research, University of Illinois, Urbana. Lessler, J. T., & Forsyth, B. H. (1996a). A coding system for appraising questionnaires. In S. Sudman (Ed.), Answering questions. Methodology for determining cognitive and communicative processes in survey research (pp. 259-292). San Francisco: Jossey Bass.
30
Cognitive models of answering questions Lessler, J. T., & Forsyth, B. H. (1996b). A coding system for appraising questionnaires. In N. Schwarz & S. Sudman (Eds.), Answering questions. Methodology for determining cognitive and communicative processes in survey research (pp. 259-291). San Franscisco: Jossey-Bass. Martin, E. (1983). Surveys as social indicators: Problems in monitoring trends. In P. H. Rossi, J. D. Wright & A. B. Anderson (Eds.), Handbook of survey research (pp. 677-743). San Diego, CA: Academic Press. Moss, L., & Goldstein, H. (Eds.). (1979). The recall method in social surveys. London: NFER Publishing Co, Ltd. Murphy, E., Marquis, K. H., Hoffman, R. I., Saner, L., Tedesco, H., Harris, C., et al. (1999). Improving electronic data collection and dissemination through usability testing. Paper presented at the Federal Committee on Statistical Methodology Research Conference, Arlington, VA. Oppenheim, A. N. (1966). Questionnaire design and attitude measurement. London: Heinemann. Oppenheim, A. N. (1992). Questionnaire design, interviewing and attitude measurement (New ed.). London: Pinter. Payne, S. L. (1951). The art of asking questions. Princeton: Princeton University Press. Presser, S., Rothgeb, J. M., Couper, M. P., Lessler, J. T., Martin, J., & Singer, E. (Eds.). (2004). Methods for testing and evaluating survey questionnaires. Hoboken, NJ: Wiley. Redline, C. D., & Lankford, C. P. (2001). Eye-movement analysis: A new tool for evaluating the design of visually administered instruments. In American Statistical Association (Ed.), Proceedings of the Section on Survey Research Methods. Alexandria, VA: American Statistical Association. Sander, J. E., Conrad, F. G., Mullin, P. A., & Herrmann, D. (1992). Cognitive modeling of the survey interview. In American Statistical Association (Ed.), Proceedings of the section on 31
Cognitive models of answering questions Survey Research Methods (pp. 818-823). Washington D.C.: American Statistical Association. Schaeffer, N. C., & Presser, S. (2003). The science of asking questions. Annual Review of Sociology, 29, 65-88. Schuman, H., & Pressesr, S. (1981). Questions and answers in attitude surveys: Experiments of question form, wording and context. San Diego: Academy Press. Schwarz, N. (1990). Assessing frequency reports of mundane behaviors. Contributions of cognitive psychology to questionnaire construction. In C. Hendrick & M. S. Clark (Eds.), Research methods in personality and social Psychology (Review of Personality and Social Psychology) (Vol. 11, pp. 98-119). Newbury Park, CA: Sage. Schwarz, N., & Sudman, S. (Eds.). (1996). Answering questions. methodologies for determining cognitive and communicative processes in survey research. San Francisco: JosseyBass. Simon, H. A. (1957). Models of man. New York: Wiley. Sirken, M. G., & Schechter, S. (1999). Interdisciplinary survey methods research. In M. G. Sirken, D. Herrmann, S. Schechter, N. Schwarz, J. M. Tanur & R. Tourangeau (Eds.), Cognition and survey research (pp. 1-10). New York: Wiley. Strack, F., & Martin, L. L. (1987). Thinking, judging, and communicating; a Process account of context effects in attitude surveys. In H.-J. Hippler, N. Schwarz & S. Sudman (Eds.), Social information processing and survey methodology (pp. 123-148). New York: Springer-Verlag. Strack, F., & Schwarz, N. (1992). Communicative influences in standardized question situations: The case of implicit collaboration. In G. R. Semin & K. Fiedler (Eds.), Language, interaction and social cognition (pp. 173-193). London: Sage.
32
Cognitive models of answering questions Strube, G. (1987). Answering survey questions: The role of memory. In H.-J. Hippler, N. Schwarz & S. Sudman (Eds.), Social information processing and survey methodology (pp. 86-101). New York: Springer-Verlag. Sudman, S., & Bradburn, N. M. (1974). Response effects in surveys: A review and synthesis. Chicago: Aldine. Sudman, S., Bradburn, N. M., & Schwarz, N. (Eds.). (1996). Thinking about answers. The application of cognitive processes to survey methodology. San Francisco: Jossey-Bass. Sudman, S., Willimack, D. K., Nichols, E., & Mesenbourg, T. (2000). Exploratory research at the U.S. Census Bureau on the survey response process in large companies. Paper presented at the Second international conference on establishment surveys, Buffalo (NY). Tanur, J. M. (1999). Looking backwards and forwards at the CASM movement. In M. G. Sirken, D. Herrmann, S. Schechter, N. Schwarz, J. M. Tanur & R. Tourangeau (Eds.), Cognition and survey research (pp. 13-20). New York: Wiley. Thurstone, L. L. (1927). A law of comparative judgment. Psychological Review, 34, 273-286. Tourangeau, R. (1984). Cognitive science and survey methods. In T. B. Jabine, M. L. Straf, J. M. Tanur & R. Tourangeau (Eds.), Cognitive aspects of survey methodology: Building a bridge between disciplines (pp. 73-100). Washington DC: National Academy Press. Tourangeau, R., & McNeeley, M. E. (2003). Measuring crime and crime victimization: Methodological issues. In J. V. Pepper & C. V. Petrie (Eds.), Measurement problems in criminal justice research: Workshop summary (pp. 10-42). Washington D.C.: National Academies Press. Tourangeau, R., & Rasinski, K. (1988). Cognitive processes underlying context effects in attitude measurement. Psychological Bulletin, 103(3), 299-314. Tourangeau, R., Rips, L., & Rasinski, K. (2000). The psychology of survey response. Cambridge: Cambridge University Press. 33
Cognitive models of answering questions Turner, A. G. (1982). What subjects of survey research believe about confidentiality. In J. E. Sieber (Ed.), The ethics of social research: Surveys and experiment. New York: Springer-Verlag. Willimack, D. K., & Nichols, E. (2001). Building an alternative response process model for business surveys. Paper presented at the Annual meeting of the American Association for Public Opinion Research, Montreal, Canada. Willis, G. B. (2004a). Cognitive interview revisited: A useful technique? In S. Presser, J. M. Rothgeb, M. P. Couper, J. T. Lessler, J. Martin & E. Singer (Eds.), Methods for testing and evaluating survey questionnaires (pp. 23-43). Hoboken, NJ: Wiley. Willis, G. B. (2004b). Cognitive interviewing. A tool for improving questionnaire design. Newbury Park, CA: Sage. Willis, G. B., & Bercini, D. (1991). The use of verbal report methods in the development and testing of survey questionnaires. Applied Cognitive Psychology, 5(3), 252-267. Willis, G. B., DeMaio, T. J., & Harris-Kojetin, B. (1999). Is the bandwagon headed to the methodological promised land? Evaluating the validity of cognitive interviewing techniques. In M. G. Sirken, D. Herrmann, S. Schechter, N. Schwarz, J. M. Tanur & R. Tourangeau (Eds.), Cognition and survey research (pp. 133-153). New York: Wiley.
34
Cognitive models of answering questions
Comprehension of the question
Retrieval of information
Judgment end estimation
Reporting an answer
Figure 1. Four step process (Groves et al., 2004, p. 202)
Cognitive state of processed information at time of interview RESPONDENT'S INFORMATION PROCESSING
Transformed and restructured acccording to new input
Integrated in cognitive organization of the moment Experienced by respondent Interaction between stimulus question and respondent's cognitive organization
EVENT
Assumed by researcher
Activation of cognition
Information retrieval or re-cognition
If question is appropriate activator of respondent's cognitive processes Conceptualized as research variable Converted into a verbal stimulus under a question form
RESEARCHER'S DESIGN OF THE QUESTIONNAIRE
Informational state of stimulus question
Figure 2. Model of information processing in the interview (Cannel, Marquis and Laurent, 1977, p. 53).
35
Cognitive models of answering questions
6 R's choice on modification of response based on cues from: a. Interviewer (status, appearance, behavior) b. The question and the preceding questions c. R's own beliefs, values, attitudes, goals
7 R gives inaccurate or incomplete response characterized by a. Conformity bias b. Desirability bias c. Acquiescence bias d. Other inadequacies
2 R's cognitive processing: 1 R's comprehension of question
a. assessment and decisions concerning information needed for accurate answer b. Retrieval of cognitions (attitudes, beliefs, experiences, facts)
3 R's evaluation of response (2c) in terms of its accuracy
4 R's evaluation of response in terms of R's other goals
5 R gives response judged accurate based on adaquate processing
c. Organization of retrieved cognitions and formulation of response on their basis
Figure 3. Diagram of respondent’s (R) question-answering process (Cannel, Miller and Osksenberg, 1981, p. 393)
36
Cognitive models of answering questions
Interpreting the question
Generating an opinion
Recall prior judgment
YES
Judgment previously stored ????
NO
Access relevant information Decide how to use the information "Compute" the judgment
Format the response Edit the response
Figure 4. Model of information processing in a survey situation (Strack and Martin, 1987, p. 125)
Stimulus (“outside”)
Process (“inside”)
Sources of Influence (“process variables”)
AT THE TIME OF THE EXPERIENCE: event --------
(encoding) -------------------------
then existing expectations, etc.
(restructuring of representation)
interfering influences
schemata,
context,
AT TIME OF THE INTERVIEW: question ----
response
(understanding) ------------------
current schemata, context, and expectations
(retrieval & reconstruction) ----
(ditto), and search strategies
(judgment & decision) ----------
strategies, heuristics, and combination rules
(editing) ----------------------------
Intentional self-presentation
Figure 5. Sources of influence for a survey question (Strube, 1987, p. 87).
37
Cognitive models of answering questions
SURVEY QUESTION ASKED ENCODE QUESTION
DECISION / JUDGMENT
RECALL
ESTIMATE
RESPONSE
Figure 6. Cognitive model of the survey response process (Willis, Royston and Bercini, 1991, p. 253, in the public domain)
38
Cognitive models of answering questions
Orient to interview
Interpret meaning
YES
YES
YES
Try direct retrieval?
Willing to answer?
NO
NO
Formulate question or statement phrasing
YES
Make calculations & judgments
Infer search strategies & retrieve information
NO
Ask question or make statement
NO
NO
YES
Information retrieved?
Question understood ?
Feeling of knowing?
Words recognized ?
YES
Try to infer answer?
NO
Retrieve information & formulate inferences
Formulate answer parsing
YES
State answer
Plausible answer?
YES
NO
NO
Figure 7. Diagram of the respondent model of information exchange theory (Jobe and Hermann, 1996, p. 1984, in the public domain.)
4
It is preferred to report the diagram depicted in Jobe and Hermann (1996) instead of the original one (Sander, Conrad, Mullin and Hermann, 1992) because it is more clear.
39
Cognitive models of answering questions Table 1. Components of the QUEST model 1. Question interpretation Parse question into logical forms Identify referents of nouns Compute presuppositions of question Identify question focus Identify question category 2. Access to relevant information sources Specific knowledge structures (experiences, texts) Generic knowledge structures (concepts, scripts, frames) Structurally organized knowledge 3. Pragmatics Goals of questioner and answer Common ground between questioner and answerer Informativeness of answer 4. Convergence to relevant answers Intersection of information from multiple information sources Structural distances from intersecting information Arc search procedures Constrain satisfaction
Source: Graesser, Bommareddy, Swamer and Golding (1996, p. 149).
Table 2. Components of the response process Component
Specific Process
Comprehension Attend to questions and instructions Represent logical form of questions Identify question focus (information sought) Link key terms to relevant concepts Retrieval Generate retrieval strategy and cues Retrieve specific, generic memories Fill in missing details Judgment Assess completeness and relevance of memories Draw inferences based on accessibility Integrate material retrieved Make estimate based on partial retrieval Response Map judgment onto response category Edit response
Source: Tourangeau, Rips and Rasinski (2000, p.8)
40
Cognitive models of answering questions
Strong satisficing
Weak satisficing
Optimizing
Figure 8. Satisficing - optimizing continuum
1) Available Information can be retrieved with minimal effort
2) Accessible Information can be retrieved with effort or prompts
1
T,C
2 Substantive Response
T,C T,C 3 O
4
O 3) Generatable Information is not exactly known but can be estimated
5
T, O
6
Item Nonresponse T, O
4) Inestimable Information is not known/no basis for estimating
7
T
8
Figure 9. Four-state mapping of cognitive states to response outcomes. Beatty and Hermann, (2002, p. 74)
41
Cognitive models of answering questions
Question posed
(a)
Viable interpretation?
no
yes
(b)
Attempt to answer?
no
yes Retrieval from cognitive state
1
4
2 or 3 Communicative intent: disclose response? no (c)
yes
Potential response adequate?
no
Communicative intent: disclose nonresponse? yes (d)
Item nonresponse
Figure 10. Decision model for item nonresponse. Beatty and Hermann, (2002, p. 77)
Table 3. Focus of the models: factual vs. attitude questions Models applicable to factual / autobiographical questions
Models applicable to attitude and opinion questions
Three steps Question answering Four step process Survey memory Encoding plus four steps Quantitative autobiographical question QUEST Flexible processing Information exchange theory Form appraisal Revised four steps
Three steps Question answering Four step process Information processing QUEST Form appraisal Revised four steps
42
X
X
X X
X
X
X
X
X
X
X
X
43
Note: Satisficing and response decision model were excluded from the comparison.
Response, editing
Convergence to rel.
Decision
Judgment
Pragmatics
Goal evaluation
Estimation
X
X
X
X
X
X
X
X
X
QUEST
X
X
X
X
X
X
Flexible processing
X
Informat. exchange
X
X
X
X
X
X
X
Encoding + 4 steps
Reconstruction
X
X
Autobiogra phical
X
X
X
X
Survey memory
X
X
X
Informat. processing
Feeling of knowing
X
X
Four step
Memory retrieval
X
Three steps
X
X
X
Question answering
Interpretation
Comprehension
At the time of the interview
Encoding
At the time of experiences
Stages
Table 4. Comparison of stages across models
Cognitive models of answering questions
X
X
X
X
X
X
Form appraisal
Cognitive models of answering questions Table 5. Methods for determining cognitive processes by task Task Retrieve
Form judgment
X
X
X
X
X
X
Behavioral coding Coders Computer coding
X X
X
Cognitive experts
X
X
X
X
Method
Comprehend / interpret
Verbal protocols thinkalouds Concurrent Retrospective
Response latency methods
X
Sorting
X
Focus groups
X
Experiments
X
Source: Sudman, Bradburn and Schwarz (1996, p. 18)
44
X
Edit answer
