aloud procedure called Articulated Thoughts in Simulated Situations (ATSS). To address ... There were two contrasting conditions for the second presentation:.
Journal of Rational-Emotive & Cognitive-Behavior Therapy Volume 7, Number 3, Fall 1989
FUNCTIONAL CONSISTENCY IN THE FACE OF TOPOGRAPHICAL CHANGE IN ARTICULATED THOUGHTS Kennon Kashima Goddard College Plainfield, VT
Gerald C. Davison University of Southern California Los Angeles
ABSTRACT: Studies of verbal reports have always assumed that thoughts verbalized on a given occasion are not the only ones that could have been expressed. This poses a potential problem for a recently developed thinkaloud procedure called Articulated Thoughts in Simulated Situations (ATSS). To address this issue, comparisons were made between the articulated thoughts of subjects over two presentations of a simulated stressful event in the ATSS procedure. Articulated thoughts were analyzed for similarity across presentations both topographically for surface content and inferentially for the amount of irrationality. While the topographic analysis generally suggested a lack of stability across presentations, the inferential analysis revealed little change. The suggestion is made that incomplete verbal protocols are not problematic for ATSS if the coding strategy taps into the underlying structure presupposed by one's theoretical interest. Interest in understanding the role of cognitive processes in the development and/or maintenance of maladaptive behavior and emotional disorders has led to the use of assessment approaches that rely priKennon Kashima, M.A., is a member of the core faculty in psychology at Goddard College. Gerald, C. Davison, Ph.D., is the Chair and Professor of Psychology, University of Southern California. We extend thanks to Thomas Weller for running the subjects, to Kevin Brown and Sandra Lew for doing the inferential content analysis, and to Sharon Dolezal and David Haaga for conducting the topographical content analysis. Requests for reprints should be sent to Kennon Kashima, Department of Psychology, Goddard College, Plainfield, VT 05667.
131
ci
1989
Human Sciences
Press
132
Journal of Rational-Emotive & Cognitive-Behavior Therapy
marily on verbal reports. While questions concerning the accuracy of verbal reports have been raised (Nisbett & Wilson, 1977), and guidelines to maximize validity suggested (Davison, Robins, & Johnson, 1983; Ericsson & Simon, 1980; Kendall & Holton, 1981; Meichenbaum & Cameron, 1981), little is known about the limitations of verbal reports obtained during cognitive assessments. Of particular concern to the cognitive assessor is the completeness of the verbal protocols. It has always been assumed in studies employing verbal reports that the thoughts a subject verbalizes on a given occasion are not the only ones that could have been articulated. The purpose of the present investigation was to examine this obvious but important reservation about verbal reports. Subjects participated in an investigation employing a thought-sampling approach called Articulated Thoughts During Simulated Situations (ATSS; Davison, Robins, & Johnson, 1983). An audio recording of a conversation serves as a simulated event. Subjects are asked to pretend that the situation is actually happening and that they are part of it. They are told that the experimenter is interested in the kinds of thoughts they are having as the situation unfolds. A brief (15-25 seconds) segment of audiotape is played, followed by a 30-second silence, during which the subjects say what they are thinking. Another segment is played, followed by the the subject's verbal report, and so on. The verbal responses of the subjects are recorded for later content analysis. In the present study, subjects listened and responded to the same simulated situation twice in a single session. The subject was informed after the first presentation that we were interested in learning more about his or her thoughts about the situation. As a consequence, he or she was asked to listen to the tape again. The situation portrayed was a teaching assistant's critical and negative evaluation of the subject's work. The tape was closely similar to one used in previous investigations (Davison, Robins, & Johnson, 1983; Davison, Feldman, & Osborn, 1984) and found to be stressful and directly relevant to college students. There were two contrasting conditions for the second presentation: half of the subjects were simply told of our interest in further understanding their thoughts; the other half were, in addition, asked to articulate different thoughts, ones they might not have verbalized the first time, as well as any new thoughts elicited during the re-presentation. This condition, in which a demand was placed on the subject for different material, was designed to pull from the subject verbalized
S
1
Kennon Kashima and Gerald C. Davison
133
thoughts that were different from those articulated on the first presentation. Two kinds of content analyses were conducted. In the first analysis, the data were coded topographically by comparing the similarity of articulated thoughts across presentations. For example, in response to the third segment of the teaching assistant tape, the subject may have expressed his or her concern about not getting a good grade. Did he or she articulate the same or similar thoughts to that same segment when listening to the tape a second time? Were the first and second set of articulations less similar to each other if the subject was in the demand condition (i.e., told to verbalize different thoughts on the second presentation)? The second content analysis was more inferential and derived from procedures earlier employed in our laboratory (Davison, Feldman, & Osborn, 1984). Almost all cognitive approaches to therapeutic behavior change assume that emotional reactions to environmental events are a function of how the events are construed. Yet the only available evidence for these intermediate processing steps are not the cognitions themselves but a verbalized component. Complex cognitive processes, like irrationality, are inferred from verbal report. Consider the possibility that subjects are found in the first content analysis to verbalize thoughts in either or both conditions that are quite different from each other when coded descriptively. The interesting question now is whether the two sets of articulated thoughts are different as well when coded for complex cognitive processes like irrational thinking. For example, suppose the subjects, at the topographical level, said different things on the second presentation. It may be that if their thoughts were also coded for irrationality, a variable presumed to reflect a cognitive process that mediates psychopathology (Ellis, 1962), there would be no difference. In other words, what appeared dissimilar at the descriptive or topographic level may look very similar at a level that reflects how the subject processes information. This was of interest because one criticism of think-aloud approaches, also the rationale for this study, is that subjects report only a portion of the thoughts that pass through their minds. Of course this is true; but the ATSS paradigm may still pull from the subject ideation that is consistent using a more inferential coding scheme, thus making incomplete verbalizations less problematic. Clinicians (as well as cognitive psychologists) are accustomed to construing their clinets' (subjects') reports in a manner not immediately obvious to the client/ subject (i.e., the person who is describing things topographically). The
134
Journal of Rational-Emotive & Cognitive-Behavior Therapy
present study was designed to examine this question experimentally and at the same time address methodological issues about the ATSS paradigm in particular and think-aloud methods in general.
METHOD
Subjects The subjects were thirty-six undergraduates (eighteen male students and eighteen female students) enrolled in the introductory psychology course at the University of Southern California. Subjects participated in order to receive extra credit and were not preselected on any measures.
Procedure Experimental Session. Students were recruited for an experiment on "The kinds of things people say to themselves. " All were presented first with a general set of instructions, then a description of the situational context, and then the teaching assistant tape itself. After they had articulated their thoughts to each of the seven segments, they were randomly assigned to one of two conditions for the second presentation of the tape. In the "free-response" condition, subjects were asked just to think aloud to the tape again in order to provide us with a better understanding of their thoughts. In the " "deman-forithugs condition, subjects were requested to articulate thoughts different from what they had articulated the first time. After subjects listened and articulated for the second time, they were debriefed, concluding the experimental session.
Coding of Verbalized Thoughts. Each subject provided fourteen response segments, seven to presentation one and seven to presentation two. Raters, blind as to condition, coded the data using two different strategies. With the first method, the data were coded topographically. Raters made judgments of similarity between the first and second presentations by comparing pairs of segments (segment one of presentation one to segment one of presentation two, etc.). The second method required coders to judge the degree of irrationality in the thoughts articulated during both presentations.
Kennon Kashima and Gerald C. Davison
135
Topographical coding The procedure for scoring the topographical similarity of each pair of ATSS segments required two steps. First, the rater made a dichotomous judgment of similar or dissimilar. Second, the rater assigned a numerical score for similarity or dissimilarity. A pair of segments judged as similar received a score of four, five, or six. A pair of segments judged as dissimilar received a score of one, two, or three (the higher scores representing greater similarity). As an aid in arriving at the numerical score, each rater first divided the segments into idea units, similar to the strategy used by Davison, Robins, and Johnson (1983). After all fourteen response-segments (i.e., seven pairs) were rated, a Sum Similarity Score was calculated for each subject by adding together the seven similarity scores. Two psychology graduate students served as independent raters of topographical similarity. They practiced on pilot data to familiarize themselves with the procedure and commenced with the actual similarity coding only after a Pearson product-moment correlation coefficient of .80 had been obtained. The overall reliability for the Sum Similarity Scores for all thirty-six subjects was a correlation coefficient of .82. Inferential coding With the second method, coders rated the degree of irrationality in subjects' articulated thoughts. The strategy employed was closely similar to that used by Davison, Feldman, and Osborn (1984). First, two different coders independently scored each segment for the presence of Ellis' (1962) Irrational Beliefs on a scale from 0 (not present) to 6 (extremely present). Second, a Sum Irrational Beliefs Score was computed for each subject on each stimulus tape by adding together the eleven Irrational Beliefs Scores for all seven segments. Third, an Irrationality Index was calculated based on the ratio of the Sum Irrational Beliefs Score to the number of words the subject verbalized. The Irrationality Index was designed to control for differential word output and possible censoring of thoughts by subjects. Table 1 shows the mean irrationality indices for each condition and presentation. Two psychology undergraduates served as independent raters of irrationality. The practiced on pilot data to familiarize themselves with the procedure and commenced with the actual irrationality coding only after a correlation coefficient of .80 had been obtained. The over-
Table 1 Irrationality Indices Presentation Condition
One
Two
Free Response M
.007
.012
.011
.025
.005
.009
.005
.009
SD
Demand-for-Different-Thoughts M SD
The Irrationality Index equals the Sum Irrational Belief Score divided by the number of words verbalized during a presentation.
all reliability for the Sum Irrational Beliefs Scores for all thiry-six subjects was a correlation coefficient of .78. RESULTS Topographical Analysis
Preliminary analyses revealed no effects of sex, so the sex of the subject was omitted from further analysis. It was predicted that subjects in the demand-for-different-thoughts condition would verbalize thoughts that were less similar topographically over the two presentations than subjects in the free-response condition. This hypothesis was confirmed. Subjects in the demand condition verbalized thoughts that were less similar over the two presentations (M = 19.50, SD = 5.62) than did subjects in the free-response condition (M = 23.75, SD = 5.84, 1(34) = 2.23, p< .05. A closer examination of these data shows that the average similarity score for all pairs of segments in the deman-forithugs condition was 2.78, while the average similarity score for all pairs of segments in the free-response condition was 3.39. These means indicated that thoughts in the free-response manipulation were coded as being generally mixed, both similar and dissimilar, with the demand-for-different-thoughts manipulation resulting in generally dissimilar thoughts.
Kennon Kashima and Gerald C. Davison
137
Inferential Analysis
For the second analysis, Pearson product-moment correlations between presentation one and presentation two for each condition were calculated. Correlation coefficients for Irrationality Indices were highly significant in both the free-response condition (r(18) = .97, p< .001) and in the demand-for-different-thoughts condition (r(18) = .73, p< .001). Thus, within each condition—demand and free-response —subjects' irrationality as content analyzed was similar across the two presentations of the stimulus tape. This occurred in spite of instructions for different (i.e., dissimilar) thoughts in the demand condition, which to be sure, lowered the variance accounted for from 94% in the free-response condition to 53% in the demand condition.
DISCUSSION This study compared the articulated thoughts of subjects over two presentations of a simulated stressful event on a single occasion. For the second presentation, half of the subjects were simply told of our interest in further understanding their thoughts; the other half were, in addition, asked to verbalize different thoughts. Articulated thoughts were analyzed both for topographical similarity across presentations and for the amount of inferred irrationality within each presentation. At the topographical level, subjects' articulated thoughts were more similar across presentations when they were told only to listen and react to the stimulus tape a second time than when subjects were exhorted to come up with different thoughts during the second trial. Furthermore, for both conditions, the actual means on the similarity-dissimilarity scale used by the coders suggest that, on average, the coders regarded more pairs of response-segments to be dissimilar than similar. This finding of relative dissimilarity is of particular interest when compared with results from the inferential analysis,' whereby coders rated the degree of irrationality in subjects' articulated thoughts. This functional analysis indicated a high degree of similarity across the two 'Ideally an index of association would have been used for both topographical similarity and inferred irrationality. Unfortunately, there is no reliable and comprehensive coding strategy for verbal protocols that would capture the meaning of topographical similarity and allow for a measure of association as comparison. Thus, we were faced with comparing two different kinds of analyses.
138
Journal of Rational-Emotive & Cognitive-Behavior Therapy
trials, especially when subjects were asked only to repeat the trial (the free-response condition). Considered together, these two sets of findings address the principal question of interest, namely, how reliable articulated thoughts are in a given situation. The topographical analysis alone suggests that perhaps we should be cautious—subjects, even when not asked to express something different a second time, do so to a degree. However, when the data are coded in a manner employed in previous research conducted with the ATSS paradigm—to be specific, inferring the degree of irrationality—there is less change from trial 1 to trial 2, suggesting that the thoughts articulated represent a good sample of what the subject is thinking about with respect to inferred irrationality. The present investigation suggests that the failure of subjects to verbalize all their thoughts is less problematic than assumed. Incomplete verbalizations may be reliable even when subjects are requested to articulate different thoughts. While there is change at the topographical level, there may be greater stability at a more theoretical level—where cognition is presumed to systematically covary with other aspects of functioning.` Of course, only inferred cognition can he studied, and the degree of stability observed will vary as a function of the particular coding strategy employed and the population investigated. The study reported here was concerned with an analogue of irrational thinking as conceptualized in rational-emotive therapy. Other ATSS investigations in our laboratory employ different coding strategies to investigate particular cognitive processes and structures that may guide information-processing in specific disorders. The method of comparison used in the present experiment—topographic versus inferential content analysis of thoughts articulated to the same stimulus-tape over two successive presentations—appears capable of assessing how much things appear to change yet remain the same. REFERENCES P. M., & Osborn, C. C. (1984). Articulated thoughts, irrational beliefs, and fear of negative evaluation. Cognitive Therapy and Research, 8, 349-362.
Davison, G. C., Feldman,
2
While some evidence has been provided for the systematic covariation of incomplete thoughts that may represent underlying cognitive processes, the present investigation cannot tell us what subjects actually omit and the relationship and/or significance of their omissions.
Kennon Kashima and Gerald C. Davison
139
Davison, G. C., Robins, C., & Johnson, M. K. (1983). Articulated thoughts during simulated situations: A paradigm for studying cognition in emotion and behavior. Cognitive Therapy and Research, 7, 17-40. Ellis, A. (1962). Reason and emotion in psychotherapy. Secaucus, NJ: Lyle Stuart. Ericsson, K. A., & Simon, H. A. (1980). Verbal reports as data. Psychological Review, 87, 215-251. Kendall, P. C., & Hollon, S. D. (1981). Assessing self-referent speech: Methods in the measurement of self-statements. In P. C. Kendall & S. D. Hollon (Eds.), Assessment strategies for cognitive behavioral interventions (pp. 85-118). New York: Academic Press. Meichenbaum, D., & Cameron, R. (1981). Issues in cognitive assessment: An overview. In Merluzzi, T. V., Glass, C. R., & Genest, M. (Eds.), Cognitive Assessment (pp. 3-15). New York: Guilford Press. Nisbett, R. E., & Wilson, T. D. (1977). Telling more than we can know: Verbal reports on mental processes. Psychological Review, 84, 231-259.
