The Variables Problem and Progress in ... - Semantic Scholar

Psychotherapy 2013, Vol. 50, No. 1, 33– 41

© 2013 American Psychological Association 0033-3204/13/$12.00 DOI: 10.1037/a0030569

The Variables Problem and Progress in Psychotherapy Research William B. Stiles Glendale Springs, North Carolina In this journal’s first article, Strupp (1963) pointed to problems specifying independent and dependent variables as a source of slow progress in psychotherapy outcome research. This commentary agrees, shows how the concept of variable loses its meaning in psychotherapy research because of participants’ responsiveness, and notes an alternative research strategy that does not depend on variables. Keywords: psychotherapy, responsiveness, variables problem, theory-building qualitative research

In his reply to Strupp (1963), Eysenck (1964) asserted, as he had in his earlier review (Eysenck, 1952), “that no data existed disproving the null hypothesis [that psychotherapy fails to facilitate recovery from neurotic disorder] scientifically. There is only one way to answer such an argument and that is to point to an experimental study or investigation conclusively disproving the null hypothesis. Strupp . . . fails to adduce a single study disproving my original conclusion” (Eysenck, 1964, p. 97). Strupp did not adduce such a study in his rejoinder either (Strupp, 1964). Fifty years on, there have been enough supportive studies, reviews, and meta-analyses that psychotherapy’s effectiveness is widely, if not universally, accepted (American Psychological Association, 2012). Perhaps even Eysenck would have accepted some of the published RCTs as experimental studies disproving his 1952 conclusion. In my opinion, however, the problems of the independent and dependent variables that Strupp (1963) identified continue to interfere hugely with scientific progress in psychotherapy research. By scientific progress, I mean improving explanatory theories of how psychotherapy works, making them more general, more precise, and more realistic. I think this is what Strupp had in mind in writing “more pressing matters must be dealt with first before we can address ourselves meaningfully to the question of the effectiveness of psychotherapy.” In the following pages, I (a) describe the phenomenon of responsiveness and how it contributes to the variables problem, (b) review some results of research in which I have been involved and which, I think, illustrates some manifestations of the variables problem while nevertheless supporting the effectiveness of psychotherapy, and (c) point to an alternative approach to psychotherapy research that could, I think, facilitate scientific progress.

Hans Strupp (1963) began this journal’s first article by pointing to “a realization on the part of researchers that a new approach to the [outcome] issue must be found and that more pressing matters must be dealt with first before we can address ourselves meaningfully to the question of the effectiveness of psychotherapy” (p. 1). Research sophistication, instrumentation, and the psychotherapy research literature have grown enormously in the intervening 50 years, but I think this remains an apt characterization. Strupp devoted the first half of his article to analyzing the problem of variables in psychotherapy outcome research: specifying what is meant by treatment and outcome, the independent variable and the dependent variable, respectively. The logic of the experiment is that if all previous conditions except one (the independent variable) are held constant (controlled), any differences in the outcome (the dependent variable) must have been caused by the one condition that varied. For example, if one client is given psychoanalysis and another identical client is not, but is treated identically in all other respects, any differences in their outcomes must have been caused by the psychoanalysis. Difficulties arise because no two people are identical and because it is impossible to treat two people identically in all respects except one. Randomized controlled trials (RCTs) are an adaptation of the experimental method that attempts to address these uncontrolled variations statistically. Rather than comparing individual patients, investigators using RCTs randomly assign patients to groups that receive the different levels of the variable— different treatments— on the assumption that any previous differences that might affect the outcomes will be more or less evenly distributed across the groups. Although individuals’ outcomes might vary within groups (because clients are not identical), any mean differences between groups beyond those due to chance should be attributable to the different treatments. Insofar as the controlled experiment is the closest science has come to a means for demonstrating causality, the effort put into RCTs seems worthwhile despite the difficulties.

Responsiveness and Independent Variables Responsiveness refers to behavior being influenced by emerging context (Stiles, Honos-Webb, & Surko, 1998). People respond to what happens around them, such as what other people do. Human behavior is responsive on time scales that range from months to milliseconds. Paying attention and being polite are responsive. If I answer your questions or repeat something when you look puzzled, I am being responsive. Psychotherapists’ skills and goals involve responsiveness. What therapists do depends on what clients do. Examples include assigning clients to treatments based on their presenting problems, planning their treatment based on how they are progressing,

I thank Tim Carey, Mikael Leiman, and Katerine Osatuke for comments on a draft of this article. Correspondence concerning this article should be addressed to William B. Stiles, P.O. Box 27, Glendale Springs, NC 28629. E-mail: stileswb@ muohio.edu 33

34

STILES

active listening, timing, staying on topic, turn-taking, attunement, adjusting interventions already in progress, and so forth. Responsiveness is a neutral term, and responsiveness is not necessarily benign. However, therapists and clients generally have benign goals, and they respond to advance those goals in ways consistent with their theoretical and personal principles. This can be called appropriate responsiveness. So, appropriate responsiveness means do the right thing. In general, therapists and clients try to do the right thing. Strupp (1963) wrote, “One of the major difficulties in psychotherapy research is that of adequately specifying the independent variable—the psychotherapeutic methods—to which therapeutic changes are being attributed” (p. 1). To illustrate, he drew from a list of potentially impactful characteristics by Knight (1941), which included the sort of influence attempted (e.g., suggestion, persuasion, exhortation, intimidation, counseling, interpretation, reeducation, retraining), the general aim (e.g., supportive, suppressive, expressive, cathartic, ventilative), the depth of the work, treatment duration, the theoretical approach, and the variant within that approach. Such lists highlight but nevertheless understate the problem. The pervasiveness of human responsiveness implies that clients in the same experimental condition of an outcome study each receives a different individually tailored treatment. Such variability impairs any study’s conclusions because the treatment names, such as psychoanalysis or cognitive– behavioral therapy (CBT) or treatment as usual, have no stable meaning. Named treatments vary not just from study to study, but from therapist to therapist, from client to client, from session to session, and from minute to minute. In the 1980s, in response to concerns about this lack of standardization, outcome researchers began using manuals to describe treatments, an innovation described as “a small revolution” (Luborsky & DeRubeis, 1984). However, manuals do not provide rigid instructions. Instead, they describe repertoires of interventions and the sorts of situations in which the interventions might be used. They emphasize building rapport, appropriate clinical judgment, timing, tact, and adapting the approach to what clients present. In other words, they prescribe appropriate therapist responsiveness within the theoretical approach. Likewise, clients are not passive recipients but active participants charged with making sense of psychotherapeutic activities and adapting what they gain to the context of their own lives. I see great value in the sort of fine-grained description of a treatment approach manuals provide, but they do not overcome responsive individualization of treatments. The standardization implied by citing manuals to define levels of an experimental independent variable is at best illusory and at worst deceptive. Of course, no two things in the universe are exactly alike; even two tablets of an antidepressant medication have differences. But the dissimilarity of two sessions of emotion-focused therapy is, I suggest, orders of magnitude greater than the dissimilarity of two fluoxetine tablets. Moreover, whereas the differences among tablets are more or less randomly distributed, the tailoring is responsive to the requirements of the client and the context. We deceive ourselves when we suppose that two courses of manualized emotion-focused therapy, interpersonal therapy, or cognitive therapy are alike in the same way as two tablets of fluoxetine, bupropion, or sertraline. The individualization achieved through responsiveness is clinically and humanly appropriate (a truly standardized script for

interventions would be clinically absurd and probably unethical), but it is disastrous for the logic of experiments. Responsiveness implies that the independent variable in outcome research is causally entangled with client, therapist, and context variables (Elkin, 1999; Krause & Lutz, 2009), so its meaning changes systematically across clients and contexts. For example, clients with overinvolved attachment styles are treated systematically differently than clients with underinvolved attachment styles, even when treated by the same therapists in the same experimental condition (Hardy, Stiles, Barkham, & Startup, 1998). Even worse, the independent variable is causally entangled with dependent variables. Therapists responsively adjust their treatment in light of client progress or lack of progress. Such causal entanglements are more serious than uncontrolled variability in extraneous conditions because they cannot be overcome by randomization (Stiles, 2009b).

Evaluation and Dependent Variables To introduce the problems with dependent variables, Strupp (1963) cited “an insightful and lucid article” by Holt (1958) regarding a “hidden trick” in global judgments of outcome. Such judgments may seem satisfying, but they are scientifically empty. “As long as one relies on global clinical judgments, like outcome,” Strupp summarized, “one substitutes something for real information” (p. 5). Global judgments are problematic when they are drawn from symptom intensity inventories as well as when they are drawn from clinicians’ impressions. Global indexes of severity or distress, total scores, summary scores, and the like similarly substitute evaluation for specific information. Evaluative measures are robust and popular, I think, because evaluation is universal, a common denominator in people’s diverse understandings. Rogers (1959) described this as the organismic valuing process. Therapists, observers, clients, and readers have firm convictions about what they think is good, how they feel, and what they like. Readers want to know how well a therapy works, so research using evaluative measures has a receptive audience. But such evaluations can short-circuit understanding. A focus on evaluating a treatment’s outcome avoids the much more difficult theoretical accounting for descriptive results. Condensing therapeutic effects and therapeutic relationships into global evaluative dimensions does violence to the theoretically specific conceptualizations that scientific research is meant to test (Stiles, 2006, 2009a). Even measures that nominally assess specific conditions, such as depression, largely reflect global distress and are at least sometimes closely correlated with more global measures (e.g., Leach et al., 2006). Unless all respondents understand and use terms and concepts in the same way, they do not emerge as common components in rating scales. This is particularly salient in the case of client self-report indexes; clients are unlikely to have a common understanding of theoretical concepts, so most of the commonality in their responses depends on the evaluative component in the items’ meaning. As a result, most internally consistent self-report measures of process and outcome are primarily evaluative. Inventories that code specific symptoms and behaviors, applied by experts trained to distinguish them, can avoid this problem. In principle, such data can be used to investigate specific changes in individuals. Much of the improvement is undone, however, when the ratings are aggregated into global indexes to compare across

THE VARIABLES PROBLEM

clients or to address questions posed in the evaluative terms of outcome or the value of a treatment. Most process or relationship variables on the list of “evidencebased” “effective elements of therapeutic relationships” assembled by Norcross (2002, 2011) are evaluative: alliance, group cohesion, empathy, goal consensus and collaboration, positive regard, congruence/genuineness, repair of ruptures, management of countertransference, and quality of relational interpretations. These elements represent achievements or desired results rather than specific conditions or volitional behaviors (Stiles & Wolfe, 2006). Such evaluative process variables incorporate appropriate responsiveness. They reflect whether the behavior is appropriate to the circumstances rather than describing specific behaviors. The actual attitudes and behaviors that yield high ratings on alliance, group cohesion, empathy, and so forth, differ across cases and times, as therapists do the right thing in response to clients’ emerging needs and circumstances. It is plausible that such achievements predict global outcome, but they do not show which specific activities predict which of therapy’s specific effects. In summary, the parties involved in psychotherapy seek good outcomes, and all treatments trade on their good intentions. Treatments work, at least in part, because the participants do their responsive best to ensure they work. Research that tests the effects of responsive treatments on global evaluations yields conclusions that may be paraphrased: if participants do their best, things usually turn out well. Reassuring, perhaps, but scientifically disappointing. The limitations of evaluative dependent variables are serious, but not so insuperable as the contamination of independent variables by responsiveness, in my opinion. Even if their potential scientific yield is limited, evaluative dependent variables can serve useful administrative, clinical, and consumeristic purposes (and there are of course many nonevaluative measures available to investigators). The effect of responsiveness on psychotherapy research’s independent variables has been insidious, however, as addressed empirically in the next section.

Treatment Characteristics and Outcome: Some Puzzling Null Findings In this section, I summarize three puzzling but replicated results relating outcome to treatment characteristics like those listed by Knight (1941) and cited by Strupp (1963): theoretical approach, treatment duration, and verbal technique. First, outcomes were positive but were not statistically related to treatment approach in routine practice (Stiles, Barkham, Mellor-Clark, & Connell, 2008a). This is often called the Dodo verdict, named after an extinct flightless pigeon that lived on the island of Mauritius. Second, outcomes were not statistically related to treatment duration in routine practice (Stiles, Barkham, Connell, & Mellor-Clark, 2008). Third, outcomes were not statistically related to verbal techniques (Stiles & Shapiro, 1994). Others have reported similar null results, but investigators seem not to believe them and keep repeating the studies. I think all three can be understood as manifestations of responsiveness. The first two studies were based on a samples drawn from large database of clients assessed for therapy in the British National Health Service. In the database of 33,587 adult clients and in the samples, approximately 70% were women, and mean

35

age was approximately 38 years. Outcomes were assessed using Clinical Outcomes in Routine Evaluation—Outcome Measure (CORE-OM; Barkham et al., 2001; Barkham, Gilbert, Connell, Marshall, & Twigg, 2005; Evans et al., 2002). The CORE-OM includes 34 items that are scored on a scale of 0 to 4; scores were calculated as the mean of all items ⫻ 10 and so could range from 0 to 40. We also drew on the CORE Assessment form, completed by the therapist after intake, which records referral information, client demographics, and information about the nature, severity, and duration of presenting problems, and the CORE End of Therapy form, completed by therapist at the end of treatment, which records which theoretical approaches were used, the number of sessions attended, and whether end was planned, among other things. These data were gathered for clinical and administrative reasons and later made available for research.

Treatment Approach and Outcome To compare the outcomes of different treatment approaches (Stiles, Barkham, Mellor-Clark, & Connell, 2008a, replicating Stiles, Barkham, Twigg, Mellor-Clark, & Cooper, 2006), we selected clients who had completed the CORE-OM at beginning and end of treatment. On the CORE End of Treatment form, therapists indicated which of the following treatment approaches they had used: psychodynamic, psychoanalytic, cognitive, behavioral, cognitive/behavioral, structured/brief, person-centered, integrative, systemic, supportive, art, and other. Many therapists indicated more than one approach. We targeted three families of approaches: (a) CBT, if the therapist indicated any combination of cognitive, behavioral, or cognitive/behavioral; (b) person-centered therapy (PCT), if the therapist indicated person-centered; and (c) psychodynamic therapy (PDT), if the therapist indicated psychodynamic or psychoanalytic. Based on these definitions, we constructed six groups. Three groups received pure treatments, that is, one and only one of the targeted approaches. The other three groups received one of the targeted approaches plus one additional treatment, such as systemic, supportive, or art therapy, abbreviated CBT ⫹ 1, PCT ⫹ 1, and PDT ⫹ 1, respectively. These treatments might be considered as diluted or enhanced. Clients who received any other combination of treatments were not included. Of the 33,587 clients in the database, 5,613 had complete data and met specifications for one of these six groups. The six groups’ pretreatment CORE-OM means were not significantly different from each other, despite the high statistical power. Figure 1 shows the similar distribution of pre–post change scores in the three groups. A repeated-measures analysis of variance, with treatment approach and degree of purity (pure vs. “⫹1”), showed the following: First, all three treatments were effective, shown by large withinclients main effect of treatment (pre–post change; F(1, 5607) ⫽ 6805.63, p ⬍ .001, partial ␩2 ⫽ .548). Most of the clients improved; on average, clients changed by 8.77 points on the CORE-OM. This is comparable with pre–post change in RCTs (Barkham et al., 2008). The statistical effect size was large (1.39). Second, CBT, PCT, and PDT were equally effective. The differential treatment effect (Treatment ⫻ Pre–post interaction) was not significant (F(2, 5607) ⫽ 0.81, p ⫽ .446, partial ␩2 ⬍ .001). That is, there was no significant difference among the three tar-

36

STILES

Figure 1. Distributions of Clinical Outcomes in Routine Evaluation—Outcome Measure (CORE-OM) pre– post change score in six treatment groups (from Stiles, Barkham, Mellor-Clark, & Connell, 2008a). In these notched box plots, the dot in the middle represents the median. The notch represents the 95% confidence interval. The box represents the middle 50% of the distribution. The whiskers represent the range. The marks above and below the whiskers represent outliers. CBT ⫽ cognitive-behavioral therapy; PCT ⫽ person-centered therapy; PDT ⫽ psychodynamic therapy. CBT ⫹ 1 ⫽ CBT combined with one other therapy; PCT ⫹ 1 ⫽ PCT combined with one other therapy; PDT ⫹ 1 ⫽ PDT combined with one other therapy. Adapted from “Effectiveness of Cognitive-Behavioural, Person-Centred, and Psychodynamic Therapies in UK Primary Care Routine Practice: Replication in a Larger Sample,” by W. B. Stiles, M. Barkham, J. Mellor-Clark, & J. Connell, 2008, Psychological Medicine, 38, p. 681. Cambridge University Press.

geted treatment approaches, despite the high statistical power of this test. As the Dodo said, “Everybody has won, and all must have prizes” (Rosenzweig, 1936, p. 412, quoting Carroll, 1865/1946). As early as 1936, it was apparent to observers: “If such theoretically conflicting procedures . . . can lead to success, often in similar cases, then therapeutic result is not a reliable guide to the validity of theory” (Rosenzweig, 1936, p. 412). Third, dilution of treatments did not impair effectiveness. The mixed therapies did as well as the pure ones. The treatment purity (pure vs. “⫹1”) was not significant (Purity ⫻ Pre–post interaction; F(1, 5607) ⫽ 3.23, p ⫽ .073, partial ␩2 ⫽ .001). The three-way interaction was not significant. Caveats include limited specification of treatments, nonrandom assignment of clients, absence of a control group, incomplete data, restriction to one self-report measure, and investigator allegiance. These objections and others (e.g., Clark, Fairburn, & Wessely, 2008) have been addressed elsewhere (Stiles, Barkham, Mellor-Clark, & Connell, 2008a, 2008b). Caution is warranted; nevertheless, if one of these treatments were really substantially better than the others, I suggest some difference should have emerged in our results. The similar distributions may reflect responsiveness within levels of the treatment variable. Different treatments may be equivalently effective because therapists are appropriately responsive to client requirements within each framework. All approaches ask therapists to be appropriately responsive as they work toward treatment goals. As a result, each treatment is responsively adjusted to each client’s changing requirements. To the extent this

succeeds, clients have optimal outcomes. This responsiveness account presumes that each approach provides ways to respond appropriately to varied client requirements.

Treatment Duration and Outcome The dose– effect model (e.g., Howard, Kopta, Krause, & Orlinsky, 1986: Kopta, Howard, Lowry, & Beutler, 1994) tries to ascertain the optimum dose of therapy, where dose is measured as the number of sessions. More sessions represent a stronger dose. We investigated the relation of treatment dose to outcome in routine practice (Stiles, Barkham, Connell, & Mellor-Clark, 2008, replicating Barkham et al., 2006). Of 33,587 clients in the database, 9,703 had complete data, planned endings as reported by the therapist, completed 20 or fewer sessions, and began with a CORE-OM score of 10 or higher, the clinical cutoff on this instrument (Connell et al., 2007). To assess the effect of different doses, we used two indexes: rate of reliable improvement and rate of reliable and clinically significant improvement (RCSI) among clients who received each dose (N ⫽ 0 –20 sessions). Following Jacobson and Truax (1991), we considered clients as reliably improved if they improved to a degree that was probably not due to chance, based on a 95% confidence interval. For this study, we considered changes larger than 4.5 points as reliable (the reliable change index). Jacobson and Truax argued further that a change could be considered as clinically significant if the client entered treatment in a dysfunctional state

THE VARIABLES PROBLEM

and left in a normal state, interpreted as moving from above to below the cutoff between the dysfunctional and normal populations. Thus, we considered clients as having achieved RCSI if their CORE-OM score changed by at least 4.5 points and went from 10 or above to below 10. Figure 2 shows the percentage of clients who made reliable improvement as a function of how many sessions they received. That is, they changed by 4.5 points or more on the CORE-OM. Approximately 80% of clients improved reliably regardless of how many sessions they had; outcomes did not improve with larger doses. The percentages were more variable for the longer treatments because fewer clients had longer treatments, and the rates were less stable. Figure 3 shows that the RCSI rate was slightly negatively correlated with number of sessions. This is partly because clients who received longer treatments had slightly higher initial scores, so although they averaged as much improvement as clients with fewer sessions, more ended above the cutoff. Finding improvement unrelated to treatment dose may seem paradoxical and surprising if treatment is considered as an experimental manipulation, but it is clinically sensible if clients and therapists are considered as responsively ending treatment when a good-enough level has been reached (Baldwin, Berkeljon, Atkins, Olsen, & Nielsen, 2009; Barkham et al., 2006; Stiles, Barkham, Connell, & Mellor-Clark, 2008). Clients change at different rates and achieve a good-enough level of gains at different treatment durations. When they have had enough, they stop. That is, participants responsively regulate treatment duration to meet client requirements. Finding that RCSI rates were lower in longer treatments suggests the good-enough level is influenced by costs. Longer treatments cost more—in money, time, time off work, finding a babysitter, and so forth. Clients are satisfied with less as more is demanded.

Verbal Technique and Outcome The drug metaphor approach to assessing process– outcome relationships (Stiles & Shapiro, 1989) suggests that the verbal and nonverbal components of psychotherapy can be treated like the ingredients of pharmacological agents in evaluating their strength, integrity, and effectiveness (cf. Yeaton & Sechrest, 1981). If a process component is an active ingredient, then administering a high level of it should yield a positive outcome. If it has no effect, the process component is presumed to be inert. Experimental studies of individual process ingredients would be prohibitively

Figure 2. Percentage of clients reliably improved as a function of treatment duration.

37

Figure 3. Reliable and clinically significant improvement (RCSI) rate as a function of treatment duration.

expensive, so most process– outcome studies have assessed naturally occurring covariation of process and outcome variables (Orlinsky, Grawe, & Parks, 1994). If a process component is an active ingredient, clients who receive relatively more of it should tend to improve more, so measures of this component should be positively correlated with measures of outcome across clients. We (Stiles & Shapiro, 1994) applied this logic to data from the First Sheffield Psychotherapy Project (Shapiro & Firth, 1987). This was a comparative trial of two treatments for depression: psychodynamic–interpersonal (PI) and cognitive– behavioral (CB). The study had an unusual cross-over design in which clients received eight sessions of each treatment with the same therapist throughout. Order of treatments was counterbalanced across clients. After an initial assessment, clients received eight sessions of one treatment, then a mid-treatment assessment, then eight sessions of the other treatment, and then an end-oftreatment assessment. The assessment battery included the Present State Examination (Wing, Cooper, & Sartorius, 1974), which is an interview-based rating schedule done by a trained assessor, and two standard self-report measures: the Beck Depression Inventory (Beck, Ward, Mendelson, Mock, & Erbaugh, 1961) and the Symptom Checklist-90 (Derogatis, Lipman, & Covi, 1973). Clients in both groups showed substantial improvement on these measures, and the mean differences between treatments were negligible after the counterbalancing was dealt with statistically (Shapiro & Firth, 1987). So this was another Dodo result, but that is not the point I want to make with this study. We coded the verbal response mode (Stiles, 1992) of every therapist and client utterance in half of the sessions in the project (eight sessions per client, four from each treatment), totaling more than 350,000 utterances (Stiles, Shapiro, & Firth-Cozens, 1988). Verbal response modes are speech acts; they describe what people do when they speak rather than the content of what they say. For this analysis, we selected four theoretically important categories of therapist interventions and one important client category. The therapist categories were Questions, which are informationseeking utterances; General Advisements, which are directives guiding client behavior outside sessions (distinguished from Process Advisements, which direct in-session behaviors); Interpretations, which include any explaining or labeling of client thoughts or actions; and Exploratory Reflections, which express client’s meanings and feelings from the client’s perspective (distinguished from Simple Reflections, which are repetitions). Disclosures, the client category, reveal the client’s private experience (see Stiles, 1992).

STILES

38

Table 1 shows the percentages of therapist or client utterances. These categories accounted for a substantial proportion of what happened in the session, especially insofar as many of the other utterances were Acknowledgments, like “mm-hm” (Stiles et al., 1988). Questions and General Advisements were much more common in the CB than in the PI treatment, whereas Interpretation and Exploratory Reflections were much more common in the PI than in the CB treatment, although the same therapists conducted both treatments. These patterns conform to the theories of how these respective treatments should be conducted. Client Disclosure was a bit more common in PI than in CB treatment, but it constituted a large proportion of client utterances in both treatments. You might expect that such common theoretically important categories would be correlated with improvement across treatment on the Present State Examination, Beck Depression Inventory, and Symptom Checklist-90. Table 2, however, shows the process– outcome correlations were negligible. These correlations encompassed the whole treatment, but separate analyses examining use of these verbal response modes in PI and CB treatments separately yielded the same null result (Stiles & Shapiro, 1994). This does not appear to reflect a failure of measurement. The verbal process components were reliably coded. They represented common theoretically important techniques. They discriminated between treatments in sensible expected ways. The outcome measures were standard, and they detected large clinically and statistically significant changes in these treatments. These null correlations replicate generally disappointing inconsistent yield of process– outcome comparisons, at least for process measures other than evaluative ones (cf. Orlinsky et al., 1994). As discussed earlier, evaluative process variables, such as empathy and the strength of the alliance, which incorporate responsiveness, do show consistent relations with outcome measures. Do the null results shown in Table 2 imply that these major therapeutic techniques are inert ingredients? I think not. The correlational model basically tests whether more is better, ignoring clients’ differing requirements and participants’ responsiveness to those requirements. In effect, the correlational model makes the absurd assumption that process components are delivered randomly with respect to client requirements. That is, it implicitly assumes that what clients need is independent of what they get. No therapy is like that. Appropriate responsiveness tends to defeat the process– outcome model; it is not just statistical noise. If the use of a verbal component varied optimally with client requirements, all clients’ outcomes would be equivalent, or at least unrelated to that process component (Stiles, 1988; Stiles et al., 1998). For example, even if interpretations were the crucial ingredient, it would not help to give clients more than they need.

Table 2 Correlations of Verbal Response Mode Percentages With Change From Intake to Termination on Outcome Measures Therapist VRM

Client Outcome General Exploratory VRM measure Question advisement Interpretation reflection Disclosure PSE BDI SCL-90

.06 .06 .21

.01 ⫺.05 .09

⫺.11 .08 ⫺.07

.11 .18 .05

.02 .23 .07

Note. N ⫽ 39 clients. VRM ⫽ verbal response mode; PSE ⫽ Present State Examination (Wing, Cooper, & Sartorius, 1974); BDI ⫽ Beck Depression Inventory (Beck, Ward, Mendelson, Mock, & Erbaugh, 1961); SCL-90 ⫽ Symptom Checklist-90 (Derogatis, Lipman, & Covi, 1973).

When responsiveness is substantial but imperfect, the correlation of crucial process components with outcomes may be misleadingly positive, null, or even negative. Suppose, hypothetically, that interpretations are an important active ingredient in psychotherapy. Clients vary in their requirements for interpretations for many reasons. Some may be motivated and receptive to therapists’ interventions; they catch on quickly and apply what they learn. Others may be resistant or slow, requiring many repetitions and explanations. Appropriately responsive therapists adapt their interventions to these variations. They give relatively few interpretations to clients who catch on quickly and relatively more to clients who require repetition or rephrasing. If therapists are good but imperfect, the quicker easier clients may have better outcomes, whereas the slower more difficult ones may have poorer outcomes, despite the therapists’ best efforts. As depicted in Figure 4, this can yield a negative correlation between interpretations and outcome, although, by assumption, interpretations are an important active ingredient in the therapy. Such seemingly paradoxical results have been reported in some studies (e.g., Sloane, Staples, Cristol, Yorkston, & Whipple, 1975). Readers and reviewers could be tempted to conclude incorrectly that interpretations are useless or even harmful.

How Can We Do Psychotherapy Research Without Variables? Let us back up a step. The central purpose of science, I submit, is to develop a coherent and accurate understanding of how the world works, that is, to construct general, precise, realistic theories. A theory is a semiotic representation of the world—a description in words, numbers, diagrams, or other signs. Among other things, a good theory is a practical guide for decisions about how to act in the world, for

Table 1 Mean Percentages of Therapist Utterances and Client Utterances in 312 Sessions of Psychodynamic–Interpersonal and Cognitive–Behavioral Therapy (Eight Sessions per Client, Which Was 50% of Each Treatment) Therapists Treatment

Question

General advisement

Interpretation

Exploratory reflection

Clients Disclosure

Psychodynamic–interpersonal Cognitive–behavioral

5.7 10.1ⴱ

0.8 7.0ⴱ

20.3ⴱ 14.3

8.3ⴱ 1.9

42.9ⴱ 36.6

Note. N ⫽ 39 clients. Percentage larger than in the other treatment, p ⬍ .001.

ⴱ

THE VARIABLES PROBLEM

Figure 4. Hypothetical result of using interpretations responsively. A ⫽ Clients who are highly motivated or who understand quickly; B ⫽ Clients who are resistant or slow and need many repetitions.

example, how to practice psychotherapy. The purpose of empirical scientific research is quality control— checking that the theory is a good one by comparing it with observations. The observations change the theory by increasing or decreasing confidence or more often by showing where modifications, elaborations, or extensions are needed (Lakatos, 1978; Stiles, 1981, 2009a; cf. Miller, 2009). One strategy for systematically bringing observations to bear on a theory is hypothesis testing. Its logic is the hypothetico-deductive method. The familiar statistical version of this is to deduce one or a few statements from a theory and compare each such statement with many observations. That is, we test a hypothesis by seeing whether it holds across cases. This may involve experiments (e.g., RCTs) but also naturalistic comparisons and correlational studies. If the observations tend to correspond to the statement, our confidence in that statement is substantially increased. We say the hypothesis was confirmed, or at least the null hypothesis was rejected. This yields a small increment or decrement of confidence in the theory as a whole, or information about where the theory needs modifications. Theory-building qualitative research offers an alternative strategy (Stiles, 2009a, 2010). In a theory-building psychotherapy case study, investigators compare each of many theoretically based statements with one or a few observations. At issue is the correspondence of theory and observation— how well the theory describes details of the case. Like variables in statistical hypothesis-testing studies, terms in theoretical descriptions have explicit links within the theory, but unlike variables, descriptions need to be applicable to only one or a few cases to be scientifically useful. Each detail may be observed only once, but many details are observed across the course of treatment. For familiar reasons, such as selective sampling, low power, investigator biases, and so forth, the change in confidence in any one statement may be small. However, because many theoretical statements are compared with case observations, the

39

gain in confidence in the theory may be as large as from a statistical hypothesis-testing study. A collection of systematically analyzed cases that match a theory in precise detail may give people a good deal of confidence in the theory as a whole, although each component assertion may remain tentative and uncertain when considered in isolation. The key is making many detailed theory-relevant observations of the case. Campbell (1979) described these multiple observations as analogous to multiple degrees of freedom in a statistical hypothesis-testing study. Whereas statistical hypothesis testing requires theoretically framed independent and dependent variables that can be observed in every case, theory-building case studies can use any case observations that can be described in theoretical terms. As Strupp (1963) pointed out and I have elaborated here, both independent and dependent variables are highly problematic in psychotherapy research, contaminated by context, responsiveness, and global judgments. In contrast, major theories of psychotherapy—psychodynamic, CB, humanistic—apply across diverse contexts, across diverse client characteristics, and across therapeutic relationships that follow responsive nonlinear courses. By comparing the rich theoretical descriptions with rich case observations, theory-building case studies can empirically assess theoretical precision and realism and accumulate improvements. The assimilation model—a theory of psychological change that tracks the assimilation of disconnected problematic experiences into the usual self in successful psychotherapy (Stiles, 1999, 2002, 2011)– has developed and continues to develop mainly through theory-building case studies (e.g., Caro Gabalda, & Stiles, in press; Gray, & Stiles, 2011; Meystre, Kramer, De Roten, Despland, & Stiles, 2012; Osatuke, Reid, Stiles, Zisook, & Mohamed, 2011; Ribeiro, Bento, Salgado, Stiles, & Gonçalves, 2011; Schielke et al., 2011; Tikkanen, Stiles, & Leiman, 2011). Assimilation model research has used theory-building case study deliberately, but I think most of our rich explanatory theories of psychotherapy have been developed in substantial part through similar strategies informally. According to their logic, both hypothesis-testing and theorybuilding qualitative strategies center on an articulated theory. Hypothesis testing assesses specific consequences of a theory; results inductively yield some small increment or decrement in confidence about the theory. If there is no link to theory, there is no systematic way to generalize the results, except perhaps in the limited sense of statistical generalization of a single statement to the population from which a particular sample was randomly drawn. The theory must be logically interconnected enough that observations on one part of the theory (the hypothesis) can affect confidence in other parts of the theory. Likewise, theory-building case studies depend on logical coherence of the theory to ensure that the theoretical descriptions of one case link to statements about other cases. The qualitative theory-building strategy does not directly address the consumeristic global evaluative question of whether psychotherapy is effective— or of which treatment approach is most effective. As Strupp (1963) said, “more pressing matters must be dealt with first before we can address ourselves meaningfully to [that] question” (p. 1). I suggest that a solid empirically supported theoretical account of how people change and how psychotherapy facilitates changes is such a pressing prerequisite. The distinction between statistical hypothesis testing and theory-building strategies parallels the classic distinction between nomothetic and idiographic research. I have here argued, in effect,

STILES

40

that idiographic observations need not be restricted to the context of discovery but can be systematically brought to bear on theory in the context of justification. In addition, theories built from clinical case studies may be more useful to clinicians than theories built from hypothesis-testing studies because case-tested theories will have had more theoretical features examined (Miller, 2009). Of course, the theory-building qualitative strategy does not overcome all of the problems of idiographic research. Problems involving the limited view offered by a small selection of cases and potential observer biases, for example, must be addressed by investigators who choose qualitative strategies. However, the theory-building logic does address the problem of generalizing from small numbers of cases. Campbell (1979), who had earlier been one of the most vociferous critics of case study research (e.g., Campbell, 1961; Campbell & Stanley, 1966), explicitly retracted much of his earlier criticism of case studies in light of the degrees of freedom logic. Nomothetic research has methodological problems of its own, and adding the variables problem targeted by Strupp in 1963 makes the idiographic alternative relatively more attractive for research on psychotherapy.

References American Psychological Association. (August 9, 2012). Resolution on the recognition of psychotherapy effectiveness—Approved August 2012. Retrieved from http://www.apa.org/news/press/releases/2012/08/ resolution-psychotherapy.aspx Baldwin, S. A., Berkeljon, A., Atkins, D. C., Olsen, J. A., & Nielsen, S. L. (2009). Rates of change in naturalistic psychotherapy: Contrasting dose– effect and good-enough level models of change. Journal of Consulting and Clinical Psychology, 77, 203–211. doi:10.1037/a0015235 Barkham, M., Connell, J., Stiles, W. B., Miles, J. N., Margison, F., Evans, C., & Mellor-Clark, J. (2006). Dose-effect relations and responsive regulation of treatment duration: The good enough level. Journal of Consulting and Clinical Psychology, 74, 160–167. doi:10.1037/0022-006X.74.1.160 Barkham, M., Gilbert, N., Connell, J., Marshall, C., & Twigg, E. (2005). Suitability and utility of the CORE-OM and CORE-A for assessing severity of presenting problems in psychological therapy services based in primary and secondary care settings. British Journal of Psychiatry, 186, 239 –246. doi:10.1192/bjp.186.3.239 Barkham, M., Margison, F., Leach, C., Lucock, M., Mellor-Clark, J., Evans, C., . . . McGrath, G. (2001). Service profiling and outcomes benchmarking using the CORE-OM: Towards practice-based evidence in the psychological therapies. Journal of Consulting and Clinical Psychology, 69, 184 –196. doi:10.1037/0022-006X.69.2.184 Barkham, M., Stiles, W. B., Connell, J., Twigg, E., Leach, C., Lucock, M., . . . Angus, L. (2008). Effects of psychological therapies in randomized trials and practice-based studies. British Journal of Clinical Psychology, 47, 397– 415. doi:10.1348/014466508X311713 Beck, A. T., Ward, C. H., Mendelson, M., Mock, J., & Erbaugh, J. (1961). An inventory for measuring depression. Archives of General Psychiatry, 4, 561–571. doi:10.1001/archpsyc.1961.01710120031004 Campbell, D. T. (1961). The mutual methodological relevance of anthropology and psychology. In F. L. K. Hsu (Ed.), Psychological anthropology: Approaches to culture and personality. Homewood, IL: Dorsey. Campbell, D. T. (1979). “Degrees of freedom” and the case study. In T. D. Cook & C. S. Reichardt (Eds.), Qualitative and quantitative methods in evaluation research (pp. 49 – 67). Beverley Hills, CA: Sage. Campbell, D. T., & Stanley, J. C. (1966). Experimental and quasiexperimental designs for research. Chicago, IL: Rand McNally. Caro Gabalda, I., & Stiles, W. B. (in press). Irregular assimilation progress: Setbacks in the context of linguistic therapy of evaluation. Psychotherapy Research.

Carroll, L. (1946). Alice’s adventures in wonderland. New York: Random House. (Original work published 1865) Clark, D. M., Fairburn, C. G., & Wessely, S. (2008). Psychological treatment outcomes in routine NHS services: A commentary on Stiles et al (2007). Psychological Medicine, 38, 629 – 634. doi:10.1017/ S0033291707001869 Connell, J., Barkham, M., Stiles, W. B., Twigg, E., Singleton, N., Evans, O., & Miles, J. N. (2007). Distribution of CORE-OM scores in a general population, clinical cut-off points, and comparison with the CIS-R. British Journal of Psychiatry, 190, 69 –74. doi:10.1192/bjp.bp.105.017657 Derogatis, L. R., Lipman, R. S., & Covi, L. (1973). SCL-90: An outpatient psychiatric rating scale—Preliminary report. Psychopharmacology Bulletin, 9, 13–27. Elkin, I. (1999). A major dilemma in psychotherapy outcome research: Disentangling therapists from therapies. Clinical Psychology: Science and Practice, 6, 10 –32. doi:10.1093/clipsy.6.1.10 Evans, C., Connell, J., Barkham, M., Margison, F., Mellor-Clark, J., McGrath, G., & Audin, K. (2002). Towards a standardised brief outcome measure: Psychometric properties and utility of the CORE-OM. British Journal of Psychiatry, 180, 51– 60. doi:10.1192/bjp.180.1.51 Eysenck, H. J. (1952). The effects of psychotherapy: An evaluation. Journal of Consulting Psychology, 16, 319 –324. doi:10.1037/h0063633 Eysenck, H. J. (1964). The outcome problem in psychotherapy: A reply. Psychotherapy: Theory, Research and Practice, 1, 97–100. doi:10.1037/ h0088591 Gray, M. A., & Stiles, W. B. (2011). Employing a case study in building an assimilation theory account of generalized anxiety disorder and its treatment with cognitive-behavioral therapy. Pragmatic Case Studies in Psychotherapy, 7, 529 –557. Hardy, G. E., Stiles, W. B., Barkham, M., & Startup, M. (1998). Therapist responsiveness to client interpersonal styles during time-limited treatments for depression. Journal of Consulting and Clinical Psychology, 66, 304 –312. doi:10.1037/0022-006X.66.2.304 Holt, R. R. (1958). Clinical and statistical prediction: A reformulation and some new data. Journal of Abnormal Psychology, 56, 1–12. doi:10.1037/ h0041045 Howard, K. I., Kopta, S. M., Krause, M. S., & Orlinsky, D. E. (1986). The dose-effect relationship in psychotherapy. American Psychologist, 41, 159 –164. doi:10.1037/0003-066X.41.2.159 Jacobson, N. S., & Truax, P. (1991). Clinical significance: A statistical approach to defining meaningful change in psychotherapy research. Journal of Consulting and Clinical Psychology, 59, 12–19. doi:10.1037/ 0022-006X.59.1.12 Knight, R. P. (1941). Evaluation of the results of psychoanalytic therapy. American Journal of Psychiatry, 98, 434 – 446. Kopta, S. M., Howard, K. I., Lowry, J. L., & Beutler, L. E. (1994). Patterns of symptomatic recovery in psychotherapy. Journal of Consulting and Clinical Psychology, 62, 1009 –1016. doi:10.1037/0022-006X.62.5.1009 Krause, M. S., & Lutz, W. (2009). Process transforms inputs to determine outcomes: Therapists are responsible for managing process. Clinical Psychology: Science and Practice, 16, 73– 81. doi:10.1111/j.1468-2850 .2009.01146.x Lakatos, I. (1978). The methodology of scientific research programs. Cambridge, England: Cambridge University Press. doi:10.1017/ CBO9780511621123 Leach, C., Lucock, M., Barkham, M., Stiles, W. B., Noble, R., & Iveson, S. (2006). Transforming between beck depression inventory and CORE-OM scores in routine clinical practice. British Journal of Clinical Psychology, 45, 153–166. doi:10.1348/014466505X35335 Luborsky, L., & DeRubeis, R. J. (1984). The use of psychotherapy treatment manuals: A small revolution in psychotherapy research style. Clinical Psychology Review, 4, 5–14. doi:10.1016/0272-7358(84)90034-5

THE VARIABLES PROBLEM Meystre, C., Kramer, U., De Roten, Y., Despland, J-N., & Stiles, W. B. (2012). Therapist intervention choice in the framework of the assimilation model: A theory-building case study. Manuscript submitted for publication. Miller, R. B. (2009). The logic of theory and the logic of practice. Pragmatic Case Studies in Psychotherapy, 5, 101–107. Retrieved from http://pcsp.libraries.rutgers.edu Norcross, J. C. (Ed.). (2002). Psychotherapy relationships that work: Therapist contributions and responsiveness to patient need. New York: Oxford University Press. Norcross, J. C. (Ed.). (2011). Psychotherapy relationships that work: Evidence-based responsiveness (2nd ed). New York: Oxford University Press. doi:10.1093/acprof:oso/9780199737208.001.0001 Orlinsky, D. E., Grawe, K., & Parks, B. K. (1994). Process and outcome in psychotherapy–noch einmal. In A. Bergin & S. Garfield (Eds.), Handbook of psychotherapy and behavior change (4th ed). New York: Wiley. Osatuke, K., Reid, M., Stiles, W. B., Zisook, S., & Mohamed, S. (2011). Narrative evolution and assimilation of problematic experiences in a case of pharmacotherapy for schizophrenia. Psychotherapy Research, 21, 41–53. doi:10.1080/10503307.2010.508760 Ribeiro, A. P., Bento, T., Salgado, J., Stiles, W. B., & Gonçalves, M. M. (2011). A dynamic look at narrative change in psychotherapy: A case study tracking innovative moments and protonarratives using state-space grids. Psychotherapy Research, 21, 54–69. doi:10.1080/10503307.2010.504241 Rogers, C. R. (1959). A theory of therapy, personality, and interpersonal relationships as developed by the client-centered framework. In S. Koch (Ed.), Psychology: A study of a science: Volume III. Formulations of a person and the social context (pp. 184 –256). New York: McGraw-Hill. Rosenzweig, S. (1936). Some implicit common factors in diverse methods of psychotherapy. American Journal of Orthopsychiatry, 6, 412– 415. doi:10.1111/j.1939-0025.1936.tb05248.x Schielke, H. J., Stiles, W. B., Cuellar, R. E., Fishman, J. L., Hoener, C., Del Castillo, . . . Greenberg, L. S. (2011). A case investigating whether the process of resolving interpersonal problems in couple therapy is isomorphic to the process of resolving problems in individual therapy. Pragmatic Case Studies in Psychotherapy, 7, 477–528. Shapiro, D. A., & Firth, J. (1987). Prescriptive vs. exploratory psychotherapy: Outcomes of the Sheffield Psychotherapy Project. British Journal of Psychiatry, 151, 790 –799. doi:10.1192/bjp.151.6.790 Sloane, R. G., Staples, F. R., Cristol, A. H., Yorkston, N. J., & Whipple, K. (1975). Psychotherapy versus behavior therapy. Cambridge, MA: Harvard University Press. Stiles, W. B. (1981). Science, experience, and truth: A conversation with myself. Teaching of Psychology, 8, 227–230. doi:10.1207/s15328023top0804_11 Stiles, W. B. (1988). Psychotherapy process-outcome correlations may be misleading. Psychotherapy, 25, 27–35. doi:10.1037/h0085320 Stiles, W. B. (1992). Describing talk: A taxonomy of verbal response modes. Newbury Park, CA: Sage. Stiles, W. B. (1999). Signs and voices in psychotherapy. Psychotherapy Research, 9, 1–21. Stiles, W. B. (2002). Assimilation of problematic experiences. In J. C. Norcross (Ed.), Psychotherapy relationships that work: Therapist contributions and responsiveness to patients (pp. 357–365). New York: Oxford University Press. Stiles, W. B. (2006). Numbers can be enriching. New Ideas in Psychology, 24, 252–262. doi:10.1016/j.newideapsych.2006.10.003 Stiles, W. B. (2009a). Logical operations in theory-building case studies. Pragmatic Case Studies in Psychotherapy, 5, 9 –22. Retrieved from http://jrul.libraries.rutgers.edu/index.php/pcsp/article/view/973/2384 Stiles, W. B. (2009b). Responsiveness as an obstacle for psychotherapy outcome research: It’s worse than you think. Clinical Psychology: Science and Practice, 16, 86 –91. doi:10.1111/j.1468-2850.2009.01148.x

41

Stiles, W. B. (2010). Theory-building case studies as practice-based evidence. In M. Barkham, G. Hardy, & J. Mellor-Clark (Eds.), Developing and delivering practice-based evidence: A guide for the psychological therapies (pp. 91–108). Chichester, UK: Wiley-Blackwell. Stiles, W. B. (2011). Coming to terms. Psychotherapy Research, 21, 367–384. Stiles, W. B., Barkham, M., Connell, J., & Mellor-Clark, J. (2008). Responsive regulation of treatment duration in routine practice in United Kingdom primary care settings: Replication in a larger sample. Journal of Consulting and Clinical Psychology, 76, 298 –305. doi:10.1037/0022006X.76.2.298 Stiles, W. B., Barkham, M., Mellor-Clark, J., & Connell, J. (2008a). Effectiveness of cognitive-behavioural, person-centred, and psychodynamic therapies in UK primary care routine practice: Replication in a larger sample. Psychological Medicine, 38, 677– 688. doi:10.1017/ S0033291707001511 Stiles, W. B., Barkham, M., Mellor-Clark, J., & Connell, J. (2008b). Routine psychological treatment and the Dodo verdict: A rejoinder to Clark et al. Psychological Medicine, 38:905–910, 2007. Stiles, W. B., Barkham, M., Twigg, E., Mellor-Clark, J., & Cooper, M. (2006). Effectiveness of cognitive-behavioural, person-centred, and psychodynamic therapies as practiced in UK National Health Service settings. Psychological Medicine, 36, 555–566. doi:10.1017/ S0033291706007136 Stiles, W. B., Honos-Webb, L., & Surko, M. (1998). Responsiveness in psychotherapy. Clinical Psychology: Science and Practice, 5, 439 – 458. doi:10.1111/j.1468-2850.1998.tb00166.x Stiles, W. B., & Shapiro, D. A. (1989). Abuse of the drug metaphor in psychotherapy process-outcome research. Clinical Psychology Review, 9, 521–543. doi:10.1016/0272-7358(89)90007-X Stiles, W. B., & Shapiro, D. A. (1994). Disabuse of the drug metaphor: Psychotherapy process-outcome correlations. Journal of Consulting and Clinical Psychology, 62, 942–948. doi:10.1037/0022-006X.62.5.942 Stiles, W. B., Shapiro, D. A., & Firth-Cozens, J. A. (1988). Verbal response mode use in contrasting psychotherapies: A within-subjects comparison. Journal of Consulting and Clinical Psychology, 56, 727– 733. doi:10.1037/0022-006X.56.5.727 Stiles, W. B., & Wolfe, B. E. (2006). Relationship factors in treating anxiety disorders. In L. G. Castonguay & L. E. Beutler (Eds.), Principles of therapeutic change that work (pp. 155–165). New York: Oxford University Press. Strupp, H. H. (1963). The outcome problem in psychotherapy revisited. Psychotherapy: Theory, Research and Practice, 1, 1–13. doi:10.1037/ h0088565 Strupp, H. H. (1964). The outcome problem in psychotherapy: A rejoinder. Psychotherapy: Theory, Research and Practice, 1, 101. doi: 10.1037/h0088579 Tikkanen, S., Stiles, W. B., & Leiman, M. (2011). Parent development in clinical child neurological assessment process: Encounters with the assimilation model. Psychotherapy Research, 21, 593– 607. doi: 10.1080/10503307.2011.594817 Wing, J. K., Cooper, J. E., & Sartorius, N. (1974). The Measurement and Classification of Psychiatric Symptoms. Cambridge: Cambridge University Press. Yeaton, W. H., & Sechrest, L. (1981). Critical dimensions in the choice and maintenance of successful treatments: Strength, integrity, and effectiveness. Journal of Consulting and Clinical Psychology, 49, 156 –167. doi:10.1037/0022-006X.49.2.156

Received August 29, 2012 Accepted September 2, 2012 䡲