the performance of publicly financed mental health organizations. Be- cause patients ... ods: In one county mental health system, routinely collected data on a.
Routine Outcome Monitoring in a Public Mental Health System: The Impact of Patients Who Leave Care Alexander S. Young, M.D., M.S.H.S. Oscar Grusky, Ph.D. Daniel Jordan, Ph.D. Thomas R. Belin, Ph.D.
Objective: An interest exists in using patient outcome data to evaluate the performance of publicly financed mental health organizations. Because patients leave these organizations at a high rate, the impact of patient attrition on routinely collected outcome data was examined. Methods: In one county mental health system, routinely collected data on a wide range of outcomes were examined, and a random sample of patients who left treatment was interviewed. Results: Of the 1,769 patients in ongoing treatment during a one-year period, 554 (31 percent) were lost to follow-up. Among a random sample of 102 patients who left treatment, two had died and 47 were interviewed. Compared with patients who left treatment, patients who stayed were older, more likely to have schizophrenia, less likely to be married, more likely to be living in an institution, more satisfied with their relationships with friends and family, and less likely to have legal problems. Average outcomes improved both for patients who stayed and for patients who left. Patients who left and could be located for follow-up were less severely ill and showed the greatest improvement and the best outcomes. Patients who left and could not be located may have been more severely ill at baseline. Conclusions: Outcomes appear to vary substantially by whether patients stay in care and whether they can be located after leaving care. Public mental health systems that wish to evaluate treatment quality using outcome data should attend carefully to which patients are being assessed. Biases can result from convenience sampling and from patients leaving care. (Psychiatric Services 51:85–91, 2000)
C
onsumers, purchasers, and providers of publicly funded mental health care are interested in using data on patient outcomes to evaluate the performance of
treatment organizations and to improve the quality of care (1–3). Although outcome assessment methods are often developed for research purposes (4), it is increasingly common
Dr. Young is assistant clinical professor in the department of psychiatry at the University of California, Los Angeles (UCLA), and associate director of the Mental Illness Research, Education, and Clinical Center of the Department of Veterans Affairs Veterans Integrated Service Network 22 at the West Los Angeles Veterans Healthcare Center, Building 210A, 11301 Wilshire Boulevard, Los Angeles, California 90073 (e-mail, ayoung@ucla. edu). Dr. Grusky is professor in the department of sociology at UCLA. Dr. Jordan is research psychologist at Ventura County Behavioral Health in Ventura, California. Dr. Belin is assistant professor in the departments of psychiatry and biostatistics at UCLA.
PSYCHIATRIC SERVICES
♦ January 2000 Vol. 51 No. 1
for outcomes to be routinely measured for all patients and for these data to be incorporated into the databases of management information systems. Although traditional administrative databases often contain information on each patient’s demographic characteristics, diagnoses, and treatment utilization, some newer databases have added variables in outcome domains, such as satisfaction, functioning, and quality of life. These new data elements promise to make it easier to study the efficiency and effectiveness of existing programs through analysis of variation in treatment exposure and outcomes (5,6). They also provide an infrastructure for evaluating new clinical interventions. However, there are important challenges to the accurate use of routinely collected outcome data. These challenges are commonly acknowledged to include the need to develop inexpensive, useful, and reliable outcome measures and the need for methods that control for severity of illness (7). The change over time in the average outcome of a population can be strongly affected by illness severity, because more severely ill patients may improve less than moderately ill patients. However, much less attention has been paid to another major methodological question: what bias is introduced by not having information on patients who become lost to follow-up? Routinely collected outcome data may be entirely unavailable for patients with certain 85
outcomes, potentially biasing evaluations of organizational performance. Few studies have been done of the direction or magnitude of this bias for patients with severe mental illness or of potential mechanisms for dealing with this problem. Bias due to attrition may be especially problematic in public mental health systems because patients in these systems often have illnesses such as schizophrenia or bipolar disorder that can cause them to drop out of treatment as their clinical condition worsens (8,9). Studies of mental health treatment have reported that between 20 and 75 percent of patients leave care (10–12) and significant variation exists in the outcomes of patients after they leave (8,13–15). Much of this variation in attrition rates and outcomes may be accounted for by differences in the time frames examined and the populations studied. Most studies of attrition have focused on patients who are not severely ill and whose treatment consists primarily of psychotherapy (16– 18). Surveys of patients with mild or moderate illness have found that reasons for leaving care can be grouped into three categories: no need for services, dislike of services, and environmental constraints (17). However, public mental health clinics often focus on treating patients with severe and persistent disorders. In this population, writers have speculated that important reasons for leaving care may include worsening of illness, failing to appreciate the benefits of treatment, being referred to a different program, or being unable to return (8). For example, people with schizophrenia have a substantially elevated risk of death due to a variety of causes (19) and are at high risk for incarceration, prolonged hospitalization, and homelessness (20,21). Two studies have examined severely ill populations, and both focused on the first five to eight sessions. In these studies, patients who left early in treatment were more severely ill, had less insight into their illness, and apparently failed to engage with the treatment offered (22,23). However, little is known about pa86
tients who leave public clinics over longer periods of time or about the degree to which patients who leave actually need further care. Although individuals who are less ill may function without professional mental health care, people with severe illnesses are at high risk for relapse and behavioral problems if, for example, they fail to accept appropriate medication treatment (24,25). Furthermore, keeping individuals with severe illness in treatment may require special treatment approaches (26, 27). Therefore, failing to keep se-
Bias due to attrition may be especially problematic in public mental health systems because patients often have illnesses that can cause them to drop out of treatment as their clinical condition worsens.
verely ill individuals in treatment may be a sign of poor clinical performance. The association of patient attrition with particular outcomes may lead to misleading conclusions if we monitor only the outcomes of patients who remain in treatment. For instance, if severely ill patients often leave care and then relapse, their leaving could signal a problem with treatment quality. However, such attrition would result in inflated estimates of treatment success and the outcome status of continuing patients could
even appear to improve. Alternatively, if moderately ill patients leave care when they recover, then the clinic may be providing effective care, but these positive effects would not be captured if outcomes were assessed only for continuing patients. The objectives of this study were to describe patients who left care in one public mental health system, to determine the effect that this attrition had on performance evaluation based on routinely collected outcome data, and to explore whether a follow-up survey of randomly selected patients who had left care could be used to adjust for the effect of attrition. We hypothesized that attrition in public mental health systems is not a random event and that it may often be related to worsening or improvement in patient outcomes. The study used outcome data collected by clinicians using a structured instrument during routine treatment, not as part of a research protocol. We studied a population of patients in publicly funded treatment in Ventura County, California, between January 1993 and June 1994. Previous analyses used longitudinal data from this population to examine the effect of case management teams on outcomes and costs, and a significant problem was encountered with missing outcome data at follow-up (28). To describe the population of patients who left care, we selected a random sample of these individuals and interviewed as many as possible about their outcomes and why they had left treatment. To understand the attrition process, we identified patients who remained in care and patients who left care. Patients who left care (and were still alive) included those who could be interviewed at follow-up and those who could not be interviewed at follow-up. We compared demographic, illness, and outcome variables between these groups at baseline. We also compared changes in outcomes over time between the two groups for which we could obtain follow-up data—that is, patients who remained in care and those who left care but who were subsequently located.
PSYCHIATRIC SERVICES
♦ January 2000 Vol. 51 No. 1
Methods The study was conducted in Ventura County, California, which is both suburban and rural and has a population of more than 700,000 people. Ventura County Behavioral Health, the county’s public behavioral health agency, is the main provider of mental health services to seriously mentally ill adults. At the time of this study, the agency operated one hospital and ten regional community mental health clinics. Clinical case managers routinely collected data on diagnosis, Global Assessment of Functioning (GAF) (29) scores, and outcomes of patients treated at community clinics. Every six months, clinicians were required to record outcome data for each patient in their care using a structured instrument called the Personal Profile. Conformance with this requirement was carefully monitored, and outcome assessment was rarely omitted. The Personal Profile includes information on living situation, income, work activities, physical health status and treatment, drug and alcohol use, social network, educational activities, and legal problems (28). It consists of a variety of highly structured questions. For example, one question documents the number of paid hours of work during the past week. Another rates the patient’s satisfaction with family members on a 5-point scale, from dissatisfied to satisfied. Patients were included in the study if they had had at least one Personal Profile completed during the one-year period between January and December 1993. This inclusion criterion resulted in a total of 1,769 patients, of which 554 (31 percent) had no follow-up profile during the subsequent six months. From these 565 individuals with missing Personal Profile data, a random sample of 113 was drawn to locate, contact, and interview. Between January and June of 1996, an intensive effort was made to locate these individuals using information from clinicians at the mental health clinic and, when necessary, patients’ families, friends, and neighbors. Sixty individuals were located; two were deceased, and 58 (51 percent) PSYCHIATRIC SERVICES
were interviewed by phone. All respondents answered two open-ended questions about why they left the program and what they were currently doing. Responses were independently classified by two raters into categories. Discrepant ratings were discussed, and complete agreement was reached. Fifty-four respondents also agreed to answer questions from the Personal Profile, which was administered by phone by one trained interviewer. Outcome variables used in these analyses were based on responses to questions in the Personal Profile. A
Although a few of the patients who left treatment had done poorly, most did remarkably well and actually had better outcomes than the group that remained in treatment.
drug or alcohol problem was scored as present if the respondent reported that drugs or alcohol had adversely affected one or more of seven specific functional domains during the past six months. Other outcome domains consisted of responses to one question from the Personal Profile. Two-tailed tests of statistical significance were used. Statistics were not adjusted for multiple comparisons. Therefore, differences with p values of less than .05 should be regarded as tentative and are presented as trends, while values of less than
♦ January 2000 Vol. 51 No. 1
.001 are presented as significant differences. Tests of between-group comparisons were performed using t tests for continuous variables and chi square tests for categorical variables. Tests of change from baseline to follow-up were performed using Wilcoxon signed-rank tests for continuous variables and McNemar’s test for categorical variables.
Results At baseline, of the 1,769 patients in treatment, 51 percent were female. Sixty-seven percent were white, 23 percent were Hispanic, and 5 percent were black. The mean±SD age was 40±12 years, and the mean±SD GAF score was 46±12. The most common primary diagnoses among the 1,769 patients were schizophrenia (50 percent), major depression (15 percent), bipolar disorder (11 percent), anxiety disorder (5 percent), adjustment disorder (4 percent), and dysthymia (4 percent). Of the 113 missing patients randomly selected for follow-up, 11 (10 percent) said that they had not, in fact, left the treatment program and were therefore excluded from further data analyses. It is likely that they had temporarily left treatment during the period when their Personal Profile was required. We learned that two patients were deceased. The 47 respondents gave a total of 73 reasons why they left treatment; some gave more than one reason. The most frequent reasons were that they had improved (15 patients, or 32 percent of respondents), were having problems with the clinician (14 patients, or 30 percent), were having problems with the treatment (11 patients, or 23 percent), or had left the area (12 patients, or 26 percent). Barriers to treatment were cited less often, by only ten patients (21 percent), and included cost, transportation problems, comorbid disorders, and bureaucratic issues. Other reasons included being referred by clinic staff to another program (six patients, or 13 percent), family problems with the clinician (four patients, or 9 percent), and being hospitalized (one patient, or 2 percent). When asked about their current 87
Table 1
Demographic and illness characteristics at baseline of patients who stayed in publicly funded mental health treatment, those who left treatment, and those who could and could not be located at follow-up Follow-up sample of patients who left treatment
Characteristic Gender Female Male Race or ethnicity White Hispanic Black Other Marital status Married Not married Diagnosis Schizophrenia Bipolar disorder Major depression Other Age (mean±SD years) Education level (mean±SD years) Global Assessment of Functioning (mean±SD current score) ∗p
