Development and Use of a Transition Readiness Scale to Help Manage ACT Team Capacity Sheila A. Donahue, M.A. Jennifer I. Manuel, Ph.D. Daniel B. Herman, Ph.D. Linda H. Fraser, M.P.A. Henian Chen, M.D., Ph.D. Susan M. Essock, Ph.D.
Objective: This article describes the creation, validation, and use of an assertive community treatment (ACT) Transition Readiness Scale (TRS) to identify clients who may be ready to transition from ACT services. Scale development was prompted by concerns over long stays on ACT teams and the resulting impact on access. Methods: Data were extracted from a centralized clinical reporting system for all 1,365 persons enrolled for at least 12 months as of August 2008 in the 42 ACT teams in New York City, including 382 clients of eight of those ACT teams. Data in seven domains deemed relevant to transition readiness were used to calculate readiness scores for each client. An algorithm assigned clients to one of three categories: consider for transition, readiness unclear, and not ready. Results: Via the TRS algorithm, of the 1,365 clients, 192 (14%) were assigned to the consider-for-transition group, 382 (28%) to the unclear group, and 791 (58%) to the not ready group. Clinicians on the eight ACT teams categorized 15% of their current clients in the consider-for-transition group, whereas the TRS algorithm classified 18% in this category. Overall, the TRS agreed with the category assigned by ACT team clinicians in 69% of cases. Conclusions: The TRS may provide ACT teams and program administrators with a tool to identify clients who may be ready to transition to less intensive services, thereby opening scarce slots. Because ACT cases are complex, data summaries can offer useful syntheses of information, particularly when data from several assessments are used to summarize a client’s trajectory. (Psychiatric Services 63:223–229, 2012; doi: 10.1176/appi.ps.201100041)
Ms. Donahue is affiliated with the Office of Performance Measurement and Evaluation, New York State Office of Mental Health, Albany. Dr. Manuel is with the School of Social Work, Virginia Commonwealth University, Richmond. Dr. Herman and Dr. Essock are with the Department of Psychiatry, College of Physicians and Surgeons, Columbia University, New York City. Dr. Essock is also with the Department of Mental Health Services and Policy Research, New York State Psychiatric Institute, New York City. Ms. Fraser is with the Office of Coordinated Mental Health Services, New York City Department of Health and Mental Hygiene, New York City. Dr. Chen is with Winthrop University Hospital, Stony Brook University School of Medicine, Stony Brook, New York. Send correspondence to Dr. Essock at the New York State Psychiatric Institute, 1051 Riverside Dr., Unit 100, Room 1599, New York, NY 10032 (e-mail:
[email protected]).
PSYCHIATRIC SERVICES
o ps.psychiatryonline.org o March 2012 Vol. 63 No. 3
A
ssertive community treatment (ACT) is an evidence-based model for providing homebased interdisciplinary treatment to adults with severe mental illness living in the community (1). A key fidelity criterion is that ACT teams provide time-unlimited services, meaning that clients can continue to receive treatment from the ACT team as long as needed (2). As ACT has spread, some teams, either intentionally or functionally, have interpreted “time unlimited” to mean “forever,” leading to extended stays and little turnover on many ACT teams. The recommendation that ACT services should be time unlimited evolved from early research that showed a decline in clinical gains 12 months after a planned termination of ACT services (3). However, when this research was carried out, the alternatives for community treatment in most parts of the United States were quite limited. Since that time, most resource-intensive systems have greatly evolved and include a range of recovery-oriented community-based services, such as clinic treatment, case management, housing, and other supports. Furthermore, the notion that most clients would require ACT as a permanent support is inconsistent with the goal of helping people maximize recovery from serious mental illnesses and move on to a life in the community that includes making one’s own decisions about how to best 223
use a range of possible treatment services and supports. Accumulating research suggests that transitioning from ACT to less intensive services without deleterious effects is possible for many clients (4–8). In the Department of Veterans Affairs (VA), Rosenheck and colleagues (4) used administrative data to assess outcomes for veterans receiving Mental Health Intensive Case Management (MHICM) services, VA’s version of ACT services. They found that only 5.7% of clients who transitioned to less intensive services were readmitted to ACT services, suggesting that most who made this transition did so successfully. Rosenheck and Dennis (5) examined postdischarge outcomes of ACT clients and found that clients could be selectively transferred from ACT to other services without a consequential decline in activities of daily living and psychiatric stability. In a retrospective study, records of 98 clients who were transferred from ACT to a step-down model were reviewed and compared with records of a nonequivalent comparison group of 108 clients receiving ACT services (6). The step-down model was conceived as a reduced-intensity form of ACT that followed many of the ACT principles and core components. Results indicated sustained improvement in functioning and less psychiatric hospitalization for step-down clients than those in the comparison group. McRae and colleagues (7) followed 72 clients for two years after they transitioned from ACT to mainstream outpatient treatment, noting that most clients remained stable two years after the transition, which the authors attributed to the enduring effects of ACT, the coordinated transition to outpatient treatment, and outpatient providers’ dedication in retaining clients in treatment. These findings are consistent with recent research on critical time intervention, which suggests that this time-limited, transitional form of case management can deliver enduring positive impacts by connecting persons with severe mental illness to ongoing support in the community (9). Limited research has shed light on individual-level characteristics that 224
are associated with more or less positive outcomes for clients after ACT discharge. For instance, Hackman and Stowell (8) compared outcomes of 48 individuals who transitioned successfully from ACT to routine services (as evidenced by their continued use of routine services) with outcomes of 19 individuals who returned to ACT or dropped out of treatment. They found that clients who returned to ACT services often had ceased to attend clinic appointments and had been hospitalized, suggesting that an important component of readiness for discharge from an ACT team is the capacity to schedule and keep appointments independently or to have adequate support from other sources (such as a case manager) to help one do so. A recent study of patterns of discharge from MHICM services found mixed results in a retrospective analysis of data from clients who terminated early (defined as less than one year in MHICM services), terminated later (one to three years in MHICM services), or had not yet terminated MHICM services (10). Compared with clients who had not terminated, clients who terminated early were more likely to be suicidal and display violent behavior. Rates of termination due to uncooperative behavior or service refusal were high for each group (36% of clients who terminated early and 31% of those who terminated later). Thus early termination from ACT, far from being a sign of quick success, may instead reflect failure to engage clients with the most complex needs. Like many states, New York State has invested heavily in creating ACT teams, many of which now have limited openings for clients because of very long stays in services. As of January 1, 2010, New York State funded 79 ACT teams that were serving 5,064 individuals, with 58% of ACT slots located in the 43 New York City (NYC) ACT teams. This capacity has been fully utilized for several years, with a 95% occupancy rate statewide and 98% in NYC. Most NYC ACT teams began more than five years ago; only seven teams have been licensed since the beginning of 2006. By the fall of 2006, these teams were full and PSYCHIATRIC SERVICES
waiting lists had begun. Data from the New York State Office of Mental Health (OMH) indicated that approximately 25% of clients had been on NYC ACT teams for more than five years (11). This issue had prompted ACT program oversight staff from the NYC Department of Health and Mental Hygiene (DOHMH) and OMH to create a utilization review process as part of managing the demand for ACT services. As part of the utilization review process, we created an ACT Transition Readiness Scale (TRS) intended to identify clients who may be ready to transition out of ACT into routine services. The scale was intended as a clinical decision aid to help clinicians and administrators identify clients for whom such a transition process might be appropriate. This feasibility report describes the development of the scale and early efforts to assess its validity. The OMH Institutional Review Board deemed that this project did not constitute research with human subjects.
Methods Identifying components for the scale ACT is intended to serve clients who have especially high levels of need for support and treatment and who have not been successfully served by traditional office-based interventions. Thus eligibility criteria typically require multiple recent hospitalizations or homelessness in addition to a serious mental illness (1,12). The job of the ACT team is to help the person engage in treatment and achieve greater stability in housing, reduced use of substances, and improved mental health. Hence a scale to assess readiness for transition from ACT to more routine services would logically include measures of whether a client was stably housed, ready to be engaged in treatment with appropriate outpatient treatment providers, avoiding risky or dangerous situations, and not a recent user of crisis services. A work group composed of staff from OMH and DOHMH (with consultation by ACT team leaders) set out to develop a tool that would verify that individuals currently served by ACT teams are in need of this inten-
o ps.psychiatryonline.org o March 2012 Vol. 63 No. 3
sive level of care and identify individuals who may be ready to begin the transition process from ACT to a lower level of support. The intent was to create and pilot test a utilization review tool using client-specific data that ACT teams already were required to report via a standardized assessment reporting system operated by OMH—the Child and Adult Integrated Reporting System (CAIRS). Routine assessments at entry into services and every six months thereafter capture client demographic characteristics and ratings on individual items, which were then collapsed into 11 domains of functioning. The work group selected seven of these domains (created by collapsing 22 individual items) as relevant to whether a client was ready to transition from ACT to routine outpatient services: housing, psychiatric hospitalization or emergency department use, use of psychiatric medication, engagement in routine services, substance abuse, forensic involvement, and any incidence of harmful behaviors. The four domains of educational and vocational activity, self-care, social relationships, and health and medical status were excluded from estimations of readiness to transition because deficits in these domains alone were deemed not sufficient to justify continuing to receive ACT-level services. Rather, clients doing well in the seven focal domains but still requiring assistance in these other four domains could presumably be successfully transitioned and have these needs met by outpatient mental health service providers or by other community supports. Scaling within a domain of functioning Table 1 shows, for each of the seven domains, the universe of possible composite scores derived from the individual scores on a client’s functioning reported by ACT staff. The composite scores range from 1 to 5, plus “unknown.” When creating a composite score on the TRS, responses of “unknown” to any of the individual items were treated as if the events monitored in the domain had not occurred in the previous six months. This approach was chosen to discourPSYCHIATRIC SERVICES
age use of “unknown” as a response by staff in order to reduce the number of clients identified for transition readiness assessments. Creating an overall readiness score An overall score was calculated by examining scores in the seven domains in Table 1. The intent was to create an algorithm to assign a client to one of three categories: consider for transition, transition readiness unclear, and not ready for transition. The category of consider for transition was operationalized as having a score of 1, 2, or 3 in the treatment engagement domain and no score greater than 2 in the other six domains relevant to transition readiness. Those in the group labeled transition readiness unclear scored 1, 2, or 3 in the treatment engagement domain, had at least one score of 3 in the other six domains, and had no score higher than 3 in any of the other four domains. The group considered not ready for transition included all others—that is, those who had at least one score of 4 or higher in any of the seven domains relevant to transition readiness. Pilot testing the TRS We employed two means of assessing the scale’s validity. First, we field tested the instrument with a convenience sample of eight ACT teams chosen by the TRS work group to assess the congruence between the clinical judgment of ACT team staff regarding readiness to transition and the transition scale algorithm. This comparison assessed the extent to which ratings generated by the TRS agreed with a team’s clinical judgment. Each participating team was asked to consider all clients on the team’s caseload who had been receiving services for at least one year and to assign each of these clients to one of the three readiness categories. Teams made these ratings at a regular morning ACT team meeting so that all team members could provide input into the rating assigned to each client. These ratings were then compared with the rating obtained on the TRS that was based on the CAIRS data previously submitted by the team. Second, we determined how well
o ps.psychiatryonline.org o March 2012 Vol. 63 No. 3
ratings produced by the TRS predicted which clients were discharged from ACT because their goals had been met (as reported by program staff via CAIRS). If the scale accurately identifies individuals who are ready for transition, clients coded at discharge as having substantially met program goals (also in CAIRS) should be more likely to have received readiness scores of “consider for transition” than clients who were reported to have been discharged for any other reason. Data extraction and analysis In August 2008, OMH staff extracted data for all clients then enrolled in the 42 NYC ACT teams and identified the subset of 1,365 individuals who had at least two follow-up assessments in CAIRS. Because follow-up assessments occur every six months, this meant that each individual in the sample had been enrolled in ACT for at least 12 months. We excluded 251 other individuals who had been mandated to participate in ACT pursuant to a court order because their transition from ACT was influenced by the legal system. Continuous data were expressed as means and standard deviations and analyzed by use of analysis of variance. Categorical data were expressed as percentages and analyzed by means of a chi square test or Fisher’s exact test. We assessed the scale’s sensitivity (the proportion of times that clients categorized by the scale as ready for transition were also identified as ready by the clinical judgment of the client’s ACT team) and specificity (the proportion of clients identified as not ready for transition by the TRS who also were identified as not ready for transition by the clinical judgment of the client’s ACT team). We also used receiver operating characteristic (ROC) analysis methods to evaluate how well the scale performed. Analysis of the ROC provides a precise and valid measure of diagnostic accuracy uninfluenced by prior probabilities, and it places diverse systems on a common scale that is easy to interpret. We examined data for clients deemed ready for transition both by the ACT TRS and by the teams’ clinical judgment. We used the teams’ clinical judgment as the gold 225
Table 1
Domains and ratings on the assertive community treatment Transition Readiness Scalea Rating Domain
1
2
3
4
5
Treatment engagement
Excellent (independently and appropriately uses services)
Good (able to partner with service provider and can use resources independently)
Fair (no independent use of services or only in extreme need)
Poor (relates poorly to providers, avoids independent contact with providers)
Not engaged (no contact with providers, does not participate in services at all)
Housing
Housed in the community for more than 12 months
Housed in the community for 7 to 12 months
Housed in the com- Living in the commu- Homeless or had days munity for 1 to 6 nity for less than 1 homeless in the past 6 months month or in another months setting, but not homeless
Psychiatric medication use
Either no medica- For past 6 months takes tions prescribed medications at least or adherent most most of the time but of the time on last may need some verbal 2 assessments assistance
Takes medications at least sometimes but may need some physical assistance
Psychiatric hospitalization and emergency room (ER) use
No inpatient admissions or ER visits in the past 12 months
No inpatient admissions and fewer than 3 ER visits in the past 12 months
Up to 1 inpatient (This category is not admission and no used) ER visits in the past 12 months, or fewer than 10 ER visits and no inpatient admissions in the past 12 months
High-risk behaviorsb
None of the 12 high-risk behaviors in at least the past year
None of the 8 most None of the 12 high-risk behaviors high-risk behaviors in at least the past year in at least the past 6 months
Substance abuse Abstinent on last Abstinent on last follow- No abuse at last two follow-ups up, either abstinent or two follow-ups used without impairment on the next-tolast follow-up, or if alcohol is the only substance used, could have used without impairment on the last and next-to-last follow-up Forensic involvement
a
b
No arrests and no (This category is not days incarcerated used) in the past 12 months
Either takes medications rarely or never as prescribed, or requires substantial help to take medications
Takes medications rarely or never as prescribed, or level of assistance needed is unknown 2 or more inpatient admissions, or 10 or more ER visits in the past 12 months
None of the 8 most high-risk behaviors in at least the past 6 months
1 or more of the 8 most high-risk behaviors in the past 6 months
Abuse indicated on at least one of last two follow-ups
Abuse or dependence listed on last 2 follow-ups
No arrests and no (This category is not days incarcerated in used) the past 6 months
Arrests or days incarcerated in the past 6 months
The rating for each domain is the composite of the individual items for that domain rated by ACT team clinicians. “Consider for transition” is operationalized as having a score of 1, 2, or 3 in the treatment engagement domain and 1 or 2 in the other six domains. In creating the TRS composite score, responses of “unknown” to any of the individual items are treated as if the events monitored in the domain had not occurred in the previous six months. The 12 high-risk behaviors are physically harmed self, damaged property, created a public disturbance, verbal assault, threatened assault or physical violence, suspected of sexual abuse, physical abuse of another, arson, threatened suicide, taken property, victim of physical or sexual abuse, and wandered or ran away. The first eight listed were considered most problematic.
standard in the ROC analysis, using the area under the curve as a summary measure for classification ability. All analyses were performed with the use of SAS, version 9.2. A two-sided p value of less than .05 was considered to indicate statistical significance.
Results We applied the TRS algorithm to the data in CAIRS for the 1,365 individu226
als enrolled in NYC ACT teams as of August 2008 who had at least two follow-up assessments. A total of 192 clients (14%) were assigned to the consider-for-transition group, 382 (28%) to the transition readiness unclear group, and 791 (58%) to the not-ready-for-transition group. Table 2 details the distribution of ratings within each domain and the overall transition scale scores. PSYCHIATRIC SERVICES
Characteristics of clients in the consider-for-transition group Individuals assigned to the considerfor-transition group had received ACT services for a mean±SD of 4.6±3.0 years (median=3.9 years). For the unclear and not ready groups, means were 4.4±3.1 and 4.0±3.0 years, respectively (medians=3.6 and 3.1 years, respectively). The difference in mean service tenure among
o ps.psychiatryonline.org o March 2012 Vol. 63 No. 3
Table 2
Ratings in domains of the Transition Readiness Scale of 1,365 clients receiving assertive community treatment servicesa
1
2
3
4
Unable to rate (missing data)
5
Domain
N
%
N
%
N
%
N
%
N
%
N
%
Treatment engagement Housing Psychiatric medication use Psychiatric hospitalization and emergency room use High-risk behaviors Substance abuse Forensic involvement
114 1,021 914 953
8 75 67 70
561 59 30 33
41 4 2 2
513 81 170 48
38 6 12 4
140 49 138 —b
10 4 10
20 55 57 331
1 4 4 24
17 100 56 0
1 7 4 —
986 910 1,285
72 67 94
66 35 —b
5 3
99 76 24
7 6 2
6 196 —b
