Implementing Evidence-Based Practices for ... - Semantic Scholar

Schizophrenia Bulletin vol. 35 no. 4 pp. 704–713, 2009 doi:10.1093/schbul/sbp041 Advance Access publication on June 2, 2009

Implementing Evidence-Based Practices for People With Schizophrenia

Robert E. Drake1,2, Gary R. Bond3, and Susan M. Essock4

individuals with schizophrenia in the United States are unlikely to receive these effective treatments.11 Epidemiologic data from the National Comorbidity Study in the early 1990s showed that 60% of persons with serious mental illnesses received no treatment in the past year, 25% received clearly inadequate treatment, and only 15% received minimally adequate (far short of evidence-based) treatment.12 The Schizophrenia PORT study similarly found that patients in 2 large state public mental health programs were unlikely to receive most of the indicated evidence-based practices.13 More recently, the 2005 National Survey on Drug Use and Health14 found that only 8.5% of adults who reported both serious psychological distress and a substance use disorder received any treatment (again far short of evidence-based treatment) for both problems in the past year. Several recent studies indicate that quality of care may be worsening rather than improving.15–17 Thus, even as researchers continue to develop more effective interventions for the treatment of schizophrenia, the preponderance of individuals with this disorder, perhaps as many as 95%, receive either no care or less than optimal care. Previous articles in this special section have reviewed the epidemiology of services and the problems of engagement and retention. In this article, we review efforts to implement effective interventions for schizophrenia in routine mental health treatment settings and offer suggestions for narrowing the gap between science and practice.

2

Dartmouth Psychiatric Research Center, Dartmouth Medical School, Lebanon, NH; 3Department of Psychology, Indiana University-Purdue University Indianapolis, Indianapolis, IN; 4 Department of Psychiatry, Columbia University, and New York State Psychiatric Institute, New York, NY

Over the last decade, a consensus has emerged regarding a set of evidence-based practices for schizophrenia that address symptom management and psychosocial functioning. Yet, surveys suggest that the great majority of the population of individuals with schizophrenia do not receive evidence-based care. In this article, we review the empirical literature on implementation of evidence-based practices for schizophrenia patients. We first examine lessons learned from implementation studies in general medicine. We then summarize the implementation literature specific to schizophrenia, including medication practices, psychosocial interventions, information technology, and state- and federal-level interventions. We conclude with recommendations for future directions. Key words: evidence-based practices/schizophrenia/ implementation research

Introduction Over the past decade, the US Surgeon General’s Report on Mental Health,1 the President’s New Freedom Commission Report,2 Schizophrenia Patient Outcomes Research Team (PORT) project,3 the Texas Medications Algorithm Project (TMAP),4 and several other systematic efforts5–10 have identified a variety of evidence-based interventions for persons with schizophrenia. Examples of effective interventions include systematic approaches to medication management, assertive community treatment, relapse prevention programs, and supported employment. Despite robust evidence on effective interventions, epidemiologic and clinical surveys have established that

Implementation Research in General Medicine Overview Shojania and Grimshaw18 summarized the general medical literature on quality improvement efforts with 4 conclusions: First, the standard approach of passive diffusion of research (ie, publication of research findings in professional journals), including dissemination of findings on effective interventions, has little or no impact on routine practice. Second, more complex efforts to synthesize research evidence in the form of systematic reviews and disseminated guidelines also have little or no effect on practice. Third, adopting total quality management/continuous quality improvement techniques from industry has produced modest but disappointing results. Quality improvement, popularized by Deming,19 is a process that

1 To whom correspondence should be addressed; Psychiatric Research Center, 2 Whipple Place, Lebanon, NH 03766, tel: 603448-0263, fax: 603-448-3976, e-mail: Robert.E.Drake@dartmouth. edu.

Ó The Author 2009. Published by Oxford University Press on behalf of the Maryland Psychiatric Research Center. All rights reserved. For permissions, please email: [email protected].

704

Evidence-Based Practices for Schizophrenia Patients

focuses on training, education, and using data to enhance the performance of an organization.20,21 In mental health treatment settings, quality improvement has taken the form of field-based supervision and systematic review of patient outcomes.22–24 Fourth, current attempts at complete systems reengineering using information technology have produced mixed results, including many prominent successes, which need to be understood in greater detail, as discussed below. Disease Management Systems and the Chronic Care Model Health maintenance organizations have widely adopted disease management systems that emphasize activating patients for self-management, ensuring that clinical teams have the relevant expertise and access to evidence-based guidelines, and monitoring patients without face-to-face contacts.25 Wagner’s26–28 chronic care model represents the most clearly articulated and widely studied approach to disease management. It explicitly aims to combine the best available implementation strategies. The Wagner model includes 6 components: (1) addressing ‘‘health care organization’’ at the level of leadership, financing, and removing barriers; (2) improving access to ‘‘community resources,’’ such as self-help, self-management classes, and nurse educator services; (3) enhancing ‘‘self-management supports,’’suchashelpingpeopletosetgoals,establishaction plans, identify barriers, and solve problems; (4) establishing ‘‘care monitoring,’’ often by nurses or pharmacists, to monitor response, self-management skills, and treatment by algorithms; (5) providing ‘‘decision supports,’’ which incorporate evidence-based guidelines into registries, flow charts, and assessments, to clinical teams; and (6) instituting ‘‘clinical informationsystems’’thatincludeassessments,treatmentplanning, services, outcomes, and algorithms. Many disease management programs include all 6 components, but others include only a portion of Wagner’s model. The evidence shows that disease management approaches improve quality of care and outcomes across diverse chronic diseases such as diabetes, asthma, cardiovascular disease, and depression.29 Implementing Medication Guidelines for Patients With Schizophrenia Since the original Schizophrenia PORT study in the 1990s, several efforts have targeted implementing effective interventions, or evidence-based practices, for schizophrenia patients on a large scale rather than as small demonstrations. Because changing medication practice involves a different set of providers and mechanisms from changing psychosocial practice, we consider implementing medication guidelines first. Antipsychotic treatments are a mainstay of treatment for schizophrenia, and most people with schizophrenia who are in treatment have access to antipsychotic medication. In the original Schizophrenia PORT study of ad-

herence to recommendations in routine practice, the likelihood of a person hospitalized with schizophrenia receiving an antipsychotic medication was high (89%), but the quality of the antipsychotic prescribing frequently deviated from the evidence-based recommendations (only 62% of inpatients and 29% of outpatients were prescribed antipsychotic medication either within the recommended dosage range or with chart documentation justifying the dosage prescribed).13 Because prescribing medications is an essential component of treatment of schizophrenia and is relatively easy to implement (compared, eg, with mounting assertive community treatment teams or multifamily psychoeducational groups), administrators, payers, and researchers have focused considerable efforts on improving prescribers’ adherence to treatment guidelines. The Texas Medication Algorithm Project The largest of these efforts, the TMAP, began in 1996 with the intent to develop, implement, and evaluate an algorithm-driven treatment approach for adults with major psychiatric disorders treated within the public mental health sector in Texas (http://www.dshs.state.tx. us/mhprograms/tmapover.shtm).30 TMAP incorporated several elements of Wagner’s chronic care model, including patient education, systematic assessments, and clear guidelines. The project encompassed 4 phases: (1) creation of evidence-informed algorithms via consensus conferences that clearly indicated (via a flow diagram) the sequence in which the various medication alternatives should be considered (eg, ‘‘Consider clozapine before polypharmacy’’), (2) a feasibility trial using the algorithms with early adopters to estimate their clinical impact and determine the resources needed to implement them, (3) a prospective comparison of the clinical outcomes and economic costs of using the algorithms vs treatment as usual in outpatient public mental health clinics, and (4) a broad implementation of the algorithms throughout the public mental health system in Texas, called the Texas Implementation of Medication Algorithms. The goals of TMAP were simple but far-reaching: to encourage measurement-based medicine to optimize patients’ outcomes. By combining sequential treatment algorithms with standardized measurements of symptoms and functioning at each visit, the prescriber could track progress or lack thereof over time to determine whether a medication change was indicated. A wide variety of stakeholders followed TMAP closely. Patients, family members, payers, and clinicians were concerned about wide variations in prescribing practices and lauded TMAP for seeking to reduce unwarranted variation and ensure access to evidence-based decisions regarding medication management. Many other states, counties, and systems of care modeled algorithm-driven interventions after TMAP, even before results of the prospective comparison study were available. 705

R. E. Drake et al.

In the prospective comparison study, after 3 months of treatment, patients with schizophrenia who received treatment in the sites that were trained and staffed to use the TMAP algorithms had greater improvement in symptoms (about 3 points on the 18-item Brief Psychiatric Rating Scale, after transformations of the scores to adjust for baseline differences and other factors known to affect health outcomes) than did patients in the comparison sites, but this difference in transformed scores disappeared over time.4 Patients in both algorithm sites and nonalgorithm sites showed improvement over time in test scores measuring cognitive functioning, with the patients in the algorithm sites showing greater improvement that was sustained as of the final (9 mo) measurement of cognitive functioning. Whether this difference in measures of cognition translated into differences in functioning in everyday life is not known. Because the impact of the score transformations used was so profound (eg, using raw scores reported in table 2 of the 2004 report, after 3 mo of treatment the symptom scores for the algorithm sites improved by about 5 points compared with improvements of 6 points at comparison sites that were implementing an algorithm for a disorder other than schizophrenia and improvements of about 6 points at other comparison sites), TMAP analyses may best be considered unfinished. Further exploration of this valuable dataset could provide much needed information on, eg, whether patients of prescribers who became more guideline adherent over time showed greater improvements in outcomes than did patients of prescribers who remained less adherent. Given the large impact in this study of score transformations on outcomes, sensitivity analyses showing the impact of various transformations could inform interpretation. By design, the TMAP intervention included components to support and encourage the use of the algorithms in addition to the algorithms themselves. The TMAP investigators recognized that passive diffusion of algorithms was unlikely to change practice. The intervention package at a site included the provision of a clinical coordinator to work with physicians and families, treatment manuals, expert consultations through conference calls and site visits, family education programs, standardized assessments of symptoms and side effects, as well as the enhanced record keeping procedures that accompanied use of the algorithms.31,32 The understandable confounding of these intervention components means that one cannot know the extent to which any of the individual components were responsible for TMAP’s findings. What at first blush looks like an expensive intervention (training across sites, implementing measurement tools for prescribers to use to monitor changes over time, and placement of a clinician in each site to work with physicians to help them follow the guidelines) for a modest impact may be too pessimistic a conclusion. No doubt, adherence to the guidelines varied across prescrib706

ers, even within the algorithm sites. A stronger test of the payoff of changing prescriber behavior to more closely follow the treatment algorithm would be to determine whether patients of psychiatrists who followed the algorithm more closely had better outcomes than patients of psychiatrists who followed the algorithms less closely. This remains a challenging analysis for a multitude of reasons. Nevertheless, knowing whether and to what extent efforts to enhance adherence to guidelines improves patient outcomes is critical for policy makers who must decide if they should go to the effort of implementing and sustaining the use of such algorithms. Beyond TMAP The zeitgeist surrounding TMAP prompted public and private payers to identify questionable patterns of prescribing and to intervene and monitor the impact of such interventions. The Psychiatric Clinical Knowledge Enhancement System in New York provides detailed current medication regimens and histories at the patient, psychiatrist, ward, and facility levels and has been used to decrease rates of antipsychotic polypharmacy.33,34 In another effort aimed at increasing prescriber adherence to evidence-based guidelines, a statewide longitudinal project in Kentucky examined the impact of prescriber training and systematic monitoring of medication practices, using Medication Management Approaches in Psychiatry (MedMAP) (mentalhealth.samhsa.gov/cmhs/ communitysupport/toolkits/community/), which is an adaptation of the TMAP schizophrenia module. The Kentucky Project used the MedMAP Fidelity Scale35 as a quality improvement tool and documented modest improvement in prescribing practices over a 16-month period.36 The last decade has also seen the emergence of pharmacy management companies that contract with employers to review patterns of utilization, establish preferred drug formularies to steer patients toward/away from particular costly and/or more effective medications, and send letters to patients or prescribers alerting them to potential concerns about the medication regimens used. The extent to which steering is influenced by cost vs quality concerns typically cannot be determined because the full list of such managed care firms’ algorithms typically is proprietary, making it difficult for the payer or a neutral third party to evaluate the attention paid to cost vs quality of care by such algorithms. Firms may use polypharmacy as an example of a clinically questionable practice their guidelines target but offer few other specifics. While physicians can be offended by receiving ‘‘Dear Prescriber’’ letters from such pharmacy management firms and suspect that the goal is cost containment as opposed to quality improvement, one also can imagine situations in which strong evidence supports the use of timely claims data to flag clinically questionable situations for review (eg, no use of an antipsychotic medication following


a hospitalization for schizophrenia). Transparency via independent review and broad opportunity for public comment helps quell understandable concerns that algorithms could be promoting cost savings at the expense of improving clinical outcomes. The business of promoting adherence to particular practices is rife with controversy as well as essential for improving care.37 Research on Implementing Psychosocial Interventions Dissemination of model psychosocial programs for individuals with schizophrenia has a long history, dating back to the 1970s with the failed effort to promote the use of community lodges for patients discharged from psychiatric hospitals.38 In the late 1970s, the National Institute of Mental Health (NIMH) began funding the dissemination and evaluation of promising models based on Community Support Program principles to promote community integration of individuals with serious mental illness.39 Many of these projects produced disappointing results, in part due to poor model specification,40,41 inadequate implementation plans,42 lack of stakeholder support for the dissemination,43 and inadequate leadership.44 Many projects launched during this era had naive assumptions about the minimum requirements for effective implementation. The challenges of implementing complex psychosocial interventions are much greater than those for medication interventions.45 One positive outcome from the Community Support Program demonstration projects was increasing awareness of the need to define adherence to program models, which in turn required clearly defined models. Building on the work on treatment integrity in the psychotherapy literature,46,47 researchers began developing fidelity measures, defined as methods to assess adherence to the standards of a program model.48 Among the earliest of these efforts were measures to assess adherence to the assertive community treatment model.49–53 Assertive community treatment was the most clearly defined of the psychosocial models at the time.54 In the 1990s, supported employment began accumulating a strong evidence base, and a fidelity scale was developed and validated for this model as well.55,56 For supported employment, fidelity has been found to be associated with higher competitive employment rates.57–63 For assertive community treatment, the correlation between fidelity and improved outcomes has been less consistent.51,64–66 Over the last few years, there has been greater emphasis on the systematic study of the implementation process. In 2006, a new journal was established devoted to this enterprise (http://www.implementationscience.com). Accordingly, the literature on strategies and barriers to implementing evidence-based practices has expanded dramatically.45,67 With regard to schizophrenia, recent projects employing systematic implementation strategies include the National Implementing Evidence-Based Practices Project, the Mental Health Treatment Study,

the Department of Veterans Affairs (VA) dissemination of supported employment, several learning communities, and a variety of implementation studies outside the United States. National Implementing Evidence-Based Practices Project In 1998, a national panel of experts convened by the Robert Wood Johnson Foundation recommended that 5 psychosocial practices and systematic medication management be offered in every community mental health center.8 The panel also recommended the development of a systematic approach to dissemination. The National Implementing Evidence-Based Practices Project was launched to address the aforementioned deficiencies.68–72 The investigators hypothesized that implementation of evidence-based practices in routine settings minimally required comprehensive, user-friendly information about the practices and their implementation—resources generally unavailable in earlier dissemination efforts. Therefore, the first phase of the National Evidence-Based Practices Project involved creation of toolkits for each practice, consisting of a variety of materials to facilitate practice implementation, such as practitioner workbooks, research articles, introductory and instructional video, and PowerPoint lectures. The toolkits aimed at multiple stakeholders, assuming that success depended on active support from the state mental health authority, agency leadership, practitioners, consumers, and family members. The investigators also hypothesized that, in addition to the toolkits, another necessary component for successful implementation was systematic training and consultation. Thus, the project researchers developed a trainingconsultation model that included the following elements71: (1) consultation to the state mental health authority, (2) consultation to community mental health center administrators, (3) a kickoff presentation to the provider agency, (4) provision of the evidence-based practice toolkit to the agency, (5) initial skill training for practitioners, (6) ongoing consultation to sites, and (7) systematic fidelity monitoring. Fidelity monitoring included 2 steps: (a) fidelity assessments based on daylong site visits by a pair of independent fidelity assessors who, guided by fidelity scale criteria, conduct structured interviews, observe team meetings and interventions, and review medical records and (b) fidelity review meetings held with agency leaders in which assessors give oral and written feedback on quality of implementation.73 The second phase of the project was a field test of 5 psychosocial evidence-based practices in 53 sites in 8 states. The sites were studied over a 2-year period. Assessing clinical outcomes would have been prohibitively expensive; therefore, the study focused on fidelity scores at baseline and 6-month intervals. Findings from this project are now appearing in the literature.24,73–84 Overall, 29 (55%) sites showed high-fidelity implementation at 2-year follow-up.79 Most sites achieving high 707

R. E. Drake et al.

fidelity did so within the first year of implementation. Two of the evidence-based practices (assertive community treatment and supported employment) were more readily implemented than others. Based on qualitative data collected within each site, several factors influenced quality of implementation, including the state mental health authority’s provision of leadership, funding, and practice standards76,80,85; skilled mentoring by trainer-consultants24; administrative support and competent clinical supervision at the site level24,74–77,81; systematic monitoring of fidelity and outcomes73,78; and staff turnover.83 Mental Health Treatment Study In 2006, the Social Security Administration launched a 23site randomized controlled trial comparing an experimental program to usual services for Social Security Disability Insurance (SSDI) beneficiaries with schizophrenia or affective disorder.86 Beneficiaries in the experimental condition receive a comprehensive package of services including supported employment, systematic medication management, and other behavioral services. A nurse care coordinator assigned in each site helps to ensure that services are individualized and evidence-based. The experimental group also receives supplemental insurance to cover charges for all necessary health care. Building on the experiences of the National Implementing Evidence-Based Practices Project, a quality management team is monitoring fidelity and providing technical assistance to the sites. Many of the same themes have emerged from this project as in earlier implementation studies. Barriers include the maze of state bureaucratic regulations even in the face of a well-funded project, the pivotal role of site leadership, the challenges of integrating supported employment with mental health treatment services, and staff turnover. Department of VA Like an earlier dissemination of the assertive community treatment model,42 current efforts by the Veterans Health Administration of the Department of VA to disseminate supported employment are ambitious in their national scope.87–89 Organizational barriers include lack of resources for supported employment, fragmentation of services (with individual clinicians working independently), and VA traditions and policies not aligned with evidence-based supported employment. In addition, longstanding sheltered and transitional work programs within the VA conflict with the principles of supported employment. This project depends on mostly remote supervision and monitoring, which may dilute the impact of technical assistance and quality improvement efforts.90 Learning Community Approaches Borrowing from higher education, some groups of healthcare organizations have adopted a learning community approach to quality, forming networks among key staff 708

from these organizations that share common values and beliefs and are actively engaged in learning together from each other.91 Learning communities are especially suited to promoting sustained adherence to a practice as well as continuous quality improvement by creating a culture of peer accountability and sharing among participating organizations. One such application to schizophrenia has been a network of state and local leaders aimed at disseminating evidence-based supported employment.92 First established in 2002, the network has now grown to include participants from 10 states and the District of Columbia. Using familiar strategies such as annual meetings, newsletters, development of videos, on-site trainings, bimonthly teleconferences, and sharing data, this network has stimulated interest in identifying innovative implementation strategies successful in one state that are adapted for use in others.93,94 Program evaluation data suggest growing access to supported employment and achievement of site-level competitive employment rates exceeding 40% for quarterly reporting periods.92 A similar project in New York State, the Wellness Self-management Program (a variation of Illness Management and Recovery95), also uses learning collaboratives.96 Teams from participating sites meet to describe implementation experiences and hear suggestions from colleagues as well as a trainer-facilitator. International Efforts Wide-scale dissemination of evidence-based practices has been attempted in Canada,97 various European countries,98–100 Australia,101 and Japan.102 Common barriers found in these projects have included funding issues, lack of cooperation among different services (eg, vocational rehabilitation, mental health, and substance abuse treatment), staff turnover, and insufficient time allocated for program leaders. Information Technology Both the Institute of Medicine103 and the New Freedom Commission2 recommended using modern information technology to improve the quality of mental health care in America. The United States lags behind other Westernized countries in the use of health information technology,104 and the US mental health system lags behind other areas of American medicine.11 Several recent reviews conclude that appropriate use of information technology can improve quality of health care.105,106 A simple example of psychiatric knowledge enhancement systems is the use of programs that check for medication interactions and incorrect dosages. Other common components include increasing patient input regarding status, concerns, and goals; increasing patient education regarding evidencebased practices; increasing practitioners’ awareness of


evidence-based treatments and algorithms and of patients’ concerns and preferences; identifying and avoiding medical errors; enhancing shared decision making; monitoring outcomes and side effects; and monitoring programs and systems of care.107 Comprehensive electronic decision support systems, which address all these components, are just emerging in mental health. The Veterans Administration has adopted a clinically oriented electronic medical record that facilitates evidence-based care via patient-specific clinical prompts, monitoring of care, and review of outcomes; the system has recently added a patient portal (http:// www.myhealth.va.gov/). Enhancements to the Veterans Administration electronic medical record for mental health are being tested,108 and feasibility testing of direct consumer use of computer systems has shown promising results.109 In several European Union countries, a computerized system that enables patients with schizophrenia and their clinicians to compare their perspectives on goals and negotiate plans has demonstrated improvements in satisfaction, reduced unmet need, and increased quality of life.110 Numerous innovative efforts are underway.111 Deegan et al112 have designed a comprehensive electronic decision support system that is used to facilitate shared decisions during medication visits. Similar systems are being created and tested within the Department of VA to support schizophrenia care as well as care more broadly and at the Dartmouth Psychiatric Research Center to support treatment planning, smoking cessation, employment services, co-occurring disorders treatments, and care of comorbid medical illnesses. The New York State Office of Mental Health also is developing a consumer portal to support decision making around medications. Comprehensive electronic decision support systems must be integrated into real-world contexts to be useful.105,106 Deegan et al112 established a recovery resource center within a routine community mental health center by providing semiprivate computer kiosks, visual and oral information, and peer supports. Patients and clinicians achieved high levels of participation and satisfaction. Of course, much of current community mental health work is done in the community rather than in clinics. Demonstrating that electronic resources can be delivered in other forms, such as through health buddies, in peer support centers, on the web, and on portable information systems that case managers and other clinicians carry with them, will be important.113

The Role of Government State Interventions In the United States, the role of state government in promoting and inhibiting the growth of evidence-based practices is enormous if not decisive.114–117 The reasons are obvious: State agencies, in conjunction with federal agencies (notably the Centers for Medicare and Medicaid

Services) to a large extent determine what services are funded or not funded. In most states, the state mental health authority has sponsored statewide initiatives to promote evidence-based practices. States have aggressively promoted adoption of assertive community treatment,76,80,118 supported employment,93 family psychoeducation,43 integrated dual disorders treatment,119 and medication management approaches.30,33,36 A popular but ineffective approach has been the sponsorship of statewide conferences.120,121 To address the enormous costs of face-to-face meetings, some states are exploring web-based options for training and ongoing supervision. A comprehensive approach to enhancing broad dissemination has often been centered in the formation of technical assistance centers providing consultation, training, and fidelity monitoring.24,118,122,123 Although it is widely assumed that the state mental health authority critically impacts the development of evidencebased practices, most research to date has been anecdotal. To test this hypothesis, a state-level fidelity scale was developed for the National Implementing Evidence-Based Practices Project to measure objective indicators of state mental health authority actions, such as the designation of a point person within the state agency responsible for dissemination, the establishment of a technical assistance center, state-level policies and regulations aligned to support the evidence-based practices, and provision of financial incentives to implement the evidence-based practices. This statelevel fidelity scale was strongly correlated with mean fidelity for the evidence-based practices in each state.85 To date, most studies of statewide implementation have examined early stages of dissemination in which enthusiasm and other Hawthorne effects abound. Much less is known regarding sustaining a statewide initiative, especially in the face of the frequent leadership changes in senior staff in many state agencies due to postelection changes in politicalappointees.Fromthestandpointofthedirectorsofcommunity agencies, decisions about adopting evidence-based practices are influenced by judgments about the perceived riskiness of the uptake, anticipated resource availability, and exposure to evidence.124,125 Early adopters are willing to mount the initiatives necessary for adoption because they see the risks associated with adopting as lower and more manageable than nonadopters.124,125 Discontinuation is driven in part by staff turnover126; hence, e-learning approaches that are permanently and conveniently accessible can also help deal with this challenge. Federal Interventions At the federal level, the 12% set-aside of the NIMH budget for services research helped launch TMAP and other studies of the effectiveness of various treatment approaches. That set-aside is long gone, but, as this article is being written, the current version of the federal economic stimulus package contains $400 million to the Agency for Healthcare Research and Quality as well as other funds 709

R. E. Drake et al.

to stimulate comparative effectiveness research. As the largest single payer for health care in the nation, the Federal government can make significant investments in narrowing the quality chasm between what is known and what is practiced. In theory, NIMH generates new knowledge, and the Substance Abuse and Mental Health Services Administration helps such knowledge make its way to real people in real-world settings by promoting ways across this chasm. Knowing how to promote such uptake and where one’s efforts are best invested are themselves pressing research issues. The Social Security Administration also has shown an interest in promoting evidencebased employment programs. It sponsored an expert panel to identify a package of evidence-based interventions to help SSDI beneficiaries with mental illness to return to work.127 The work of this panel led to the funding of the Mental Health Treatment Study described earlier. The Social Security Administration has also sought to incentivize employment services through its Ticket to Work program.128 The President’s New Freedom Commission was an inspirational product of the executive branch of the federal government, even if the executive branch also required that the Commission’s recommendations be revenue neutral. Revenue neutrality, especially in the short term, may be incompatible with implementing evidence-based practices unless a comparable amount of ineffective services can be discontinued. Summary and Conclusions Implementation research in schizophrenia care has followed the pattern observed in general health care18: moving gradually from passive diffusion to system reengineering based on complex electronic records, decision supports, and Wagner’s26 chronic care model. Simple implementation efforts are often fruitless and waste resources, while traditional continuous quality improvement approaches are costly and often only moderately successful. Complex reengineering of systems is needed. However, public mental health systems are currently mired in financing constraints (eg, Medicaid regulations are not aligned with evidence-based practices), economic survival problems (state budget problems have eroded public mental health funding), workforce problems (average tenure in some front-line positions is less than 18 mo), regulatory problems (fears of Medicaid audits prevent systems from innovating), and information technology problems (currently many states have no electronic medical records while others have electronic medical records that emphasize billing and regulatory requirements rather than quality of services and clinical outcomes). Public mental health systems need better alignment between evidence-based practices and payments, sufficient funding to create a sustainable and professional workforce, electronic medical records to monitor process and outcomes, and a systemic commitment to quality. The research commu710

nity needs much greater attention to the implementation of effective services in order to improve the care of schizophrenia in the United States. References 1. U.S. Surgeon General. Mental Health: A Report of the Surgeon General—Executive Summary. Rockville, MD: U.S. Department of Health and Human Services, Substance Abuse and Mental Health Services Administration, Center for Mental Health Services, National Institutes of Health, National Institute of Mental Health; 1999. 2. New Freedom Commission on Mental Health. Achieving the Promise: Transforming Mental Health Care in America. Final Report. Rockville, MD: Substance Abuse and Mental Health Services Administration, 2003. DHHS Pub No SMA-03-3832. 3. Lehman AF, Kreyenbuhl J, Buchanan RW, et al. The Schizophrenia Patient Outcomes Research Team (PORT): updated treatment recommendations 2003. Schizophr Bull. 2004;30:193–217. 4. Miller AL, Crismon ML, Rush AJ, et al. The Texas Medication Algorithm Project: clinical results for schizophrenia. Schizophr Bull. 2004;30:627–647. 5. Canadian Psychiatric Association Working Group. Clinical practice guidelines: treatment of schizophrenia. Can J Psychiatry. 2005;50:1S–56S. 6. Chambless D, Baker M, Baucom D, et al. Update on empirically validated therapies, II. Clin Psychol. 1998;51:3–16. 7. Cook JA. Blazing trails: using evidence-based practice and stakeholder consensus to enhance psychosocial rehabilitation services in Texas. Psychiatr Rehabil J. 2004;27:305–318. 8. Drake RE, Merrens MR, Lynde DW, eds. Evidence-Based Mental Health Practice: A Textbook. New York, NY: WW Norton & Company; 2005. 9. National Institute for Health and Clinical Excellence. NICE Clinical Guideline 82: Schizophrenia: Core Interventions in the Treatment and Management of Schizophrenia in Adults in Primary and Secondary Care 2009; www.nice.org.uk/ CG082. Accessed April 11, 2009. 10. Thornicroft G, Susser E. Evidence-based psychotherapeutic interventions in the community care of schizophrenia. Br J Psychiatry. 2001;178:2–4. 11. Mojtabai R, Fochtman L, Chang S, Kotov K, Craig TJ, Bromet E. Unmet need for care in schizophrenia. Schizophr Bull. 2009; doi:10.1093/schbul/sbp045. 12. Wang PS, Demler O, Kessler RC. Adequacy of treatment for serious mental illness in the United States. Am J Public Health. 2002;92:92–98. 13. Lehman AF, Steinwachs DM. Patterns of usual care for schizophrenia: initial results from the Schizophrenia Patient Outcomes Research Team (PORT) client survey. Schizophr Bull. 1998;24:11–23. 14. SAMHSA. 2005 National Survey on Drug Use and Health 2006;Rockville, MD: Substance Abuse and Mental Health Services Administration, Office of Applied Studies. 15. Cunningham P, McKenzie K, Taylor EF. The struggle to provide community-based care to low-income people with serious mental illness. Health Aff. 2006;25:694–705. 16. NAMI. Grading the States: A Report on America’s Health Care System for Serious Mental Illness 2006;Arlington, VA: National Alliance on Mental Illness.


17. Young GJ, Mohr DC, Meterko M, Siebert M, McGlynn G. Psychiatrists’ self reported adherence to evidence-based prescribing practices in the treatment of schizophrenia. Psychiatr Serv. 2006;57:130–132. 18. Shojania KG, Grimshaw JM. Evidence-based quality improvement: the state of the science. Health Aff. 2005; 24:138–150. 19. Deming WE. Out of Crisis. Cambridge, MA: Center for Advanced Engineering Study, Massachusetts Institute of Technology, 1986. 20. Nadler DA. Feedback and Organization Development. Using Data Based Methods. Reading, MA: Addison-Wesley, 1977. 21. Shannon HS, Robson LS, Sale JE. Creating safer and healthier workplaces: role of organizational factors and job characteristics. Am J Ind Med. 2001;40:319–334. 22. Gowdy E, Rapp CA. Managerial behavior: the common denominators of successful community based programs. Psychosoc Rehabil J. 1989;13:31–51. 23. Harkness DR, Hensley H. Changing the focus of social work supervision: effects on client satisfaction and generalized contentment. Soc Work. 1991;36:506–512. 24. Rapp CA, Etzel-Wise D, Marty D, et al. Evidence-based practice implementation strategies: results of a qualitative study. Community Ment Health J. 2008;44:213–224. 25. Todd WE, Nash D. Disease Management: A Systems Approach to Improving Patient Outcomes. San Francisco, CA: Jossey-Bass, 1997. 26. Wagner EH, Austin BT, Davis C, Hindmarsh M, Schaefer J, Bonomi A. Improving chronic illness care: translating evidence into action. Health Aff. 2001;20:64–78. 27. Wagner EH, Glasgow RE, Davis C, et al. Quality improvement in chronic illness care: a collaborative approach. Jt Comm J Qual Improv. 2001;27:63–80. 28. Wagner EH, Bennett SM, Austin BT, Greene SM, Schaefer JK, Vonkorff M. Finding common ground: patientcenteredness and evidence-based chronic illness care. J Altern Complement Med. 2005;11(Suppl 1):S7–S15. 29. Coleman K, Mattke S, Perrault PJ, Wagner EH. Untangling practice redesign from disease management. How do we best care for the chronically ill? Annu Rev Public Health. 2009;30:1.1–1.24. 30. Miller AL, Chiles JA, Chiles JK, Crismon ML, Rush AJ, Shon SP. The Texas Medication Algorithm Project (TMAP) schizophrenia algorithms. J Clin Psychiatry. 1999;60:649–657. 31. Toprac MG, Dennehy EB, Carmody TJ, et al. Implementation of the Texas Medication Algorithm Project Patient and Family Education Program. J Clin Psychiatry. 2006; 67:1362–1372. 32. Toprac MG, Rush AJ, Conner TM, et al. The Texas Medication Algorithm Project Patient and Family Education Program: a consumer-guided initiative. J Clin Psychiatry. 2000;61:477–486. 33. Finnerty M, Altmansberger R, Bopp J, et al. Using state administrative and pharmacy databases to develop a clinical decision support tool for schizophrenia guidelines. Schizophr Bull. 2002;28:85–94. 34. Uttaro T, Finnerty M, White T, Gaylor R, Shindelman L. Reduction of concurrent antipsychotic prescribing practices through the use of PSYCKES. Adm Policy Ment Health. 2007;34:57–61. 35. Taylor AC, Bond GR, Tsai J, et al. A scale to evaluate quality of medication management: development and psycho-

36.

37.

38. 39.

40.

41.

42.

43.

44.

45.

46. 47.

48.

49. 50.

51.

52.

53.

54.

55.

metric properties [published online ahead of print February 27, 2009]. Adm Policy Ment Health. 2009; doi: 10.1007/s10488-009-0209-z. Howard PB, El-Mallakh P, Miller AL, et al. Prescriber fidelity to a medication management evidence-based practice in the treatment of schizophrenia. Psychiatr Serv. In press. Tanenbaum SJ. Evidence-based practice as mental health policy: three controversies and a caveat. Health Aff. 2005;24:163–173. Fairweather GW. The Fairweather lodge: a twenty-five year retrospective. New Dir Ment Health Serv. 1980;7:3–100. Turner JC, TenHoor WJ. The NIMH community support program: pilot approach to a needed social reform. Schizophr Bull. 1978;4:319–348. Brekke JS. What do we really know about community support programs? Strategies for better monitoring. Hosp Community Psychiatry. 1988;39:946–952. Noble JH. The Benefits and Costs of Supported Employment for People With Mental Illness and With Traumatic Brain Injury in New York State. Buffalo, NY: Research Foundation of the State University of New York; 1991. Report No: C-0023180. Rosenheck R, Neale M, Leaf P, Milstein R, Frisman L. Multisite experimental cost study of intensive psychiatric community care. Schizophr Bull. 1995;21:129–140. McFarlane WR, McNary S, Dixon L, Hornby H, Cimett E. Predictors of dissemination of family psychoeducation in community mental health centers in Maine and Illinois. Psychiatr Serv. 2001;52:935–942. Backer T, Liberman R, Kuehnel T. Dissemination and adoption of innovative psychosocial interventions. J Consult Clin Psychol. 1986;54:111–118. Fixsen DL, Naoom SF, Blase KA, Friedman RM, Wallace F. Implementation Research: A Synthesis of the Literature. Tampa, FL: University of South Florida; 2005. Moncher FJ, Prinz RJ. Treatment fidelity in outcome studies. Clin Psychol Rev. 1991;11:247–266. Waltz J, Addis ME, Koerner K, Jacobson NS. Testing the integrity of a psychotherapy protocol: assessment of adherence and competence. J Consult Clin Psychol. 1993;61:620–630. Bond GR, Evans L, Salyers MP, Williams J, Kim HK. Measurement of fidelity in psychiatric rehabilitation. Ment Health Serv Res. 2000;2:75–87. Brekke JS. The model-guided method for monitoring program implementation. Eval Rev. 1987;11:281–299. Essock SM, Kontos N. Implementing assertive community treatment teams. Hosp Community Psychiatry. 1995;46: 679–683. McGrew JH, Bond GR, Dietzen LL, Salyers MP. Measuring the fidelity of implementation of a mental health program model. J Consult Clin Psychol. 1994;62:670–678. Teague GB, Drake RE, Ackerson TH. Evaluating use of continuous treatment teams for persons with mental illness and substance abuse. Psychiatr Serv. 1995;46:689–695. Teague GB, Bond GR, Drake RE. Program fidelity in assertive community treatment: development and use of a measure. Am J Orthopsychiatry. 1998;68:216–232. McGrew JH, Bond GR. Critical ingredients of assertive community treatment: judgments of the experts. J Ment Health Adm. 1995;22:113–125. Becker DR, Swanson S, Bond GR, Merrens MR. EvidenceBased Supported Employment Fidelity Review Manual. Lebanon, NH: Dartmouth Psychiatric Research Center, 2008.

711

R. E. Drake et al.

56. Bond GR, Becker DR, Drake RE, Vogler KM. A fidelity scale for the individual placement and support model of supported employment. Rehabil Couns Bull. 1997;40:265–284. 57. Becker DR, Smith J, Tanzman B, Drake RE, Tremblay T. Fidelity of supported employment programs and employment outcomes. Psychiatr Serv. 2001;52:834–836. 58. Becker DR, Xie H, McHugo GJ, Halliday J, Martinez RA. What predicts supported employment program outcomes? Community Ment Health J. 2006;42:303–313. 59. Catty J, Lissouba P, White S, et al. Predictors of employment for people with severe mental illness: results of an international six-centre RCT. Br J Psychiatry. 2008;192:224–231. 60. Gowdy EA, Carlson LS, Rapp CA. Practices differentiating high-performing from low-performing supported employment programs. Psychiatr Rehabil J. 2003;26:232–239. 61. Hayward BJ, Schmidt-Davis H. Longitudinal Study of the Vocational Rehabilitation Services Program Final Report: VR Services and Outcomes (Report to the Rehabilitation Services Administration Under ED Contract No. HR92022001). Research Triangle Park, NC: RTI International; 2003. 62. McGrew J, Griss M. Concurrent and predictive validity of two scales to assess the fidelity of implementation of supported employment. Psychiatr Rehabil J. 2005;29:41–47. 63. McGrew JH. IPS fidelity survey of 17 supported employment programs in Indiana: final report to SECT Center. Indianapolis, IN: Indiana University-Purdue University Indianapolis; 2007. 64. Burns T, Catty J, Dash M, Roberts C, Lockwood A, Marshall M. Use of intensive case management to reduce time in hospital in people with severe mental illness: systematic review and meta-regression. Br Med J. 2007;335:336. 65. Bond GR, Salyers MP. Prediction of outcome from the Dartmouth ACT Fidelity Scale. CNS Spectr. 2004;9:937–942. 66. McHugo GJ, Drake RE, Teague GB, Xie H. Fidelity to assertive community treatment and client outcomes in the New Hampshire Dual Disorders Study. Psychiatr Serv. 1999; 50:818–824. 67. Bhattacharyya O, Reeves S, Zwarenstein M. What is implementation research? Rationale, concepts and practices. Res Soc Work Pract. In press. 68. Drake RE, Goldman HH, Leff HS, et al. Implementing evidence-based practices in routine mental health service settings. Psychiatr Serv. 2001;52:179–182. 69. Mueser KT, Torrey WC, Lynde D, Singer P, Drake RE. Implementing evidence-based practices for people with severe mental illness. Behav Modif. 2003;27:387–411. 70. Torrey WC, Drake RE, Dixon L, et al. Implementing evidence-based practices for persons with severe mental illness. Psychiatr Serv. 2001;52:45–50. 71. Torrey WC, Finnerty M, Evans A, Wyzik PF. Strategies for leading the implementation of evidence-based practices. Psychiatr Clin North Am. 2003;26:883–897. 72. Torrey WC, Lynde DW, Gorman P. Promoting the implementation of practices that are supported by research: the National Implementing Evidence-Based Practice Project. Child Adolesc Psychiatr Clin N Am. 2005;14:297–306. 73. Bond GR, Drake RE, McHugo GJ, Rapp CA, Whitley R. National Evidence-Based Practices Project Research Group. Strategies for improving fidelity in the National EvidenceBased Practices Project. Res Soc Work Pract. In press. 74. Bond GR, McHugo GJ, Becker DR, Rapp CA, Whitley R. Fidelity of supported employment: lessons learned from the

712

75.

76.

77.

78.

79.

80.

81.

82.

83.

84.

85.

86. 87.

88.

89.

90. 91. 92.

93.

National Evidence-Based Practices Project. Psychiatr Rehabil J. 2008;31:300–305. Brunette MF, Asher D, Whitley R, et al. Implementation of integrated dual disorders treatment: a qualitative analysis of facilitators and barriers. Psychiatr Serv. 2008;59:989–995. Mancini AD, Moser LL, Whitley R, et al. Assertive community treatment: facilitators and barriers to implementation in routine mental health settings. Psychiatr Serv. 2009;60: 189–195. Marshall T, Rapp CA, Becker DR, Bond GR. Key factors for implementing supported employment. Psychiatr Serv. 2008;59:886–892. Marty D, Rapp CA, McHugo G, Whitley R. Factors influencing consumer outcome monitoring in implementation of evidence-based practices: results from the National EBP Implementation Project. Adm Policy Ment Health. 2008; 35:204–211. McHugo GJ, Drake RE, Whitley R, et al. Fidelity outcomes in the National Implementing Evidence-Based Practices Project. Psychiatr Serv. 2007;58:1279–1284. Moser LL, DeLuca NL, Bond GR, Rollins AL. Implementing evidence based psychosocial practices: lessons learned from statewide implementation of two practices. CNS Spectr. 2004;9:926–936. Rapp CA, Etzel-Wise D, Marty D, et al. Barriers to evidence-based practice implementation: results of a qualitative study. Community Ment Health J. In press. Whitley R, Gingerich S, Lutz W. Facilitators and barriers to the implementation of the illness management and recovery program in community mental health settings. Psychiatr Serv. 2009;60:202–209. Woltmann EM, Whitley R, McHugo GJ, et al. The role of staff turnover in the implementation of evidence-based practices in mental health care. Psychiatr Serv. 2008;59:732–737. Woltmann E, Whitley R. The role of staffing stability in the implementation of integrated dual disorders treatment: an exploratory study. J Ment Health. 2007;16:697–701. Finnerty MT, Rapp CA, Bond GR, Lynde DW, Goldman HH. The State Health Authority Yardstick (SHAY) [published online ahead of print March 21, 2009]. Community Ment Health J. 2009; doi: 10.1007/s10597-009-9181-z. Frey W, Azrin S, Goldman HH, et al. The Mental Health Treatment Study. Psychiatr Rehabil J. 2008;31:306–312. Resnick SG, Rosenheck RA. Scaling up the dissemination of evidence-based mental health practice to large systems and long-term time frames. Psychiatr Serv. In press. Resnick SG, Rosenheck R. Dissemination of supported employment in Department of Veterans Affairs. J Rehabil R D. 2007;6:867–878. Rosenheck RA, Mares AS. Dissemination of supported employment for homeless veterans with psychiatric and/or addiction disorders: two-year client outcomes. Psychiatr Serv. 2007;58:325–333. Bond GR. Modest implementation efforts, modest fidelity, and modest outcomes. Psychiatr Serv. 2007;58:334. Ovretveit J, Bate P, Cleary P, et al. Quality collaboratives: lessons from research. Qual Saf Health Care. 2002;11:345–351. Drake RE, Becker DR, Goldman HH, Martinez RA. Best practices: the Johnson & Johnson—Dartmouth Community Mental Health Program: disseminating evidence-based practice. Psychiatr Serv. 2006;57:302–304. Becker DR, Lynde D, Swanson SJ. Strategies for state-wide implementation of supported employment: the Johnson &


94.

95.

96.

97.

98.

99. 100.

101.

102.

103.

104.

105.

106.

107.

108.

109.

110.

111.

112.

Johnson—Dartmouth Community Mental Health Program. Psychiatr Rehabil J. 2008;31:296–299. Becker DR, Baker SR, Carlson L, et al. Critical strategies for implementing supported employment. J Vocat Rehabil. 2007;27:13–20. Mueser KT, Corrigan PW, Hilton DW, et al. Illness management and recovery: a review of the research. Psychiatr Serv. 2002;53:1272–1284. Institute for Healthcare Improvement. The Breakthrough Series: IHI’s Collaborative Model for Achieving Breakthrough Improvement. IHI Innovation Series White Paper (www.IHI.org). Boston, MA: Institute for Healthcare Improvement; 2003. Corbie`re M, Bond GR, Goldner E, Ptasinski T. The fidelity of supported employment implementation in Canada and the United States. Psychiatr Serv. 2005;56:1444–1447. Boyce M, Secker J, Floyd M, Schneider J, Slade J. Factors influencing the delivery of evidence-based supported employment in England. Psychiatr Rehabil J. 2008;31:360–366. Rinaldi M, Perkins R. Implementing evidence-based supported employment. Psychiatr Bull. 2007;31:244–249. van Erp NH, Giesen FB, van Weeghel J, et al. A multisite study of implementing supported employment in The Netherlands. Psychiatr Serv. 2007;58:1421–1426. Waghorn G, Collister L, Killackey E, Sherring J. Challenges to implementing evidence-based supported employment in Australia. J Vocat Rehabil. 2007;27:29–37. Ito J, Oshima I, Nishio M, Kuno E. Initiative to build a community-based mental health system including assertive community treatment for people with severe mental illness in Japan. Am J Psychiatr Rehabil. In press. Institute of Medicine. Improving the Quality of Health Care for Mental and Substance-Use Conditions. Washington, DC: National Academies Press, 2006. Schoen C, Osborn R, Huynh PT, Doty M, Peugh J, Zapert K. On the front lines of care: primary care doctors’ office systems, experiences, and views in seven countries. Health Aff. 2006;25:w555–w571. Dorr D, Bonner L, Cohen A, et al. Informatics systems to promote improved care for chronic illness: a literature review. J Am Med Inform Assoc. 2007;14:156–163. Garg AX, Adhikari NK, McDonald H, et al. Effects of computerized clinical decision support systems on practitioner performance and patient outcomes: a systematic review. J Am Med Assoc. 2005;293:1223–1238. Drake RE, Teague GB, Gersing K. State mental health authorities and informatics. Community Ment Health J. 2005;41: 365–370. Owen RR. Department of Veterans Affairs Quality Enhancement Research Initiative Annual Report. Little Rock, AR: Center for Mental Healthcare and Outcomes Research; 2008. Chinman M, Hassell J, Magnabosco J, Nowlin-Finch N, Marusak S, Young AS. The feasibility of computerized patient self-assessment at mental health clinics. Adm Policy Ment Health. 2007;34:401–409. Priebe S, McCabe R, Bullenkamp J, et al. Structured patientclinician communication and 1-year outcome in community mental healthcare. Br J Psychiatry. 2007;191:420–426. Drake RE, Wilkniss SM, Frounfelker RL, et al. The ThresholdsDartmouth partnership on shared decision making. Psychiatr Serv. 2009;60:142–144. Deegan PA, Rapp C, Holter M, Riefer M. Best practices: a program to support shared decision making in an

113.

114.

115.

116.

117.

118.

119.

120.

121.

122.

123.

124.

125.

126.

127.

128.

outpatient psychiatric medication clinic. Psychiatr Serv. 2008;59:603–605. Ruland CM. Handheld technology to improve patient care: evaluating a support system for preference-based care planning at the bedside. J Am Med Inform Assoc. 2002;9:192–201. Ganju V. Implementation of evidence-based practices in state mental health systems: implications for research and effectiveness studies. Schizophr Bull. 2003;29:125–131. Isett KR, Burnam MA, Coleman-Beattie B, et al. The state policy context of implementation issues for evidence-based practices in mental health. Psychiatr Serv. 2007;58:914–921. Isett KR, Burnam MA, Coleman-Beattie B, et al. The role of state mental health authorities in managing change for the implementation of evidence-based practices. Community Ment Health J. 2008;44:195–211. Rapp CA, Bond GR, Becker DR, Carpinello SE, Nikkel RE, Gintoli G. The role of state mental health authorities in promoting improved client outcome through evidencebased practice. Community Ment Health J. 2005;41:347–363. Bjorklund RW, Monroe-DeVita M, Reed D, Toulon A, Morse G. Washington State’s initiative to disseminate and implement high-fidelity ACT teams. Psychiatr Serv. 2009;60:24–27. Wilson DC, Crisanti AS. Psychometric properties of the Dual-Disorder Treatment Fidelity Scale: inter-rater reliability and concurrent validity [published online ahead of print October 11, 2008]. Community Ment Health J. 2008; doi: 10.1007/s10597-008-9167-2. Davis D, Thomson O’Brien M, Freemantle N, Wolf FM, Mazmanian P, Taylor-Vaisey A. Impact of formal continuing medical education: do conferences, workshops, rounds, and other traditional continuing education activities change physician behaviors or health care outcomes? J Am Med Assoc. 1999;282:867–887. Salyers MP, Rollins AL, McGuire AB, Gearhart T. Barriers and facilitators in implementing illness management and recovery for consumers with severe mental illness: trainee perspectives [published online ahead of print]. Adm Policy Ment Health. 2009;36:102–111. Biegel DE, Swanson S, Kola LA. The Ohio Supported Employment Coordinating Center of Excellence. Adm Soc. 2007;39:504–512. Salyers MP, McKasson M, Bond GR, McGrew JH, Rollins AL, Boyle C. The role of technical assistance centers in implementing evidence-based practices: lessons learned. Am J Psychiatr Rehabil. 2007;10:85–101. Panzano PC, Roth D. The decision to adopt evidence-based and other innovative mental health practices: risky business? Psychiatr Serv. 2006;57:1153–1161. Seffrin B, Panzano PC, Roth D. What gets noticed: how barrier and facilitator perceptions relate to the adoption and implementation of innovative mental health practices. Community Ment Health J. 2008;44:475–484. Massatti RR, Sweeney HA, Panzano PC, Roth D. The deadoption of innovative mental health practices (IMHP): why organizations choose not to sustain an IMHP. Adm Policy Ment Health. 2007;35:50–65. Aron L, Burt M, Wittenburg D. Recommendations to the Social Security Administration on the Design of the Mental Health Treatment Study. Washington, DC: Urban Institute; 2005. Livermore GA, Goodman N, Wright D. Social Security disability beneficiaries: characteristics, work activity, and use of services. J Vocat Rehabil. 2007;27:85–93.

713