13 6

Access Provided by Rand Corporation at 02/28/13 6:54PM GMT

Part 2: Original Paper

Lessons Learned from a Quality Improvement Intervention with Homeless Veteran Services Matthew Chinman, PhD Gordon Hannah, PhD Sharon McCarthy, PhD Abstract: Homeless veterans are a vulnerable population, with high mortality and morbidity rates. Evidence-based practices for homelessness have been challenging to implement. This study engaged staff members from three VA homeless programs to improve their quality using Getting-To-Outcomes (GTO), a model and intervention of trainings and technical assistance that builds practitioner capacity to plan, implement, and self-evaluate evidencebased practices. Primarily used in community-based, non-VA settings, this study piloted GTO in VA by creating a GTO project within each homeless program and one across all three. The feasibility and acceptability of GTO in VA is examined using the results of the projects, time spent on GTO, and data from focus groups and interviews. With staff members averaging 33 minutes per week on GTO, each team made significant programmatic changes. Homeless staff stated GTO was helpful, and that high levels of communication, staff member commitment to the program, and technical assistance were critical. Key words: Quality improvement, homeless people, veterans, evidence-based practice, capacity building.

S

eventy-six thousand veterans are homeless on any given night and about 136,000 veterans may have experienced homelessness during 2011.1 The Department of Veterans Affairs (VA) has adopted a no-tolerance policy towards veteran homelessness and created a five-year plan to end homelessness among veterans by 2014. A central VA program addressing homelessness is HUD-VASH (Housing and Urban DevelopmentVeteran Affairs Supportive Housing). Incorporating principles and practices from a number of evidence-based housing programs for addressing homelessness, such as Critical Time Intervention,2–4 Supportive Housing,5 and Housing First,6–7 HUD-VASH provides Housing Choice Section 8 vouchers to assist homeless veterans with rent payment and VA case management. However, a study of 36 HUD-VASH sites found that most HUD-VASH programs were not adhering to the model as intended.8 In contrast to HUD-VASH’s designed emphasis on rapid placement, sustained intensive case

Matthew Chinman is a Research Scientist at the VISN 4 Mental Illness Research, Education and Clinical Center (MIRECC) at the VA Pittsburgh Healthcare System. He can be reached at the VA Pittsburgh Healthcare System; 7180 Highland Drive (151-R); Pittsburgh, PA 15206; (412) 954-4338; [email protected]. Gordon Hannah and Sharon McCarthy are Research Associates at the VISN 4 MIRECC at the VA Pittsburgh Healthcare System. © Meharry Medical College

Journal of Health Care for the Poor and Underserved 23 (2012): 210–224.

Chinman, Hannah, and McCarthy

211

management, rehabilitation services, and permanent housing, this study found that the process of obtaining a housing voucher and becoming housed was relatively slow and highly variable. The HUD-VASH study also found that the types and intensity of supports provided to veterans decreased dramatically over time, with little emphasis on rehabilitation-oriented activities. The VA has also implemented Supported Employment9—an evidence-based vocational program—at VA Centers nationwide. However, a two year evaluation of a national roll-out of Supported Employment in VA found that although significant improvement was observed in rates of competitive employment, 20% of sites did not achieve acceptable model fidelity.10 The degree of improvement in employment was also smaller than that found in previous research. The researchers concluded that more intensive onsite training and performance monitoring was needed to optimize adoption and implementation of Supported Employment. Developing this capability will be crucial if the VA is to be successful in its plan to end Veteran homelessness. Clinicians and other staff members in VA homeless programs have faced difficulty in implementing evidence based practices (EBPs) with fidelity for several reasons. While EBPs may offer strategies for effective practice, they often do not address implementation barriers that exist at both the individual and organizational levels that predict successful implementation of EBP.11–17 Further, trainings alone do not result in change because attendees often face barriers applying what they learned.18,19 Various tools have been developed to help organizations monitor service quality, such as Balanced Score Cards,20 Whole System Measures,21 and Performance Dashboards,22 and processes also exist to help organizations improve service quality, such as PDSA Cycles.23 These mechanisms are helpful for improving the implementation of current programming but do not usually address situations where current programming is not evidence-based. An approach called Getting To Outcomes (GTO) may offer a more comprehensive and systematic approach to quality improvement because GTO addresses both the selection and implementation of EBPs and the monitoring and quality improvement of these programs. We piloted the GTO approach, reviewed below, with three VA homeless programs. Getting To Outcomes strengthens the knowledge, attitudes, and skills practitioners need to carry out the various tasks required for strong implementation of EBPs. Getting To Outcomes does this by posing a series of steps practitioners should follow in order to obtain positive results and then providing practitioners with the guidance necessary to complete those steps with quality. The 10 GTO steps roughly correspond to four general areas: (1) diagnosing problems and setting priorities (steps 1–2), (2) choosing, planning and using EBPs (steps 3–6); (3) evaluating programming and outcomes (steps 7 and 8); and (4) improving and sustaining changes (steps 9 and 10). All of these steps are designed to be logically linked. Goals and performance targets are associated with program activities that will meet those targets, which are linked to process and outcome measures to assess if the targets are being met, which are linked to quality improvement activities that makes use of the process and outcome data. Guidance from the GTO approach comes in the form of tools, face to face training for homeless program staff members, and ongoing technical assistance (TA) provided by GTO staff members who meet with program staff members to help complete needed steps. The

212

QI with homeless veteran services

goal is to work with leadership and staff members to integrate the GTO improvements into routine operations. Getting To Outcomes uses social cognitive theories of behavioral change,24,25 such that exposure to GTO training and TA leads to more knowledge about performing GTO-related activities (i.e., planning, implementation, self-evaluation), which leads to more positive attitudes towards these activities, which in turn leads to the execution of more GTO-related behaviors. These behaviors support the successful implementation of EBPs,26 reflected in improved fidelity and, ultimately, improved outcomes. The GTO 10 step model has been applied to a wide variety of domains, yielding written guides for substance abuse prevention,27 teen pregnancy prevention,* positive youth development,28 and prevention of underage drinking.29 The GTO intervention (tools, training, TA) has been shown to improve individual capacity and program performance to facilitate the planning, implementation, and evaluation of substance abuse prevention programs.30,31 The purpose of this study was to adapt and pilot GTO for use with homeless programs within the VA, which have not regularly engaged in quality improvement, and test its feasibility and acceptability. Using GTO, the staff members in three homeless programs at a large urban VA in the Northeast were engaged to improve the quality of their programs’ services. A separate team was also created to work across all three programs. Getting To Outcomes’ feasibility and acceptability was assessed by examining the results of these four projects, staff member time spent on GTO via a weekly email query, and focus groups and interviews with staff members from each program.

Methods Participants. Staff members from three homeless programs in a large urban VA in the Northeast were invited to participate in the study. The three programs were Healthcare for Homeless Veterans (HCHV), Domiciliary Care for Homeless Veterans (Dom), and Vocational Services. HCHV provides homeless outreach, health care and community housing services. The Dom is a 50-bed residence with average stays of 75 days and with a focus on work and recovery. The Vocational Services program serves both homeless and housed veterans with the primary goal of achieving competitive work in the community. Thirty-two staff members participated in the study. The VA Institutional Review Board approved the study and all participating staff members provided written, informed consent. Intervention. The intervention began with a half-day training for all staff members. Technical assistance (TA) was then provided over a two-year period. A second half-day training was held 16 months into the intervention. Trainings. Both trainings were provided to all of the staff members in the three homeless programs. The initial training walked homeless staff members through the 10 GTO steps and then involved the three teams in discussions about how GTO could be used in their specific program. Initial action planning based on these ideas was later followed up on through meetings with the technical assistants. *Lesesne CA, Lewis KM, Fisher D, et al. Promoting science-based approaches using Getting To Outcomes. Unpublished manual. Atlanta, GA: Centers for Disease Control and Prevention, 2011.


213

The second training began with a presentation of GTO work in progress, presented by staff members on each team, and organized to reinforce concepts from the original training and to educate new staff members. This was followed by a group exercise in which each of the three programs analyzed the impact of their programs and how they could be improved using a tool based on an Ishikawa (fishbone) diagram.32 These analyses were used to guide future GTO work in conjunction with the TA. Technical assistance. After the training, two PhD level technical assistants met with teams of staff members from each of the three homeless programs biweekly and provided additional support face-to-face, by phone, and by email as needed. Technical assistance began with detailed discussions between TA and program staff members about how the program was operating in each of the domains targeted by the GTO 10 steps (done before the training). From there, the TA and program staff members developed a plan (started at the training) for the GTO work. These project plans involved working through specific GTO steps, selected based on the project goals. After this planning phase, the TA providers regularly monitored progress of each of the programs’ projects and provided feedback as a regular part of the bi-weekly TA sessions. Finally, the technical assistants also met with the group of program directors monthly to obtain feedback and troubleshoot implementation barriers of the GTO projects. Each GTO project is discussed in the results section. Data collection. Project outcomes. Quantitative data were collected to assess the outcome of each program’s project. Since each of these projects was unique and was designed collaboratively with staff members after the initiation of the intervention, the specific methodology used for each project will be described in the Results section. Focus groups and interviews. Three focus groups were conducted 20 months into the intervention, one for each of the three homeless programs. The focus groups used a standardized set of questions to explore each programs’ beliefs about: 1) any improvements GTO created; 2) how GTO lead to improvements; 3) barriers to using GTO/aspects that were not helpful; 4) ways in which GTO could be improved; and 5) likelihood GTO would be continued and why or why not. Each program’s leader was interviewed separately using a similar script. About 74% of staff involved participated in either a focus group or interview. The focus groups were conducted by a PhD-level researcher with no connection to the project. The constant comparative method was used to analyze the transcripts.33,34 First, verbatim transcripts were made from the audio recordings. Then inductive category coding was done by having one researcher read through the transcripts and group together text with similar content into provisional categories. Then a second researcher categorized the text using these provisional categories. Coding discrepancies were discussed until consensus was reached and category inclusion criteria refined to minimize future discrepancies. Inclusion criteria were written as propositional statements. After coding, patterns and relationships were explored across categories to generate additional propositional statements. These propositional statements formed the basis for the results reported on in the results section. Technical assistance. The types and amounts of technical assistance (TA) given to GTO participants were recorded by each TA using a TA Log. The TA Log tracked date, type of TA (in person, phone, e-mail), and duration of the TA.

214


Staff member burden. Homeless staff members reported the amount of time spent on GTO activities in two ways. Initially, staff member time was estimated retrospectively via paper survey, and initial hours spent in meetings were calculated based on attendance records from meeting minutes. At 23 weeks, staff members began reporting their time spent on GTO with a weekly email system. This reporting included time spent in meetings and time spent on other work preparing for meetings. These two data sources were merged to develop a total number of hours spent by each staff member on GTO for the total project duration.

Results GTO projects and outcomes. 1. HCHV: A process-of-care policy was created and implemented to monitor and place veterans who screened positive for mild to moderate suicide risk. This project involved reviewing a number of evidence-based suicide risk management guidelines to identify existing procedures that were not evidence-based. Recommendations from the evidence-base were tailored for the VA Healthcare System. For example, HCHV drafted a policy that incorporated a number of resources unique to the VA Healthcare System to re-engage veterans with some suicidal risk who dropped out of treatment. Staff members were provided two trainings on the care of suicidal patients. Project Outcome: Staff tracked the percentage of veterans with mild or moderate suicide risk for who the new policy was applied. An initial review showed that all nine cases did not trigger the policy because they were either referred to other programs, could not be contacted, or chose not to enter an HCHV program. Based on these findings, the policy was modified to address referral procedures in more detail. This evaluation will continue to be conducted semi-annually. 2. Domiciliary: Dom staff members conducted a needs assessment survey of their current veterans, assessing what programming would be most helpful. Based on survey results, staff members developed new goals and objectives for their program, which the team used to incorporate two new evidence-based vocational practices. The new program plan introduced a new risk assessment process during the initial screening to help objectively determine the Veteran’s challenges to vocational success.35 Those veterans who face multiple challenges are now immediately placed in a therapeutic work environment where vocational staff members can evaluate the Veteran’s ability to work. The immediate use of a vocational experience was based on supported employment research36 suggesting that actual work experience is more helpful for vocational success than extended pre-training or development programs. Project Outcome: The Domiciliary redesign was implemented near the end of this study, and outcomes are still incomplete, but will identify the influence of the new risk assessment measure and vocational experiences on vocational success. 3. Vocational Services: Staff members identified long-term employment after discharge as one of their key outcomes for veterans. Their GTO project established


215

a process for staff members to call veterans after discharge to assess employment status. Project Outcome: Staff tracked the percentage of discharged veterans who were employed at 30 and 90 days after discharge. Of an initial group of 16 veterans working at discharge, 13 (81%) were still employed at 30 days. Of a group of 12 veterans who were working at discharge and at one month, six (50%) remained employed at 90 days. Overall, the contact rate for the 30-day sample was 81% and for the 90-day group it was 77%. Based on these data, staff are exploring which factors that predict long-term employment that might be used to improve programming. 4. Cross-program team: After review of the evidence-base in relapse intervention, the Cross-program team developed a protocol to offer additional substance abuse treatment to veterans who relapsed while keeping them in their current VA housing. The various housing programs varied widely in size, location, average length of stay, and veteran demographics, so the protocol was customized for each participating program. Project Outcome: Staff tracked the percentage of veterans enrolled in the relapse program who remained housed. Across all the housing programs, 22 of the 37 veterans (59%) who relapsed were referred to the new program and of those entering the relapse program, 12 of 18 (67%) were still in their housing 30-days after relapse (previously all of these veterans would have been discharged from their housing). Focus groups and interviews. What did GTO improve? Both staff members and leaders commented that their GTO projects were generally helpful and important to veterans, albeit a lot of work. For example, one commented that the Dom redesign “started to transition to a model where the veterans are gaining more independence.” The vocational post-discharge phone calls provided useful information for quality improvement and service design, and helped maintain good relationships with veterans. The relapse program created by the cross-program team was cited as helpful to veterans by multiple programs and leaders. Staff members commented that veterans who relapsed in the program now had a chance to start again instead of being thrown out. How did GTO foster improvement? Several points were made about how GTO was helpful. One mechanism frequently cited for this was providing a regular time away from daily duties to think about their programs. This “set-aside time” was often driven by the biweekly meeting between TA and program staff members and most of the program staff members viewed the TA staff members as very helpful, patient, and a key part of GTO. A third benefit frequently noted about the GTO process was that it brought multiple perspectives to work together to find solutions. This increased staff members’ understanding of each other’s programs and helped with problem solving. As a result of working together more, a number of comments indicated that communication improved between homeless programs and with other VA Departments and veterans. Fourth, a few commented that GTO allowed the staff members to recommend changes rather than waiting for changes to come from leadership. This was seen as being time efficient and also promoting commitment and ownership of the program. Finally, a few comments indicated that GTO had increased the staff members’ capacity related to leadership and problem solving.

216


What were barriers to the GTO work? Many felt suspicious of GTO at first and felt it would result in more work. Others commented that although the process was described as “voluntary,” in fact some staff members saw it as mandatory. Finally, a few people indicated feeling overwhelmed at the amount of material presented at the first training, which was the introduction to GTO for most staff members. Misgivings resulting from the introduction of the project appeared to affect commitment to the program from some staff members. A second barrier was the perceived openness and inclusiveness of the GTO process including who got to participate, how progress was communicated, and how decisions were made. There was uncertainty about how people became involved with GTO projects, and some staff members felt that while they might have been “invited” to come to meetings, they were not really wanted. These feelings occurred even though all staff members were invited to participate in all GTO projects at the start and periodically re-invited throughout the projects. These feelings were compounded by a sense that what occurred in the meeting was not communicated well to them. These feelings seem to have come as a surprise to those who had participated in the project team. Some felt decisions were not made openly in GTO groups, but based on “a hidden blueprint,” and with information that “was never shared anywhere.” These perceptions became a barrier because those who did not feel included often lacked commitment to the changes that resulted from the process. Others countered these assertions with examples of collaborative decision-making. A third barrier was the large amount of work GTO required and how that work was shared among program staff members. Some indicated that it was a lot of work but the outcomes were worth it, but some questioned whether the time cost was worth the benefit. Some found the number of meetings required to reach consensus before changes could be made, frustrating. The time factor was seen as a significant barrier to GTO’s continued use, since GTO was viewed as competing against the more immediate clinical needs of veterans. In addition, staff members commented on the inability to get full participation on the GTO project teams, which sometimes left a few people doing a lot of work. The lack of authority of the chairs of each team to require work from their peers was viewed as a contributing factor in this dynamic. Those who did end up working on projects sometimes felt exhausted by them. Attempts at addressing this issue, for example by rotating staff members through the teams, met with mixed success as some felt it disrupted the continuity while others felt it was a good way to get everyone involved. How could GTO be improved in the VA? Staff members made several suggestions. First, many staff members and leaders stated that more commitment to the program on the part of staff members would be desirable and that could be achieved by providing more information up front about what GTO is and what the process entails. Staff members suggested working hard to obtain input from all participants. Next, several mentioned ways that the initial GTO training could be improved. For example, several staff members commented that they liked the interactive format of the second training better. Others suggested that a different type of introduction would be more successful; rather than a half day intensive introduction, provide “small doses . . . maybe some emails and prompting before” any in-depth training. Many thought that teams should


217

be smaller so that groups could be more efficient. Many in one focus group suggested that small teams disseminate minutes or conclusions from meetings to non-participants to get more commitment to the program and consensus. Some comments addressed the time issue and suggested incentives, shorter projects, fewer meetings per person, explaining GTO’s value, and making sure projects reflect staff members’ priorities. How will the GTO process be sustained? Staff members split on whether GTO would continue to be used in their programs after the technical assistance and research aspects concluded. Some felt that GTO would continue, some thought that a scaled down version might be possible, others felt it would not be sustained. There was agreement that leadership from management and commitment to the program from staff members would be required to sustain it. Several commented that continued TA support would be helpful or necessary for GTO to continue. One team leader commented that current quality improvement staff members at the medical center could benefit from learning GTO and then “they can become support staff members to programs like mine.” Staff member burden and TA utilization. Participant time data collected via email over the last 82 weeks of the intervention was 83% complete. Missing data occurred if participants failed to respond to an initial email query and a follow-up reminder. Review of the data suggested that staff members often neglected to report their time data during periods in which they were not actively involved in the intervention, so replacing missing data with zeros appeared to yield the best estimate of participation. Total participant time, TA utilization, and average time spent per week were calculated by program (see Tables 1 and 2). The Dom had the most participants involved in GTO and involved the most participant and TA time. The HCHV program involved a much smaller percentage of their staff members in the GTO process than the other two programs. Participants reported spending more time in meetings than doing

Table 1. TOTAL PARTICIPANT AND TA TIME BY PROGRAM Dom Total # of Staff membersc # of Active Participantsd Total Participant Timee Total TA Timeb

Vocational Services

HCHV

CrossPrograma

Allb

18 14 22 N/A 54 14 9 9 N/A 32 619 hours 437 hours 414 hours N/A 1470 hours 430 hours 232 hours 214 hours 257 hours 1133 hours

Since participants in the Cross-Program team came from the 3 programs and participants were not asked to distinguish between time spent on Cross-Program activities and other work, no participant information is presented for the Cross-Program team. b Times are rounded to the nearest hour. Totals may not equal sum of columns or rows due to rounding. c This is the number of staff members who were eligible to participate over the course of the study. d This is the number of staff members who participated at some point over the course of the study. The number participating at any given time would be smaller due to staff members’ transitions. e Total participant time adds estimate for first 23 weeks with data collected in the last 82 weeks. a

218


Table 2. AVERAGE PARTICIPANT AND TA TIME PER WEEK BY PROGRAM

Average Participant Time/Week In Meetings Preparation Work Totalb Lowestc Highestc Average Technical Assistant Time/Week Face-to-face Support Phone Support Email Support Preparation Work Total2

Dom

Vocational Services

HCHV

CrossPrograma

All

21.6 min 11.3 min 33.0 min 3.2 min 97.3 min



N/A N/A N/A N/A N/A


139.3 min 1.4 min 4.6 min 100.3 min

55.9 min 0.1 min 6.3 min 70.4 min

52.9 min 3.7 min 18.1 min 47.7 min

52.9 min 2.7 min 19.4 min 71.9 min

75.2 min 2.0 min 12.1 min 72.6 min


a Since participants in the Cross-Program team came from the 3 programs and participants were not asked to distinguish between time spent on Cross-Program activities and other work, no participant information is presented for the Cross-Program team. b Totals may not equal sum of columns or rows due to rounding. c These rows indicate the highest and lowest average time per week for individual participants.

reparatory work for meetings. For every one hour of participating staff members’ p time invested in the project, TAs provided 46 minutes of support. TA was primarily delivered through face-to-face meetings. For technical assistants, a one hour face-toface meeting corresponded roughly with one hour of prep work. Homeless staff members worked an average of about 33 minutes a week, or approximately an hour every two weeks, roughly the duration of a GTO meeting. However, a distribution of participants’ average total time spent per week showed that a small number of participants (one per project) were doing a relatively large amount of work (90–100 minutes per week) while most participants were spending only 20–30 minutes per week, and a sizeable minority were spending even less time (see Figure 1). The top 10% of the most involved staff members provided 30% of the hours worked, and the bottom 30% contributed only 4% of the hours. Participant and TA time was plotted longitudinally to look for patterns over the course of the intervention (see Figure 2). Increases and decreases in time spent on GTO did not follow any general pattern, although during the last four months, both participant and TA time tended to decrease as activities drew to a conclusion.


219

Figure 1. Distribution of average total minutes/week by participant.

Discussion With the assistance of TA staff members, training, and written tools, staff members of the three homeless programs were able to make noticeable improvements in their programming. Although none of the improvements were themselves the wholesale adoption of a specific evidence-based program, most improvements involved programs becoming more evidence based (using evidence based guidelines to manage high risk patients, moving vocational programming more toward a Supported Employment model, supporting relapse with more substance abuse treatment rather than discharge). Further, while quality improvement often involves discrete, time-limited activities, GTO is unique in that it attempts to engage all staff members in continuous improvement across many domains of work (goals, planning, implementation, evaluation). While these improvements are encouraging, they came with certain costs. Many staff members experienced challenges adding GTO activities to their already high workload, and some staff members felt that the process was not always transparent or as inclusive as it could have been. The time data analysis provides a first look at the time burden of GTO projects, and the role of technical assistance in relation to time spent by staff members. The data show that staff members committed a wide range of time to the GTO process. One or two project leaders provided a large portion of effort, doing the majority of the preparation and outside work for the GTO projects; a second, large group contributed primarily by

220


participating in GTO meetings, and about 30% of participants did nominal amounts towards the projects. These data support the qualitative data of those project leaders who described being exhausted by their experience with GTO. Based on these data, it seems critical to put in place mechanisms to encourage all project staff members to contribute meaningfully. Whether GTO work is formally added to job duties and monitored by program leadership at regular intervals or incentivized by special incentives and rewards, this is an area where early and open discussion with team leadership and staff members can help establish clear expectations. Both time and the qualitative data on technical assistance support the conclusion that TA plays a critical role in the GTO process. The utility of TA in general and how the GTO TA was proactively delivered, mostly in person, and tailored to local needs is consistent with other GTO research30,31 and research on other models that similarly use technical assistance to facilitate evidence-based practices.37 However, in a large review of TA literature, Katz and colleagues also conclude that more research is needed to better understand the parameters of how TA works. This study takes a next step in this regard, for the first time documenting the ratio of TA to staff members’ time (46:60 minutes) that yielded the above program changes. This rather high level of support potentially reflects the difficulty of implementing evidence-based program changes in a large and complex homeless services setting with staff members who are inexperienced in systematic quality improvement. This study also showed the importance of TA’s

Figure 2. Hours spent per month by participants and TAs.


221

adapting to the setting—in this case, showing up and talking in person was found to be most effective given the chaotic nature of providing homeless services. Feedback from the focus groups and interviews offer important suggestions for implementing GTO in a large, bureaucratic environment like the VA. Perhaps most important is the critical role of communication. In some instances, communication about services and veterans improved between homeless programs or with other VA entities and staff members viewed this as an achievement. However, many pointed to challenges in communication about the GTO work itself among program staff members. Staff members noted that providing minutes from meetings to all team members would have been helpful, instead of just within GTO team members. More formal communication about the establishment of the GTO projects could have been helpful, as well as varied types of communication across teams. The second training, when each team presented their work-to-date to all staff members, was well received and appreciated, and could be a model for how to disseminate information about GTO work. A second key finding was the importance of introducing GTO in a successful way. A wide range of research supports giving great attention to the introduction of new ideas, programs and teams.38 While a successful introduction is tied to good communication about GTO, it also involves establishing clear, honest expectations for staff members, as well as providing specific information about how the GTO process will benefit veterans. Staff members repeatedly mentioned the need for “buy-in” by staff members, and the importance of seeing how the GTO projects would improve care for their clients. Third, results show that staff members were extremely sensitive to the nature of participation that GTO provided. Specifically, staff members voiced a need for everyone to be included and welcomed and for differing views and opinions to be respected. Research has long shown that the benefits of participation are seriously compromised if employees perceive that decisions have already been made, and if employee opinions and views are not honestly taken into account.39 This factor may have been exacerbated by the strong hierarchical nature of medical settings generally, where employees may wait for changes to be explained, rather than try to effect changes from the bottom. Although the GTO research staff members repeatedly asked for volunteer participation, busy staff members waited, or hesitated, and later did not remember being invited. The voluntary nature of the participation was also viewed with skepticism, as employees felt pressured by their team leaders to participate, and in some cases were told to participate. Although the GTO framework evolved from an empowerment model that supports voluntary participation, in a large bureaucracy it may be better to adapt the model and present the projects as a new initiative, with clear expectations of participation. For example, it could be beneficial to provide staff with a realistic preview of GTO, such that staff members understand the time and effort required. Specific criteria for who is chosen to participate in the projects are also important, since although staff members were repeatedly asked to volunteer, there were still some staff members who did not feel included. Inclusion and exclusion on project teams is a critical issue for early discussion with team leadership, as some leaders welcomed volunteers, and some managed the process more, choosing specific individuals or rotating staff members on and off the teams. It is important to work closely with leadership to establish the process for inclusion so that honest and accurate information can be given to staff members.

222


The strengths of this study include deep, rich qualitative data, and a first look at the time burden for staff members working on improvement projects. This study shows that, with support, a relatively small outlay of staff member time, on average, can result in significant movement towards EBPs, and program quality improvements. Limitations of the study derive from its exploratory nature, and qualitative design. All three project programs were at a single VA site, so findings may not generalize to other sites. The study was a pilot, adapting GTO for use within a VA homeless service setting and, therefore, no control or comparison groups were used, limiting the ability to draw firm conclusions about the impact of the intervention. In conclusion, although GTO provided many challenges for staff members, eventually most staff members clearly appreciated the improvements to veteran services, and recognized the benefits of their efforts. Many commented that they were surprised to look back and see how much they had accomplished, and how successful the changes had been to their programs. Given the urgent need among the vulnerable population of homeless veterans, with refinements, GTO could serve as a model to support VA homeless services providers’ planning, implementation, and self-evaluation of evidencebased practices.

Acknowledgments This study was funded by the National Center on Homelessness Among Veterans. The authors are grateful for the support and assistance of Vince Kane, MSW, Director, National Center on Homelessness Among Veterans. We also gratefully acknowledge the staff of the VA Homeless Center where this research was conducted.

Disclaimer The contents do not represent the views of the Department of Veterans Affairs or the United States Government.

Notes 1. U.S. Department of Housing and Urban Development/U.S. Department of Veteran Affairs. Veteran homelessness: a supplemental report to the 2009 annual homeless assessment report to Congress. Washington, DC: U.S. Department of Housing and Urban Development/U.S. Department of Veteran Affairs, 2010. Available at: http:// www.hudhre.info/documents/2009AHARVeteransReport.pdf. 2. Herman DB, Conover S, Gorroochurn P, et al. A randomized trial of critical time intervention to prevent homelessness after hospital discharge. Psychiatr Serv. 2011 Jul;62(7):713–9. 3. Kasprow WJ, Rosenheck RA. Outcomes of critical time intervention case management of homeless veterans after psychiatric hospitalization. Psychiatr Serv. 2007 Jul;58(7):929–35. 4. Susser E, Valencia E, Conover S, et al. Preventing recurrent homelessness among mentally ill men: a “critical time” intervention after discharge from a shelter. Am J Public Health. 1997 Feb;87(2):256–62.


223

5. Rog DJ. The evidence on supported housing. Psychiatr Rehabil J. 2004 Spring; 27(4):334–44. 6. Lanzerotti L, National Alliance to End Homelessness. Housing first for families: research to support the development of a housing first training curriculum. Washington, DC: Lanzerotti L, National Alliance to End Homelessness, 2004 Mar. 7. Tsemberis S, Gulcur L, Nakae M. Housing first, consumer choice, and harm reduction for homeless individuals with a dual diagnosis. Am J Public Health. 2004 Apr; 94(4):651–6. 8. O’Connell M, Kasprow W, Rosenheck RA. National dissemination of supported housing in the VA: model adherence versus model modification. Psychiatr Rehabil J. 2010 Spring;33(4):308–19. 9. Becker D, Whitley R, Bailey EL, et al. Long-term employment trajectories among participants with severe mental illness in supported employment. Psychiatr Serv. 2007 Jul;58(7):922–8. 10. Rosenheck RA, Mares AS. Implementation of supported employment for homeless veterans with psychiatric or addiction disorders: two-year outcomes. Psychiatr Serv. 2007 Mar;58(3):325–33. 11. Schoenwald SK, Hoagwood K. Effectiveness, transportability and dissemination of interventions: what matters when? Psychiatr Serv. 2001;52(9):1190–7. 12. Ennett ST, Ringwalt CL, Thorne J, et al. A comparison of current practice in schoolbased substance use prevention programs with meta-analysis findings. Prev Sci. 2003 Mar;4(1):1–14. 13. McCormick LK, Steckler AB, McLeroy KR. Diffusion of innovations in schools: a study of adoption and implementation of school-based tobacco prevention curricula. Am J Health Promot. 1995 Jan–Feb;9(3):210–9. 14. Parcel GS, Eriksen MP, Lovato CY, et al. The diffusion of school-based tobaccouse prevention programs: project description and baseline data. Health Educ Res. 1989;4(1):111–24. 15. Rohrbach LA, D’Onofrio CN, Backer TE, et al. Diffusion of school-based substance abuse prevention programs. Am Behav Sci. 1996 Jun;39(7):919–34. 16. Rohrbach LA, Graham JW, Hansen WB. Diffusion of a school-based substance abuse prevention program: predictors of program implementation. Prev Med. 1993 Mar;22(2):237–60. 17. Steckler A, McLeroy KR, Goodman RM, et al. Toward integrating qualitative and quantitative methods: an introduction. Health Educ Q. 1992 Spring;19(1):1–8. 18. Cividin TM, Ottoson JM. Linking reasons for continuing professional education participation with post-program application. J Continuing Educ Health Prof. 1997 Winter;17(1):46–55. 19. Ottoson J. After the applause: exploring multiple influences on application following an adult education program. Adult Educ Q. 1997 Winter;47(2):92–107. 20. Kaplan RS, Norton DP. Balanced scorecard: translating strategy into action. Cambridge, MA: Harvard Business School Press, 1996. 21. Martin LA, Nelson EC, Lloyd RC, et al. Whole system measures: IHI innovation series white paper. Cambridge, MA: Institute for Healthcare Improvement, 2007. 22. Eckerson WW. Performance dashboards: measuring, monitoring, and managing your business. Hoboken, NJ: John Wiley & Sons, 2006. 23. Deming WE. Out of crisis. Cambridge, MA: MIT Press, 2000. 24. Bandura A. Health promotion by social cognitive means. Health Educ Behav. 2004 Apr;31(2):143–64.

224


25. Fishbein M, Ajzen I. Attitudes towards objects as predictors of single and multiple behavioral criteria. Psychol Rev. 1974 Jan;81(1):59–74. 26. Durlak J, DuPre E. Implementation matters: a review of research on the influence of implementation on program outcomes and the factors affecting implementation. Am J Community Psychol. 2008 Jun;41(3–4):327–50. 27. Chinman M, Imm P, Wandersman A. Getting to outcomes 2004: promoting accountability through methods and tools for planning, implementation, and evaluation. Santa Monica, CA: RAND Corporation, 2004. Available at: http://www.rand.org/publications /TR/TR101. 28. Fisher D, Imm PS, Chinman M, et al. Getting To outcomes with developmental assets: ten steps to measuring success in youth programs and communities. Minneapolis, MN: Search Institute Press, 2006. Available at: http://www.searchinstitutestore.org /ProductDetails.asp?ProductCode=0158%2DW. 29. Imm P, Chinman M, Wandersman A, et al. Preventing underage drinking: using getting to outcomes with the SAMHSA strategic prevention framework to achieve results (RAND TR-403-SAMHSA). Santa Monica, CA: RAND Corporation, 2007. Available at: http://www.rand.org/pubs/technical_reports/TR403.html. 30. Chinman M, Hunter S, Ebener P, et al. The Getting To outcomes demonstration and evaluation: an illustration of the prevention support system. Am J Community Psychol. 2008 Jun;41(3–4):206–24. 31. Chinman M, Tremain B, Imm P, et al. Strengthening prevention performance using technology: a formative evaluation of interactive getting to outcomes. Am J Orthopsychiatry. 2009 Oct;79(4):469–81. 32. Bauer JE, Duffy GL, Westcott, RT. The quality improvement handbook. Milwaukee, WI: American Society of Quality/Quality Press, 2002. 33. Maykut P, Morehouse R. Beginning qualitative research: a philosophic and practical guide. London, United Kingdom: Falmer Press, 1994. 34. Lincoln Y, Guba E. Naturalistic inquiry. Beverly Hills, CA: Sage, 1985. 35. LePage JP, Bluitt M, House-Hatfield T, et al. Improving success in a Veterans homeless domiciliary vocational program: model development and evaluation. Rehab Psychol. 2005 Aug;50(3):297–304. 36. Bond GR. Supported employment: evidence for an evidence-based practice. Psychiatr Rehabil J. 2004 Spring;27(4):345–59. 37. Katz J, Wandersman A, Ellis-Wiggan I. Evaluating the benefits of proactive communitycentered technical assistance. Presented at: SCRA Biennial Meeting, Montclair State University, Montclair (NJ), 2011 Jun 6. 38. Katzenbach J, Smith D. The wisdom of teams. New York, NY: Harper Collins, 1993. 39. Pateman C. Participation and democratic theory. Cambridge, United Kingdom: Cambridge University Press, 1970.