DSP Workforce Development Demonstration 1

DSP Workforce Development Demonstration Training and Technical Assistance in Direct Support Professional Workforce Development: Results from a three-year demonstration project

Nancy McCulloh, M.A. [email protected] Sheryl A. Larson, Ph.D. [email protected] John Sauer, M.S.W. [email protected] Amy Hewitt, Ph.D. [email protected] Research and Training Center on Community Living University of Minnesota 204 Pattee Hall, 150 Pillsbury Drive SE Minneapolis, MN 55455 Marianne Taylor, M.A. [email protected] Bristol Community College 777 Elsbree St. Fall River, MA 02720

Running Header: DSP Workforce Development Demonstration

For more information contact: Nancy McCulloh, [email protected], 320-253-5661 or 612-626-7765

Reference: McCulloh, N., Taylor, M., Larson, S., Sauer, J., and Hewitt, A. (2008). Training and Technical Assistance in Direct Support Professional Workforce Development: Results from a three-year demonstration project. University of Minnesota, Research and Training Center on Community Living.

Acknowledgement: NTIFFS-TA project was a Field Initiated Project supported by the U.S. Department of Education’s National Institute on Disability and Rehabilitation Research (NIDRR) Grant #H133G030058 and conducted by the Research and Training Center on Community Living at the University of Minnesota.

1

DSP Workforce Development Demonstration Abstract A train-the-trainer and technical assistance model was used to teach community residential and vocational organizations strategies to reduce direct support staff turnover. Activities included: assessing organizational workforce challenges; web-based training and distance education; two onsite train-thetrainer institutes; provider based supervisor training; on-site and remote technical assistance to organizations; implementing interventions and measuring outcomes; and sharing tools and resources. Organizations from five states serving people with intellectual and developmental disabilities were selected from 100 applicants to participate in this three-year project. Across the organizations, direct support professional average turnover declined from 40% in 2004 to 23% in 2006.

2

DSP Workforce Development Demonstration Introduction Effective direct support services for people with disabilities are based on the intangible qualities of human relationships, especially the ability of the support person to know and understand the preferences and needs of the individual receiving support, and to cultivate the reciprocity and trust that are essential to healthy relationships. Such relationships take time, attention, and consistency to mature to support adequately the focus person in achieving life goals. For people with disabilities who need assistance in their day-to-day lives, these relationships are essential to daily well-being and form the foundation of quality community services. This foundation is eroding due to the ever-increasing shortage of caring and competent direct support professionals (ASPE, 2006). In 2002, President Bush’s New Freedom Initiative articulated a vision for the inclusion of people with disabilities in all aspects of community life, a fitting capstone to the half-century of building support services embedded in communities so that no one would have to live in an institution to get the support they need. There is a clear need for an infrastructure of skilled, trained, and well-supported direct support professionals to make possible the success of the New Freedom Initiative. Changing demographics, low wages, and increased demand for DSPs in human service settings have created a workforce crisis. Community program administrators list DSP recruitment as one of their most significant challenges (Cohen, 2000; Hewitt, Larson, & Lakin, 2000; Barry Associates, 1999; Test, Solow, & Flowers, 1999). National residential DSP vacancy rates were 10.7% in 2001 (ANCOR, 2001). Other studies reported vacancy rates 10.4% in Illinois (Illinois Direct Support Workforce Initiative, 2005) and 8% in Minnesota (Hewitt, Larson, & Lakin, 2000). High turnover rates exacerbate the effects of recruitment challenges. Turnover rates of DSPs in community settings have consistently ranged between 43% and 70% averaging 50% nationally (ANCOR, 2001; Braddock, et al, 1992; Hewitt & Larson, 2007; Lakin & Bruininks, 1981; Larson, Lakin, & Hewitt, 2002). In an evaluation of Minnesota’s Home and Community Based Services Waiver (HCBS) 75% of all providers (residential and vocational) reported difficulties findings qualified applicants; 46% reported turnover problems; and 28% reported that training also posed a challenge (Hewitt, Larson, & Lakin, 2000).

3

DSP Workforce Development Demonstration Frontline Supervisors (FLS) are a critical piece in the recruitment and retention puzzle (Buckingham & Coffman, 1999). Among the key contributors to DSP turnover are organizational socialization, training practices, supervisor tenure and the extent to which FLS are viewed as being fair (Larson, Lakin, & Bruininks, 1998; Hewitt et al, 2000; Bachelder, 1994; Braddock & Mitchell, 1992; Lakin & Bruininks, 1981). Low wages, conflicts with co-workers and problems with supervisors are among the reasons DSPs give for leaving their current employment (Larson, et al, 1998; Lakin & Bruininks, 1981). FLS have significant staffing responsibilities. For example, 52.9% of FLS reported screening applicants; 79.4% interviewed applicants; 76.5% hired employees; 59.5% provided agency orientation; 94.1% provided house orientation; 91.2% provided on-going training; 97.1% conducted performance evaluations and 82.4% fired employees (Hewitt, Larson, & Lakin, 2000). Effective DSP workforce development interventions focus on training FLS and providing targeted technical assistance to address staff challenges in reducing recruitment and retention challenges. For example, the Department of Labor’s “Partnerships for Success” project offered community provider agencies advanced training opportunities for their supervisors, as well as technical assistance (Hewitt, Larson, Sauer, Anderson, & O’Nell, 2001). Participants decreased turnover and improved supervisory practices. Similar projects have been sponsored by state Developmental Disabilities Councils and the Center for Medicaid and Medicare Services (Kansans Mobilizing for Direct Support Workforce Change Project, 2005; University of Minnesota, Research and Training Center on Community Living and

The Lewin Group, 2007). Methods Eight community human service agencies from five different states participated in this demonstration project. Using a “train-the-trainer” model, the project team employed distance learning strategies and two onsite workshops to build each agency’s capacity to teach and apply best practices in frontline supervisory competency areas. FLS workforce development content experts taught, mentored, and coached organizational representatives to apply those practices in their organizations.

4

DSP Workforce Development Demonstration Participants Applications were disseminated nationally to community-based organizations employing direct support professionals and their frontline supervisors, as well as to professional associations of employers such as the American Network of Community Options and Resources (ANCOR). Information about the project, its benefits to participants, project activities and expectations were distributed with the application forms through email lists and list-serves, encouraging those receiving materials to forward the information to others. We received 100 applications. After excluding incomplete applications, the remaining applicants were ranked by project staff based on a scoring protocol. Eight organizations in five states were selected jointly by the project advisory committee and the project staff. Selected organizations provided a letter of commitment agreeing to participate in the proposed activities. They also designated two or three senior trainers and administrators to serve as primary agency representatives in fulfilling project activities including attending the onsite training workshops, and recruiting and training FLS participants from their organizations to participate in training and implementation activities. Organizations committed to complete the following activities: •

Provide turnover and vacancy rate information at the beginning of the project and annually for the three years of the project

•

Send two or three organizational representatives to the two National Training Institutes for Frontline Supervisors in Minnesota including paying for part of their travel expenses

•

Assure adequate commitment of staff time and investment in project activities and training sessions

•

Provide equipment such as computers, on-line access, and printers for participation in the College of Frontline Supervision and Management (CFSM)

•

Train at least 80% of the Frontline Supervisors in their organization (or region within the state of the organization) on the Removing the Revolving Door curriculum.

5

DSP Workforce Development Demonstration •

Build the capacity of other organizations in their region by teaching at least 10 - 25 Frontline Supervisors in other agencies to implement strategies from the Removing the Revolving Door curriculum.

•

Participate in all project related training and technical assistance activities.

Project Staff Project activities were planned and implemented by staff members who had been trained as trainers on best practice strategies to improve recruitment and retention outcomes. A technical assistant was assigned to each organization. Most technical assistants had been involved in creating and pilot testing the Removing the Revolving Door curriculum and all were trained to deliver technical assistance and training on workforce development issues. All had co-authored chapters in the textbook developed by the Research and Training Center on Community Living on staff recruitment and retention strategies (Larson & Hewitt, 2005) or had coauthored online curriculum for the College of Frontline Supervision and Management in the College of Direct Support on this topic. Education and Capacity Building A training and technical assistance model was developed to assist partner agency representatives to master best practices in workforce development and supervisory staff development, and methods to teach these practices in engaging and effective ways. The distance-learning phase began with an initial phone conference with all partner organizations, the independent evaluator, and project team members to review project activities, describe the learning plan and associated resource materials, and offer participants an opportunity to connect with others and ask questions. These preparation activities helped participants “get organized,” review pre-workshop learning assignments and phone conference schedules, and plan for the first national training institute workshop where they would learn how to teach workforce development planning and intervention strategies to frontline supervisors as well as managers and administrators within their respective agencies. A course syllabus detailing weekly readings and phone conference dates and learning obectives was distributed. Organizational participants read weekly assignments from the Removing the Revolving Door curriculum

6

DSP Workforce Development Demonstration (http://rtc.umn.edu/publications/#training). Project staff members facilitated teleconferences with each organization to review and discuss assignments to enable participants to gain an initial awareness of the theories and skills reviewed in more detail and mastered in the onsite workshop. In May 2004, participants traveled to attend a 5-day train the trainer institute on workforce development challenges, interventions, and training strategies. Two or three representatives from each organization participated in a comprehensive, evidence-based educational intervention designed to build capacity in human service organizations to reduce turnover rates, improve recruitment and selection capabilities and deliver more effective training to their frontline supervisors and direct support employees, and to teach others how to achieve these outcomes. The first-year institute helped participants understand national workforce development challenges and master evidence-based workforce development intervention strategies, including organizational assessment, intervention planning, frontline supervisor development and effective adult education strategies. Following the training institute, designated project staff members provided ongoing technical assistance to each organization. Teleconferences were scheduled to share additional information on assessing workforce challenges and help in developing and implement intervention strategies matched to these challenges. The technical assistant was also available by phone and email for consultations throughout the project. He or she provided mentoring on workforce development intervention strategies and offered additional assessment tools and resources needed by agency staff to carry out their Workforce Development Intervention Plans. Those tools include planning worksheets, data base templates, sample structured interview questions, realistic job preview selection guidelines, baseline worksheets, exit questionnaires, staff satisfaction surveys, organizational commitment surveys, and training needs assessments to name a few (many are available in PDF format at http://rtc.umn.edu/dsp/index.html). Project staff also conducted least one on-site TA visit to each participating organization to share project processes and expectations, gather information and foster effective communication between the technical assistant and partnering organizations, discuss and evaluate baseline and ongoing data collection efforts, and to provide organization specific training.

7

DSP Workforce Development Demonstration Initially the technical assistant provided extensive structured assistance to agencies as they develop and implement their first workforce development plan. Over time, the nature of the assistance shifted to supporting the agency representatives as they provide technical assistance to their frontline supervisors using the skills they gained in the training institute workshop and through other technical assistance provided by the TA consultant. A second face to face training institute was held in May 2005 as a peer-learning vehicle where partners had the opportunity to describe the nature of the activities undertaken in their agencies, reflect on barriers and pathways to success, and obtain feedback and ideas from the other partners. Most partners described the “learning community” aspects of the meetings, including peer feedback, brainstorming, benchmarking, and discussion as the most important experiences of the project and a process that contributed significantly to their ability to stay motivated and receive guidance about change. Along with sharing what they had learned, participants developed plans to sustain their Workforce Development Initiatives beyond the scope of the project. In the final phase of the project, each agency developed a plan to deliver the train-the-trainer and technical assistance model to other community human service agencies in their geographical region. Five of six partners completing the project provided external training on workforce development intervention plans extending the community of learning beyond agency borders. In this phase, the learners became the teachers, technical assistants, and mentors to other agencies like their own. Training Content: Workforce Development Intervention Strategies The content of the curriculum was defined and refined as a result of research on frontline supervisor and direct support professional workforce development issues and through several years of presentations of one and two day workshops to several thousand frontline supervisors throughout the U.S. and Canada (e.g., Larson & Hewitt, 2005). In addition, extensive technical assistance had been provided to organizations by project staff members (e.g., Kansans Mobilizing for Direct Support Workforce Change, 2005, Hewitt, Larson, Sauer, Anderson, & O’Nell, 2001).

8

DSP Workforce Development Demonstration The Removing the Revolving Door curriculum includes modules that cover the following topical areas and competencies: 1. The role of FLS in influencing DSP recruitment, retention and training outcomes, including their ability to identify the impact of recruitment and retention issues on consumers, DSPs, and the organization; effectively participate in and communicate about organization-wide activities to address these issues; identify a range of participatory management techniques, and use strategies that collaboratively involve DSPs in management decisions. 2. Using effective recruitment and selection strategies to find and hire DSPs who will stay, including tapping new recruitment sources; using structured interviews and other selection methods; providing balanced and clear information about job roles and expectations; and clearly differentiating the importance of recruitment and selection strategies. 3. Effectively using socialization, orientation, mentoring and training strategies including using formal and informal training, orientation, and mentoring practices to respond to the needs, desires, and interests of new employees; using accurate competency-based DSP job descriptions to develop training and to conduct performance appraisals; and coordinating and participating in DSP orientation and in-service training. 4. Effectively using team building, conflict management and employee recognition strategies including understanding the importance of recognition in promoting job satisfaction and organizational commitment; matching specific recognition techniques to the needs of individual DSPs; and enhancing staff relations by using effective communication skills, encouraging growth and selfdevelopment, facilitating teamwork, employing conflict resolution skills and adequately supporting DSPs. 5. Effectively using, selecting, implementing and evaluating targeted recruitment and retention interventions including working with stakeholders (e.g., consumers, families, DSPs, agency administrators) to establishing baselines, selecting intervention strategies to address identified problems, and implementing and evaluating the outcomes of selected interventions.

9

DSP Workforce Development Demonstration Several research-based intervention strategies designed to reduce vacancies, turnover and to improve training are covered in the Removing the Revolving Door curriculum and the on-line version of the College of Frontline Supervision and Management curriculum. These strategies include but are not limited to: 1. Using inside recruitment sources- Involving current employees, consumers and family members, volunteers and/or board members who are familiar with the agency and the type of work, and who have an investment in new employee success to recruit potential DSPs. 2. Realistic job previewing (RJP) - Involving current DSPs, consumers, and family members in providing a balanced description of the positive and negative aspects of the job to potential employees before they decide whether to take the job. 3. Structured interviewing - An interviewing method that solicits accurate information from a candidate relating their previous experiences to their potential success in the job. 4. Socialization - Methods for helping new employees to connect positively with existing employees, consumers, and to buy in to the agency mission and vision. 5. Mentoring - Assisting employees in socialization, developing new skills and in connecting with other employees through peer mentoring programs. 6. Effective orientation - Helping new employees feel welcome and a part of their agency/family and confident in their jobs. 7. Improved training practices - Using adult learning principles, competency-based training, and other methods that assure employees fully understand and can implement their job duties. 8. Job carving - Restructuring jobs to help employees be successful by reducing the number of duties and the amount of training required to learn new skills. 9. Supporting immigrant workers - Understanding the unique needs and challenges of being a new American in the workplace, adapting the workplace to be welcoming to all employees. 10. Recognition – Promoting networking and career advancement opportunities, using effective formal and informal recognition strategies, and remembering long-term DSPs.

10

DSP Workforce Development Demonstration 11. Team building- Using effective conflict management techniques, building camaraderie. 12. Participatory management- Providing opportunities for DSP voices to be heard, involving DSPs in management decisions, and reducing hierarchy. 13. Evaluating recruitment, retention and training outcomes- Developing accurate baselines and using data to diagnose needs and evaluate the effectiveness of interventions. Instrumentation Participating organizations completed baseline and annual assessment surveys measuring turnover rates, vacancy rates and early leaving for direct support professionals and frontline supervisors (Larson, Laliberte & Salmi, 2005). We measured turnover using crude separation rate, the number of staff who left the organization during the measurement period divided by the number of people in the role at the end of the period plus the number of people needed to be fully staffed times 100. Organizations reported turnover at the beginning of the project, after 12 months, after 24 months, and after 30 months. For the 30-month assessment, the number of people who left their positions in six months was doubled and divided by the total number of current staff plus the number of people needed to be fully staffed to estimate annual turnover. Organizations also used other assessments such as computing the cost of turnover, measuring job satisfaction to assist them in selection specific interventions to implement. Face to face training activities were evaluated used items developed by the American Society for Training and Evaluation (Van Buren,

Erskine, 2002) while online training was monitored through a satisfaction survey embedded in the College of Frontline Supervision and Management. In addition, participants completed annual surveys identifying the intervention strategies they had implemented, and the number of people who had experienced each intervention. Evaluation The project independent evaluator and national Project Advisory Committee (PAC) used formative and summative evaluation methods to assess progress outcomes and assure timeliness and

11

DSP Workforce Development Demonstration quality of products and activities. The national Project Advisory Committee (PAC) shaped, defined, modified and monitored proposed activities. Monthly project staff meetings were used to assess progress toward project goals and make any needed adjustments to accomplish project outcomes. The PAC engaged in formative development and review of project activities and products to improve the quality of those activities and products as the project was implemented. The PAC met or had a project conference call at least twice annually and remained in contact with project staff via e-mail, telephone conference calls, project newsletters, and written correspondence throughout the project. The evaluation plan included both formative and summative components. The formative evaluation strategies in year one involved focusing on strengthening project approaches, defining projected outcomes and achieving consensus on the measures that will be used to evaluate outcomes. The summative phase of the evaluation began at the end of the first year. It included measurement of project level outcomes predicted for that period and analysis of the correspondence of these findings to predicted outcomes. Summative methods concluded with measurement of project level outcomes predicted for year three and an analysis of the project’s success in achieving its stated goals using a logic model as a framework for analysis. Logic models provide a graphic representation of how project or programmatic activities relate to the theories basic to the project’s design and to proposed outcomes. The logic model approach offers a means of defining and measuring similar and unique outcomes across distinct activities and approaches, also known as “theory based” evaluation (Chen et al. 1989; Finney, Moos, & Rudolf, 1989; Fitz-Gibbon, Morris, & Lyons, 1996). This approach was used to identify false assumptions, misguided applications of theory, insufficient resources and other flaws in the early stages of project design and implementation. The evaluator and the project staff collaborated to reach consensus on the problems or issues the project was confronting, the theories that formed the basis and structure of the project’s design in response to the perceived problem, and an examination of the strategies selected as the key interventions and the hoped for outcomes. The process of designing a logic model is a formative evaluation method in itself as it provides an opportunity early in a project’s life for the research team to grapple concretely with

12

DSP Workforce Development Demonstration key aspects of the project. This information provides a useful map to determine how to evaluate the project’s progress and impact. Once these concepts were extracted, they were recorded in a “logic model” (see Table 1). The model served as a way of examining individual pilot efforts. While project expectations across pilots were the same, each site was a unique organization in a different state. The logic model makes it possible to understand multi-site pilots with disparate characteristics by using common theories that cut across all of the individual pilots thus forming holistic evaluation criteria applicable to the effort as a whole. Insert Table 1 about here Participants completed surveys following the annual technical assistance workshop and responded to annual key informant interviews. The independent evaluator also conducted observations of this institute in project year one. Data from the baseline and annual surveys were used to analyze the project’s movement toward desired outcomes. The independent evaluator provided both written and oral feedback to the project team as a formative evaluation activity to help the project team make continual course corrections to improve the project. Evaluation examined the logic model dimensions to determine the project’s impact on the key issues: •

Training model design - The effectiveness of the model in disseminating expert knowledge about frontline supervisor skill development;

•

Model sustainability - The presence of a community of learning and practice to renew and sustain learning and application of learning over time;

•

Curriculum relevance - The relevance of the curriculum content and the level of application of the newly acquired skills and knowledge on the job, and;

•

Implementation Success - The ability of partner organizations to implement plans that effectively improved supervisor competence and improved DSP retention. Results and Discussion

Training Model Design

13

DSP Workforce Development Demonstration The project design accommodated the decentralized nature of community based service organizations and the field’s challenge in funding and delivering high quality employee development programs. Design elements included: the selection of partner organizations through a competitive process; the use of hybrid distance learning activities combining on-line learning with telephone meetings leading to a capstone five day workshop; and ongoing technical assistance by the project team supporting the partner organization’s development of a plan to replicate training activities within their organizations and with other human service organizations in their regions. Model Sustainability This project was designed to demonstrate an effective model for moving expert knowledge beyond the partner agency borders to others in their region by replicating the training that partners received from expert trainers. Initially, the evaluation focused on the extent to which knowledge about frontline supervisor performance was disseminated to and applied by partner agencies. This is a necessary pre-requisite building regional capacity in the project content and practices. The evaluation of whether the project has penetrated the region surrounding each agency and whether it has sustained beyond the boundaries of project resources was conduceted in the last year of the project. In interviews conducted early in the project and at the end, partners were unanimous in their support of the project’s success in creating a “community of learning and practice” among the partner agencies. They cited the annual on-site training institutes as key to promoting the learning community and derived great satisfaction from networking with peers across the country and having the opportunity to discuss similar challenges in a supported environment. The project team received high praise for their level of expertise in bringing the community of partners to a deeper understanding of recruitment and retention practices through technical assistance and “train-the-trainer” programs that modeled effective active learning adult education methods. The project’s expert knowledge penetrated each participating agency fulfilling the project goal of creating a pocket of expertise in workforce development that could further disseminate knowledge throughout a region.

14

DSP Workforce Development Demonstration In their final interviews, partners reflected on the project goal of reaching beyond their agency boundaries to build regional capacity in workforce development by replicating the lessons they had learned from the workforce experts. Most partner agencies reported a strong and enduring community of learning and practice with greater capabilities in workforce development. Five out of the six final partners also fulfilled their external training goals described in their intervention plans indicating that the model is effective in extending the community of learning beyond agency borders. Barriers to outreach beyond partner agency borders reported by partners included: no resources for marketing; regional conditions affecting interest in training; partner problems with meeting internal agency project goals; and limitations in training budgets. For example, one agency had difficulty extending the learning across the divisional borders of their own agency making it difficult to focus energy outward when their own “house” was not yet in order. Another agency met their outreach goal and had trained 21 people throughout the state and hoped to do more in the subsequent year but received a response from just three people. These barriers were not unexpected because the project did not provide funds to assist partners in marketing their training to others, and many employers were focused on basic survival – finding people to fill a high number of vacancies generated by the booming economy. A third small agency met its goal of training 10 people from other agencies and ended up training 16. Of these, 12 committed to train supervisors in their own agencies using RRD curriculum. The CEO of this agency lead their NTIFFS team which may have helped support its success. The agency that did not succeed as well in dissemination did not benefit from direct involvement by the CEO during the project. Another CEO-led team mentored 10 other agencies in RRD skills. Their approach evolved as they became more experienced and they realized they needed to enter a written agreement with mentee agencies stating the time commitment and expectations. Mentee agencies also needed assistance from the mentoring agency in obtaining agency “buy-in” from leadership. This agency was most successful conducting onsite group meetings. They had less success in completing online modules. They also

15

DSP Workforce Development Demonstration recommended avoiding Course 4 of the CFSM as the first course for new learners due to its focus on statistics. Two project partners representing larger agencies located in the same state offered jointly a “Train the Trainer” project for other providers and trained 19 participants in two separate sessions covering the northern and southern tiers of the state. In addition, these partners provided ongoing technical assistance to the providers to implement the RRD curriculum in their organizations. One of these partners also developed and implemented an organization specific “New Supervisor Orientation” quarterly. Information specific to performance management, HR law, quality improvement, licensing, and finance was included, along with a reference manual and guidelines. That multi-state organization rolled out the training throughout their national network following successful completion of the demonstration project. Based on interviews with partners, facilitators of successful dissemination included: leadership involvement; formal provider networks; collaboration across agencies; and well-capitalized agencies that can afford to mount marketing and training in anticipation of revenues. Participating organizations were highly engaged as a community of learning and practice. All agencies transferred the expert knowledge to their own agencies with five of the six agencies motivating agency wide participation in high performance practice and one larger agency achieving division wide dissemination. Five of six participating agencies met their goals for external dissemination suggesting that the project was successful in creating a model for disseminating expert knowledge with the potential to provide a sustained chain of capacity building. Training Relevance The most widely used interventions implemented by participating organizations were Realistic Job Preview Videos (RJPs) shown to job applicants during the hiring process (100% of participating organizations adopted this practice and 2,021 people viewed an RJP during the project) and the revision of job candidate interview, welcoming and orientation processes (88% of partners engaged in these activities with these interventions experienced by 1,708 people; see Table 2). Three fourths of the

16

DSP Workforce Development Demonstration organizations reported improved statistical tracking of staffing indicators including turnover and retention data. Most organizations also implemented most of the other tracked interventions. Penetration fell below two thirds participation in only 4 of 30 interventions tracked: experiencing promotional opportunities; participating in credential or apprenticeship programs; participating in College of Direct Support (CDS) training; and providing training other than RRD for other agencies. These less used strategies come with high costs (promotions, providing additional training to other agencies), high difficulty, or both (credentialing and apprenticeship). Online training was available to supervisors and direct support professionals through the project. In all, 101 DSPs completed one or more on-line lessons in the College of Direct Support and 317 supervisors and managers completed one or more on-line lessons in the College of Frontline Supervision and Management. Insert Table 2 about here In a survey completed at the end of the first training institute, participants reported that the content of training was extremely well aligned (4.7 on a 5 point scale where five is “to a great extent”) with the challenges they meet every day on the job. They also reported that the training transformed their performance on the key training objectives significantly (4.8) and improved overall job performance significantly (4.6). Participants rated all aspects of the experience: administration, content, design, instruction and impact above 4 on a five point scale where five was the highest rating. These ratings exceed national normed indicators for judging the effectiveness of training (Van Buren & Erskine, 2002). Comments offered by participants about what worked well in the project and how they benefited from participating include: •

“The supervisors that are going through the curriculum are very eager with the information and excited about implementing these new ideas. So I would say one of the biggest benefits is seeing the mind sets of the supervisors changing with this project.”

•

“Seeing our group of staff who have the same responsibilities but who have not worked closely together, come together and work well together. This project will begin to allow the Frontline Supervisors and DSPs to become more involved instead of just being told what will happen.”

17

DSP Workforce Development Demonstration •

“Personally, I have become more inspired about my job and my contribution to the organization. I feel that this program is embraced throughout our management and I have the privilege of being the operation to introduce this program to the foundation.”

•

“This was a once in a lifetime opportunity. We appreciate your hard work, guidance, and support. We will always be impacted by this training. We always are discussing what we learned and how we can implement this into our program. In final interviews conducted with five of the six completing agencies, partner leadership teams

rated the overall project impact on supervisory skills, attitudes and application somewhat lower than in initial stages but still meeting nationally established norms for effective training (Van Buren & Erskine, 2002). Informants reported that the impact on skills and attitudes was 3.8 on a scale where five indicates significant positive impact. Barriers to improved competence identified by informants included the presence of longer-term staff who found it difficult to change and/or were less motivated to change; the compressed timelines of completing training that made it harder for staff to assimilate and master training; and the lack of resources to implement project activities making it harder to move information through agencies in some cases. Success in achieving an impact did not appear to be associated with agency size but more with agency’s organization of work. For example, hierarchical organizations that extend less autonomy to teams fared more poorly in skill development. Partners worked within existing budgets and staffing to complete the additional responsibilities of project work. Making the project work in a low resource environment was possible because of the commitment of agency leadership to the project. Readiness to change as indicated by the presence of agency activities (outside of the project) focused on making some changes in professional development. Pathways to change noted in interviews included: becoming more aware of data and using it to track progress; greater knowledge of effective practice; better communication with peers about high performance practice; mentoring; the high relevance of the content to the supervisors being trained; and

18

DSP Workforce Development Demonstration the attention that the project focused on supervisors providing recognition and understanding of their need for development. When queried about the extent of RRD application by supervisors on the job at the end of the project, partners reported an average rating of 3.8 where five is “significant application.” Barriers to application included perceptions that supervisors required a high level of prompting and guidance to apply the new skills (this was cited in a setting of a larger, more centralized facility); and excessive job demands on supervisors diminishing opportunity for quality work, especially being on-site with their staff. Facilitators to application of skills included agency wide support of applied interventions (e.g., putting new staff photos and “welcome” message in the agency newsletter to support a new welcoming process); shift in values and expectations (e.g., moving away from the goal of “filling the shift” to the goal of “hiring the right person”); coordination of intervention with non-supervisory staff (e.g., change in receptionist role in a revised hiring process); providing incentives to mentors; and development of a professional attitude especially focusing on positives rather than negatives and positive reinforcement of skill application. Organizational changes and performance outcomes reported included improved communication, better teamwork, and a realignment of mental models of work. Those interviewed confirmed the project’s relevance and contribution to the participating organizations. They also confirmed that knowledge and theory offered by the project was applied in the day-to-day work activities of participating supervisors. Were the partner organizations able to develop and implement their intervention plans and improve performance and retention of supervisors and direct support staff? Each partner agency developed a customized intervention plan based on their organization’s conditions, needs, strengths and barriers to workforce development (See Table 2). The key informants reported successes and challenges in bringing knowledge to others beyond their agencies’ borders. Four of five key informants reported collecting data on participant satisfaction with the training they had

19

DSP Workforce Development Demonstration provided and these data indicate high levels of learner satisfaction with their learning experiences suggesting that the organizations had succeeded in offering high quality training. Will activities sustain beyond the three-year project period? Key informants discussed their plans for carrying project activities forward with all reporting plans for some level of continuation. One large agency planned to limit its future work to internal agency training remarking that turnover has improved so much that it is motivating to continue. Another agency planned to continue the internal transformations it had achieved through project interventions including: mentoring, improved interviewing and job descriptions, and employee recognition programs. The third planned to continue with some external training and focus as well on reinforcing new knowledge internally and sustain changes they had made in instituting an RJP, and revised interviewing and orientation. The fourth agency planned to continue both external and internal activities including RJPs, Behavioral Interviewing as well as training new supervisors. The final organization will focus internally on training new supervisors and will use courses in the College of Frontline Supervision for continuing education of supervisors. Project Outcomes Six of eight organizations completed the project. Two partners from a three-organization coalition in one state did not complete the project. The six partners from five states who finished the project reported significant decreases in direct support professional turnover. On average, DSP turnover declined from an average of 28% (four partners) in 2003 to 23% (six partners) in 2006 (See Table 3). For the four organizations that provided data all four years, turnover declined from an average of 28% to an average of 18%. For the two organizations that provided data for 2004 through 2006, turnover declined from 57% to 35% (these organizations applied as a coalition so 2003 data were not available). Across the six partners, DSP turnover declined from 40% in 2004 to 23% in 2006. Of the two organizations that did not complete the project, one provided only one year of turnover data and the other reported DSP turnover of 86% at baseline, and 85% after one year. Insert Table 3 about here

20

DSP Workforce Development Demonstration Four of six partners reported decreases averaging 6.5% in frontline supervisor turnover. Two organizations reported increases in FLS turnover. Overall, frontline supervisor turnover increased from an average of 16% (four partners) in 2003 to 20% (six partners) in 2006. For the four organizations that provided data all four years supervisor turnover declined from 16% to 12%. For the two organizations that provided data for 2004 through 2006 supervisor turnover declined from 39% to 36%. Despite the implementation of Realistic Job Previews, an intervention intended to reduce the turnover of newer hires, the number of DSP leavers within six months of hire rose on average over the project period. Only two of the six organizations reported improvements in this outcome (in both cases, the improvement was dramatic). Participants reported the following as the most significant positive outcomes of the project: •

The creation of a competency based performance evaluation system that provides wage increases for increased competence

•

Increased competence of frontline supervisors that positively influence staff and clients

•

Significant decreases in turnover

•

Training provided opportunity for cross agency networking among supervisors

•

Instituting a mentoring program helped staff to feel important and valued

•

Knowing that you are able to effect change and not powerless in reducing turnover

•

Changes in marketing jobs leading to more applicants and cost savings by eliminating news ads and using a web page and solid employee cash incentives for finding job candidates

•

Changing the job title from “residential trainer” to direct support professional and people feel more valued

•

Reducing 6 month turnover through an RJP and high performance interviewing processes

Advice offered by participants to organizations considering a project such as this one included: •

Don't be too ambitious. Pick a small part of the project and try that first, before moving on to larger projects.

21

DSP Workforce Development Demonstration •

Allocate sufficient resources to remain on course- and then consider doubling the resources.

•

Go for it! This is a great experience and whatever you do - big or small - will impact your recruitment and retention.

•

Teach the RRD in conjunction with the CFSM. And don't rush it! By being "gung ho" and teaching all the RRD components in 6 weeks as outlined by my supervisor did not, in my opinion, allow supervisors to assimilate and "buy in" to the concepts. Therefore, we have had to do much re-training on an individual basis if someone is interested in new recruiting and retention strategies.

•

They must have the passion and commitment for removing the revolving door because there is a lot of work involved in this project. However, the results are incredible and it is well worth the time and energy.

Other comments included: •

A very valuable and beneficial project. Would have been useful to have a face-to-face meeting at the end of the project. (Note: The partners gathered in Montreal Canada after the evaluation was complete for a presentation of project outcomes to the 2006 American Association on Intellectual and Developmental Disabilities Annual meeting).

•

The national networking was great and the [project staff] folks were a pleasure to work with.

•

This was a once in a lifetime opportunity. We appreciate your hard work, guidance, and support. We will always be impacted by this training. We always are discussing what we learned and how we can implement this into our program

•

Some of the modules of the CFMS would be more effective if they included segments that could be used by supervisors in training/reinforcing the concepts with DSPs (especially the teamwork module). It would be nice to have a PowerPoint attachment that FLSs could download and use to help their teams evaluate and develop their teamwork.) It could also be beneficial to be able to print lessons so the completed hard copy could be referred to in the future.

22

DSP Workforce Development Demonstration •

The tools are effective and once embraced can easily be adopted it into your culture. In summary, participants reported being very satisfied with the quality of support and information

they received receiving from the NTIFFS project team and that they are applying these skills and practices within their agencies as well as beyond their agencies validating further use of the NTIFFS approach with some important modifications discussed in preceding sections.. This project shaped powerful transformations in the workforce development practices of participating organizations and facilitated positive outcomes in improving recruitment and retention as well as transforming the thinking and communication at the agency level about workforce development. These agencies now serve as a new and rich source of expertise in workforce development within their regions. A remaining challenge for these agencies will be to find regional resources to provide a foundation for the ongoing dissemination of the important project lessons and fueling the movement to strengthen this vital workforce. Conclusions This project was implemented to disseminate knowledge about effective workforce development strategies focusing on improving supervisory performance as a key lever for change. This mission extended to building capacity within selected organizations to serve as an enduring and sustaining source of expertise and training within their organizations and for others in their region. Each of the partner organizations was, therefore, seen not only as a partner learning how to teach their own employees new skills, but as a catalyst for sparking and sustaining a regional community of learning and practice focused on best practices in frontline supervision. The project’s mission consisted of building, demonstrating and evaluating a distance learning, train-the-trainer, and technical assistance model that would be replicable and sustainable beyond the scope of the project. The educational model disseminated expert knowledge from national to local levels and stimulated local expertise and dissemination. While several national and local employer associations hold annual conferences and websites to share workforce development information, these sources offer little in-depth information about the supervisory role. Moreover, the conference format is a poor vehicle for

23

DSP Workforce Development Demonstration assuring focused learning that is retained, applied and taught with accuracy and fidelity. Also, there are few academic certificates or degree programs that provide this content. The project’s content and methods are thus an important response to a growing national problem. Evaluation results showed high levels of participant satisfaction, especially with the onsite institutes and the high level of ongoing technical assistance they enjoyed. All participants who sustained their commitment to the project were able to achieve important outcomes including transferring knowledge and practices to their own organization, seeing outcomes of reduction in turnover and higher job satisfaction, and, in most cases, disseminating this knowledge to peer organizations in their region. Participants liked having the institute in close time proximity to the completion of the prelearning segment. It enabled them to maintain momentum on their intervention planning while ideas remained fresh. The power of the on-site institute was that people entered with knowledge, knowing that technical assistance was available and they were able to connect their initial plans with what others were doing and were not afraid to share ideas and learn from others. After the first institute workshop, participants shared what they had learned within their own agency. They provided training on workforce development intervention to key supervisors with the expectation that these supervisors would develop specific workforce intervention plans for the work sites they supervised. Most agencies use the Removing the Revolving Door (RRD) curriculum as the basis for this training and presented the information over a series of several training workshops to frontline supervisors and other supervisors and human resource managers. Some agencies also use the book, Staff Recruitment, Retention, & Training Strategies for Community Human Services Organizations (Larson & Hewitt, 2005), to share information about workforce development with supervisors and others in their agencies. Organizations also used the on-line College of Frontline Supervision and Management for training supervisors in their organizations. The combination of providing a train-the-trainer learning format and technical assistance helped participants improve their knowledge and understanding of recruitment, retention and effective training techniques needed to maintain high quality direct support workforce. The projects focus was to train

24

DSP Workforce Development Demonstration “change agents” within the participating organizations in practices aimed at training and supporting frontline supervisors (FLS). This in turn provides a more stable and better-trained direct support professional (DSP) workforce, which ultimately leads to better services and supports for people with disabilities. By the end of the project, participants had the skills, knowledge and ability to provide the same type of training and technical assistance to other organizations and FLS. Participants learned research based methods for addressing workforce development challenges in community human services agencies. The result was to lower agency costs for hiring and retaining workers and improved supports to the people receiving services in these agencies. Regionally based “experts” developed skills and knowledge in Workforce Development strategies. This created a group of trained experts able to share their knowledge about DSP Workforce Development Planning with other agencies, thus, spreading the benefits of the project to others. This demonstration project provided intensive training and support on recruitment, retention, and training interventions over three years to organizations in five states. Of the eight organizations, six completed the full three years. Two organizations dropped out when they experienced changes in organizational leadership and substantial increased local competition for workers from higher paying industries. All eight organizations completed the first round of training and implemented at least some of the interventions. The six organizations that completed the project reported improvements in direct support turnover associated with implementation of a wide array of interventions introduced through project activities. While many encouraging results were reported by participating organizations, the project was not without challenges. The initial application for funding for this project was rejected because of concerns that the model would be difficult for organizations to complete. This concern was addressed by requiring applicants to identify several people who would participate in training as a team. This approach was successful in sustaining the involvement of all five selected states and six of the eight individual project partners. The teams that included a Chief Executive Officer from the organization were a bit more successful in sharing what they learned with other organizations in their organizations or states.

25

DSP Workforce Development Demonstration Another key to success in this project was the opportunity to select the top five applications from a pool of 100. The selected organizations knew that they had been chosen through a highly selective process and were highly motivated to succeed. The individuals selected by organizations to participate in this project were highly skilled trainers and administrators who took the materials made available through the project and enhanced and customized them for their organizations. This combination of talented trainers and highly motivated organizations is not always easily replicated. A key learning was that this intervention model requires substantial sustained commitments from participants. Future research and evaluation is needed to identify traits of the most successful organizations that can be discerned during the application process for technical assistance and training projects. Given the costs of providing two face-to-face learning institutes along with sustained individual technical assistance over a three year period of time, identifying organizations most likely to benefit from this approach will help target limited resources most effectively. A clear limitation the evaluation of this demonstration project is that the final quantitative results are based on just six organizational participants. Further investigation is needed to learn whether and under what conditions these results can be replicated on a wider scale. On the other hand, while only eight organizations started and only six completed the project, 447 supervisors, and as many as 2,021 direct support professionals experienced at least one of the interventions implemented within the participating organizations, and another 143 supervisors or managers outside of those organizations received at least some training on improving recruitment and retention outcomes. Other demonstration projects such as those funded by the Centers on Medicare and Medicaid Services (CMS) Demonstration to Improve the Direct Service Community Workforce project have reported similar outcomes using various combinations of the interventions covered in this curriculum (University of Minnesota, 2007). References

Association of National Community Options and Resources (ANCOR). (2001). ANCOR staff vacancy/turnover study. Alexandria: Author.

26

DSP Workforce Development Demonstration Bachelder, L. & Braddock, D. (1994). Socialization practices and staff turnover in community homes for people with developmental disabilities. Chicago: University of Illinois, Institute on Disability and Human Development, College of Associated Health Professionals. Barry Associates. (1999). The Ohio Provider Resources Association 1999 salary and benefits survey. Columbus: OPRA. Bickman, L. (1997). Resolving Issues Raised by the Fort Bragg Evaluation: New Directions for Mental Health Services Research. American Psychologist, 52(5), 562-565. Bickman, L. (1996). The Application of Program Theory to the Evaluation of a Managed Mental Health Care. Evaluation and Program Planning, 19(2),111-119. Bond, G. R., Becker, D.R., Drake, R.E., & Vogler, K.M. (1997). A Fidelity Scale for the Individual Placement and Support Model of Supported Employment. Rehabilitation Counseling Bulletin, 40(4), 265-284. Braddock, D., & Mitchell, D. (1992). Residential services and developmental disabilities in the United States: A national survey of staff compensation, turnover and related issues. Washington, D.C.: American Association on Mental Retardation. Brekke, J. S. Test, M. (1987). An Empirical Analysis of Services Delivered in a Model Community Support Program. Psychosocial Rehabilitation Journal, 10(4),53-61. Buckingham, M., & Coffman, C. (1999). First, break all the rules: What the world’s greatest managers do differently. New York: Simon & Schuster. Chen, H. & Rossi, P.H. (1989). Issues in the Theory-Driven Perspective. Evaluation and Program Planning, 12, 299-306.

27

DSP Workforce Development Demonstration Cohen. (2000). Focus in the front line: Perceptions of workforce issues among direct support workers and their supervisors: A staff report. Boston. Costner, H.L. (1989). The Validity of Conclusions in Evaluation Research: A Further Development of Chen and Rossi's Theory-Driven Approach. Evaluation and Program Planning, 12, 345-353. DeLeon, P. H. & Williams, J. G. (1997). Are Psychologists Collectively Willing to Accept Unpopular Findings. American Psychologist, 52(5), 551-552. Feldman, S. (1997). The Fort Bragg Demonstration and Evaluation. American Psychologist, 52(5). Finney, J. W. & Moos, R. H. (1989) Theory and Methods in Treatment Evaluation. Evaluation and Program Planning, 12, 307-316. Fitz-Gibbon, C., Morris, T., & Lyons, Lynn. (1996). What Do We Mean by Theory-Based Evaluation? Evaluation Practice, 17(2), 177-184. Furlong-Norman, K. (Ed.). (1996). Community Support Network News, Center For Psychiatric Rehabilitation, 11. Grusky, O. & Tierney, K. (1989). Community Mental Health Journal, 25(1), 3-20. Hewitt, A. (1998). Identification of competencies and effective training practices for direct support staff working in community residential services for people with disabilities. Minneapolis: University of Minnesota. Hewitt, A.S., & Larson, S.A. (2007). The Direct Support Workforce in Community Supports to Individuals with Developmental Disabilities: Issues, Implications and Promising Practices. Mental Retardation and Developmental Disabilities Research Reviews, 13, 178-187.

28

DSP Workforce Development Demonstration Hewitt, A., Larson, S., Sauer, J., Anderson, L., & O’Nell, S. (2001). Partnerships for Success: Retaining Incumbent Community Support Human Service Workers by Upgrading their Skills and Strengthening Partnerships among Workforce Centers, Educational Programs, and Private Businesses: University of Minnesota Subcontract Final Report. Minneapolis: University of Minnesota, Research and Training Center on Community Living, Institute on Community Integration (UAP). Hewitt, A., Larson, S.A., & Lakin, K.C. (2000). An independent evaluation of the quality of services and system performance of Minnesota’s Medicaid Home and Community Based Services for persons with mental retardation and related conditions. Minneapolis: University of Minnesota, Research and Training Center on Community Living, Institute on Community Integration (UAP). Hewitt, A., Larson, S.A., O’Nell, S., Sauer, J., & Sedlezky, L. (1998). The Minnesota Frontline Supervisor Competencies and Performance Indicators. Minneapolis: University of Minnesota, Research and Training Center on Community Living, Institute on Community Integration (UAP). Hewitt, A., O'Nell, S., & Larson, S.A. (1996). Overview of direct support workforce issues. In Jaskulski, T. & Ebenstein, W. (Eds.). Opportunities for Excellence: Supporting the Frontline Workforce. Washington D.C.: President's Committee on Mental Retardation. Hoagwood, K. (1997). The Fort Bragg Experiment: A Comparative Success or Failure? American Psychologist, 52(5), 546-550. Hollister, R. G. Jr., Kemper, P. & Woolridge, J. Linking Process and Impact Analysis: The Case Supported Work. Qualitative and Quantitative Methods in Evaluation Research, 141158.

29

DSP Workforce Development Demonstration Holmbeck, G. N. (1997). Toward Terminological, Conceptual, and Statistical Clarity in the Study of Mediators and Moderators: Examples From the Child-Clinical and Pediatric Psychology Literatures. Journal of Consulting and Clinical Psychology, 65(4), 599-610. Huberman, M. (1996). A Critical Perspective on the Use of Templates as Evaluation Tools. New Directions for Evaluation, 72, 99-109. Illinois Direct Support Workforce Initiative (2005). Quality Support 2005: An Agenda to Strengthen the Developmental Disabilities Direct Support Professional Workforce in Illinois. Springfield IL: IL Council on Developmental Disabilities. Johnston, K. (1998). Developmental disabilities provider direct service worker study: Results and findings. Anchorage: Governor’s Council on Disabilities and Special Education. Julian, D. A. Jones, A., & Deyo D. (1995). Open Systems Evaluation and the Logic Model: Program Planning and Evaluation Tools. Evaluation and Program Planning, 18(4), 333341. Julian, D. A. (1997). The Utilization of the Logic Model as a System Level Planning and Evaluation Device. Evaluation and Program Planning, 20(3), 251-257. Kansans Mobilizing for Direct Support Workforce Change Project (2005). Kansans Mobilizing for Direct Support Workforce Change: A statewide workforce development initiative to resolve the direct support workforce crisis. Topeka, KS: Kansas Council on Developmental Disabilities.

Lakin, K.C., & Bruininks, R.H. (1981). Occupational stability of direct-care staff of residential facilities for mentally retarded people. Minneapolis: University of Minnesota, Center on Residential and Community Services. Lamberti, M. & Katzenmeyer, C. (1996). Transforming Qualitative Data from Templates into Quantitative Assessment of Multisite Programs. New Directions For Evaluation, 72, 8189. 30

DSP Workforce Development Demonstration Larson, S.A., & Hewitt, A.S. (Eds.). (2005). Staff recruitment, retention and training strategies for community human services organizations. Baltimore: Paul H. Brookes Publishing, Co. Larson, S.A., Hewitt, A., & Anderson, L. (1999). Staff recruitment challenges and interventions in agencies supporting people with developmental disabilities. Mental Retardation, 37, 36-46. Larson, S.A., Hewitt, A., & Lakin, K.C. (1994). Residential services personnel: Recruitment, training and retention. In M. Hayden & B. Abery (Eds.), Challenges for a Service System in Transition: Ensuring Quality Community Experiences for Persons with Developmental Disabilities (pp. 313-341). Baltimore: Paul H. Brookes. Larson, S.A., Hewitt, A.S., & Knobloch, B. (2005). Recruitment, retention and training challenges in community human services. In S.A. Larson & A.S. Hewitt (Eds.). Staff recruitment, retention and training strategies for community human services organizations. (pp. 1-20). Baltimore: Paul H. Brookes Publishing, Co.

Larson, S.A., Lakin, K.C., & Bruininks, R.H. (1998). Staff Recruitment and Retention: Study Results and Intervention Strategies. Washington, DC: American Association on Mental Retardation. Larson, S.A., Lakin, K.C., & Hewitt, A.S. (2002). Direct support professionals: 1975-2000. In D. Croser, P. Paker, & R. Schalock (Eds.). Embarking on a New Century (pp. 203-219). Washington, DC: American Association on Mental Retardation. Larson, S.A., Laliberte, T., & Salmi, P. (2005). Assessing challenges and evaluating workforce outcomes. In S.A. Larson & A.S. Hewitt (Eds.). Staff recruitment, retention and training strategies for community human services organizations. (pp. 287-320). Baltimore: Paul H. Brookes Publishing, Co.

31

DSP Workforce Development Demonstration Larson, S.A., Sauer, J., Hewitt, A., O’Nell, S., & Sedlezky, L. (1998). SOS training and technical assistance project for direct support professionals, trainers and frontline supervisors: Final Report. Minneapolis: University of Minnesota, Research and Training Center on Community Living, Institute on Community Integration (UAP). Lipsey, M. W. & Pollard, J. A. (1989). Driving Toward Theory in Program Evaluation: More Models to Chose From. Evaluation and Program Planning, 12, 317-328. Loucks-Horsley, S. (1996) The Design of Templates as Tools for Formative Evaluation. New Directions for Evaluation, 72. Prouty, R.W., Smith, G., & Lakin, KC. (2001). Residential services for persons with developmental disabilities: Status and trends through 2000. Minneapolis: University of Minnesota, Research and Training Center on Community Living, Institute on Community Integration (UAP). Randolph, F. L., Blasinsky, M., Leginski, W.,Parker, L.B. & and Goldman, H.H. (1997). Creating Integrated Service Systems for Homeless Persons with Mental Illness: The ACCESS Program: Access to Community Care and Effective Services and Supports. Psychiatric Services, 48(3), 369-73. Razzano, L. & Cook J.A. Report on Sampling, Recruitment, & Fidelity Studies: Employment Intervention Demonstration Program. Scheirer, M.A. (1996). Roles for Program Templates in Evaluation: A Broader Look. New Directions for Evaluation, 72, 89-97. Scheirer, M.A. (1994). Designing and Using Process Evaluation. Handbook of Practical Program Evaluation, 40.

32

DSP Workforce Development Demonstration Scheirer, M.A. & Rezmovic E.L. (1983). Measuring the Degree of Program Implementation. Evaluation Review, 7(5), 599-633. Sechrest, L. & Walsh M. (1997). Dogma or Data: Bragging Rights. American Psychologist, 52 (5), 536-540. Shern, D. L., Trochim W.M.K., & LaComb, C.A. (1995). The Use of Concept Mapping For Assessing Fidelity of Model Transfer: An Example From Psychiatric Rehabilitation. Evaluation and Program Planning, 18(2),143-153. Test, D., Solow, J., & Flowers, C. (1999). North Carolina direct support professionals study: Final report. Charlotte: University of North Carolina at Charlotte. University of Minnesota, Research and Training Center on Community Living and The Lewin Group (2007). CMS Direct Service Workforce Demonstration: Promising Practices in Marketing, Recruitment and Selection Interventions. Baltimore, MD: Centers on Medicaid and Medicare Services. http://www.dswresourcecenter.org/index.php/dsw/content/download/384/1556/file/DSW %20Promising%20Practices%20Marketing,%20Recruitment%20and%20Selection%20FI NAL.pdf Van Buren, M. E. & Erskine, W. (2002) Learning Outcomes: ASTD’s Fourth Annual Report on Standards for Evaluating Organizations’ Investments in Education and Training. Alexandria: American Society for Training and Development. Weiss, C.H. (1996). Methods of Assessing Program Effectiveness. Evaluation Practice, 17(2), 173-175.

33

DSP Workforce Development Demonstration Table 1. Logic model for National Training Institute for Frontline Supervisors Project Issues/ problems /challenges this project addresses Problem: Knowledge of expert practice in FLS is poorly distributed Challenge: Achieving high quality & Efficient dissemination of expert knowledge to a broader group Problem: System is poorly organized to build regional capacity thru regional leadership & expanding local pockets of excellence Challenge: Building capacity of a region to train others in FLS is important Problem: FLS are not competent in critical competencies Challenge: Improve effectiveness of FLS Problem: Industry does not support deep learning & networks of expertise in DSP/FLS development

Relevant Theory

Transfer of expertvalid knowledge will improve DSP work conditions DSPs leave prematurely due to poor supervision

Local expertise is critical to sustaining interventions and renewing knowledge over time

Interventions

Design and implement a TA model structure to offer an effective, efficient and high quality TA product

Build an efficient TA Model with features to build capacity at the regional level

Outcomes

Expert knowledge is mastered and used at regional levels

Measures

Observation of training institute Key informant interviews

Process of training & TS dissemination is seen as working and efficient

Observation of local training activities by project staff

Targets for numbers trained are met.

Records of numbers trained throughout project

Trainees use information effectively Regional capacity sustains beyond initial efforts

Training will have a positive impact on FLS performance

Groups working together on joint mission creates energy, enthusiasm, sustainability

Model provides an effective way of enhancing FLS competencies

Model fosters a community of practice

Project Challenge: Creating a robust community of expert practice around training FLS

34

FLS perceive training as useful, organized, high quality FLS are using skills at work Presence of an active engaged group –working on FLS training, & ideally other training too

End of training survey and 6 month follow up survey)

Observation Key informant interviews (MT does these)

DSP Workforce Development Demonstration Issues/ problems /challenges this project addresses Problem: Industry does not pay sufficient attention to end user concerns & impact of training on quality of supports

Project Challenge: Providing satisfaction in training at each point (institute, regions, orgs) Determine impact Problem: poorly supervised DSPs leave agencies prematurely Project Challenge: Teach others how to develop cultures of competence in recruiting, supervising, mentoring and valuing re DSPs

Relevant Theory

Interventions

Project impact will increase in relationship to satisfaction of customers

Assure project interventions are high quality and responsive to the needs of customers (end users)

Impact can and should be measured

Implement Metrics that provide reliable and valid measures of important effects

More competent supervisors improves satisfaction and commitment of DSPs

Provide TA that facilitates organizational competence in identified intervention plans

System cultures that value and develop DSPs & FLS will improve quality and continuity

35

Outcomes

End users of each intervention are satisfied and see training as successful Learners at central and regional levels apply learning at work

FLS and DSPS are competent and have increased commitment to job

Measures

Reaction sheets at end of institute & 6 month follow up of institute trainees (End of training reaction sheets for 100 trainees and 6 month follow up)

Reaction sheets, site data on intervention plans, turnover measure.

DSP Workforce Development Demonstration

Table 2 Penetration of Project Interventions N Staff Project Activity Impacted Leadership Development and Agency Capacity Building Attend Removing the Revolving Door (RRD) train the trainer Partner leadership team personally complete College of Frontline Supervision and Management Courses (CFSM) Partner agencies participating in project TA calls Supervisory Development Attend RRD own agency 447 Attend other supervisory training own agency 238 Registered for CFSM online own agency 237 Registered for CFSM online other agency 156 Completed on or more CFSM lesson (3,075 lessons total) 317 Attend RRD other agency 143 Provided other sup training other agency Changes in Recruitment and Hiring Practices Viewed a Realistic Job Preview 2,021 Employees welcomed in a new way 635 Interviewed using revised procedures 552 Received revised orientation 521 Viewed customized Realistic Job Preview 469 Participated in recruitment or hiring bonus program 429 Recruited with project marketing toolkit 194 Direct Support Professional (DSP) Training and Development Participating in credential or apprenticeship programs 448 Participating in revised DSP training 222 Completed one or more College of Direct Support lesson 101 N in competency based training 92 N in mentoring program 85 High Performance Retention Strategies Participated in employee recognition program 459 Participated in promotional opportunities 446 Assess Recruitment Retention and Turnover data annually Use participatory management style Systems Change Activities Participate in professional association activities 388 Shared information with other agencies 152 Participated in conference presentations on Project 147 Provided technical assistance to other organizations 49 Legislators contacted regarding direct support workforce concerns 19 CFSM College of Frontline Supervision and Management online training RRD – Removing the Revolving Door Classroom Based Training

36

N Orgs

% of Orgs

8

100%

8 6

100% 75%

8 6 8 6 8 5 1

100% 75% 100% 75% 100% 63% 13%

8 7 7 7 5 7 5

100% 88% 88% 88% 63% 88% 63%

3 5 3 3 5

38% 63% 38% 38% 63%

7 3 8 6

88% 38% 100% 75%

5 8 5 6 6

63% 100% 63% 75% 75%

DSP Workforce Development Demonstration

Table 3 Organizational Outcomes for Project Year Organization DSP Turnover FLS Turnover 2003 A 35% 13% B 29% 25% C 26% 14% D 23% 13% E F Average (4) 28% 16% 2004 A 50% 14% B 38% 20% C 12% 0% D 27% 26% E 74% 43% F 39% 35% Average (4) 32% 15% Average (6) 40% 23% 2005 A 35% 12% B 36% 35% C 10% 0% D 25% 12% E 46% 7% F 29% 30% Average (4) 27% 15% Average (6) 30% 16% 2006 A 22% 0% B 19% 35% C 9% 7% D 21% 5% E 42% 14% F 28% 58% Average (4) 18% 12% Average (6) 23% 20%

Hires Leaving w/in 6 months

Note: 2006 outcomes are estimates based on 6 months of data

37

42% 30% 26% 34%

33% 58% 32% 26% 47% 50% 27% 41% 40% 42% 36% 14% 46% 49% 45% 35% 39% 60% 49% 11% 58% 36% 53% 44% 44%