Child welfare worker reports of buy-in and readiness

0 downloads 0 Views 216KB Size Report
This study views the extent to which staff buy-in for an organizational innovation in child welfare (CW) relates to implementation progress. The study occurs ...
Children and Youth Services Review 37 (2014) 28–35

Contents lists available at ScienceDirect

Children and Youth Services Review journal homepage: www.elsevier.com/locate/childyouth

Who's on board? Child welfare worker reports of buy-in and readiness for organizational change☆ Julie S. McCrae a,⁎, Maria Scannapieco b,1, Robin Leake a,2, Cathryn C. Potter a,3, David Menefee c,4 a b c

University of Denver, Graduate School of Social Work, Butler Institute for Families, 2148 S High Street, Denver, CO 80208, USA University of Texas at Arlington, School of Social Work, Judith Birmingham Center for Child Welfare, P.O. Box 19129, Arlington, TX 76019, USA Colorado Department of Human Services, Division of Child Welfare, 1575 Sherman Street, Denver, CO 80203, USA

a r t i c l e

i n f o

Article history: Received 1 August 2013 Received in revised form 6 December 2013 Accepted 7 December 2013 Available online 14 December 2013 Keywords: Child welfare Implementation Buy-in Readiness Practice model NIRN

a b s t r a c t This study views the extent to which staff buy-in for an organizational innovation in child welfare (CW) relates to implementation progress. The study occurs during implementation of a statewide practice model that was supported with technical assistance from the Mountains and Plains Child Welfare Implementation Center (MPCWIC) and framed around the National Implementation Research Network model. Mixed methods were used to address three study questions: (1) what is the level and nature of buy-in related to the innovation? (2) does buy-in vary according to staff characteristics, and (3) what is the relationship between buy-in, local level agency readiness, and implementation status one year after project start? Survey data were collected from 568 CW staff in 13 local county agencies and 12 implementation specialists assigned as coaches. Focus groups and interviews were conducted with 52 staff in four agencies. Bivariate chi-square analyses and multivariate regression using a cumulative logit model showed that buy-in was related to gender and agency tenure. Implementation progress was higher among smaller agencies, and agencies with lower levels of job stress. Qualitative themes centered on staff inclusivity in project design, communication, and supervisor support. Findings highlight the need to adapt implementation strategies in urban and rural locales, and to attend strongly to staff selection, supervision, and inclusion during implementation. Addressing job stress may help bolster implementation. © 2013 Elsevier Ltd. All rights reserved.

1. Introduction Research concerning the uptake of new practices in human services stresses that interventions should be viewed as compatible by the potential implementers as a key aspect of implementation success (Aarons & Palinkas, 2007; Proctor et al., 2011; Rogers, 1995). Alternatively associated with terms such as fit, change valence, appropriateness, and buy-in, the notion is that change is more likely when adopters believe that the change is necessary, important, beneficial and worthwhile (Bouckenooghe, 2010; Weiner, 2009). As early as the 1950s, organizational change theorists proposed that change begins by “unfreezing” the organization through altering the existing views of ☆ This research was supported by an award to the University of Texas at Arlington, School of Social Work, Mountains and Plains Child Welfare Implementation Center from the U.S. Department of Human Services, Administration for Children and Families, Children's Bureau (90CO1046/01). The authors wish to thank Ann Deaton Wacker and Stacie Hanson for their assistance with this work. ⁎ Corresponding author. Tel.: +1 303 871 4533. E-mail addresses: [email protected] (J.S. McCrae), [email protected] (M. Scannapieco), [email protected] (R. Leake), [email protected] (C.C. Potter), [email protected] (D. Menefee). 1 Tel.: +1 817 272 3535. 2 Tel.: +1 303 871 6813. 3 Tel.: +1 303 871 2913. 4 Tel.: +1 303 866 4379. 0190-7409/$ – see front matter © 2013 Elsevier Ltd. All rights reserved. http://dx.doi.org/10.1016/j.childyouth.2013.12.001

staff and creating the motivation to change (Lewin, 1951; Weiner, Amick, & Lee, 2008). Since then, implementation research has emphasized the need to assess multiple levels of system readiness, with most evidence still supporting two overarching components to preparing for change: staff motivation—being willing, and organizational capacity— being able (Weiner et al., 2008). Child welfare organizations are complex environments in which to introduce change. First, public child welfare agencies are large— encompassing a state governing body and multiple local service providers (CW agencies), which are county or regionally based, and can number upwards of 100 agencies in one state. Second, these local CW agencies, particularly those that are large or moderate-size, consist of multiple internal work units that perform distinct (and often siloed) functions. One hand may not know what the other is doing. Conversely, in smaller agencies, workers perform all work functions, from intake to foster care. This local diversity contributes to varying training and professional development needs, and can create tension between urban and rural locales. Finally, child welfare work is unpredictable and crisis-oriented, requiring staff to spend significant time offsite, attending court hearings or visiting with families. This makes internal communication challenging, and primarily centered on the welfare of children and families rather than on organizational strategic planning or change. How do child welfare agencies successfully implement change, and to what extent do staff attitudes influence implementation? Multiple

J.S. McCrae et al. / Children and Youth Services Review 37 (2014) 28–35

studies have been conducted to examine the adoption of organizational change and evidence-based practices in the fields of psychology, health promotion, and education (Greenhalgh, Robert, Macfarlane, Bate, & Kyriakidou, 2004; Powell Davies et al., 2012). Very little research has been conducted in child welfare agencies to understand what contributes to successful implementation in these unique environments. This study aims to contribute to what is known about introducing large-scale change in a public child welfare organization. The study uses data from over 568 child welfare staff in 13 local agencies, along with case study interviews and focus groups in a subset of four agencies (n = 52), all involved in implementing a statewide set of standard child welfare practice, values, and protocol (“practice model”). The study uses mixed methods to address three study questions: (1) what are the level and nature of buy-in related to the innovation? (2) does buy-in vary according to staff characteristics, and (3) what is the relationship between buy-in, local level agency readiness, and implementation status one year after project starts? The study draws on implementation science, diffusion of innovations theory, and organizational change management theories to guide the study questions. We hypothesize that agencies whose staff report higher levels of awareness, understanding, and belief that the innovation is worthwhile will be more likely to reach implementation compared with agencies with lower levels of awareness and buy-in. The study tests this hypothesis controlling for organizational characteristics such as readiness for change at the outset, job stress, and leadership. 1.1. Staff buy-in, organizational readiness, and the adoption of new practices The seminal article by investigators Greenhalgh et al. (2004) recommends that there not be any additional studies of the individual patterns of the adoption of an innovation. This includes the idea that there are different types of adopters, from someone who is an early adopter with a particular set of characteristics, to another who is more reluctant and a later adopter—who could then be persuaded to adopt the innovation in a wholly different manner (Rogers, 1995). Instead, because there is little empirical evidence for these adopter categories, Greenhalgh and colleagues propose a focus on (1) why and how people and organizations reject an innovation after adopting it, and (2) what from the fields of cognitive and social psychology are transferrable, given a particular innovation and circumstance or setting? Weiner et al. (2008) also stress the mix between individuals and the organization, in conceptualizing readiness for change. According to this review, individual readiness includes motivation and willingness to consider adopting new ideas at the outset, whereas receptivity and openness to change reflect one's attitudes toward change in general (Weiner et al., 2008). Later on in a change initiative, whether the change is accepted or resisted by individual members becomes relevant. In 2010, a set of outcomes was proposed to distinguish implementation outcomes in human services from service and client-level outcomes (Proctor et al., 2010). This also categorizes individual views of an innovation that are related to the innovation itself or to the practice setting (Proctor et al., 2010). Acceptability, in this framework, refers to the perception among members that the innovation is agreeable, palatable, or satisfactory, based on the members' direct experience with the innovation. The authors distinguish this from appropriateness, which refers to whether the innovation is perceived as a good fit or is relevant to a particular issue, problem, consumer, or setting (Proctor et al., 2010). These distinctions are so finite, however, that it is difficult to separate and conclude how implementation strategies might be adjusted to meet each goal. The Diffusion of Innovation theory (Rogers, 1995) describes the process that individuals go through to adopt an innovation, from knowledge, persuasion, decision, implementation and confirmation. The process includes being first exposed to the innovation, weighing the relative advantage of the innovation and deciding whether to continue,

29

adopting the innovation to varying degrees, and confirming that the innovation was worthwhile to do. Diffusion theory stresses that interpersonal communication channels are key to achieving implementation success in that individuals are most influenced by those who are closest to them, such as their immediate supervisor or team member (Rogers, 1995). In complex change, such as large-scale implementation, however, research supports that spread and implementation occur through a more messy, organic process (Greenhalgh et al., 2004). Yet, there are virtually no studies to support any given approach to achieving buy-in during practice and organizational changes (Fixsen, Naoom, Blasé, Friedman, & Wallace, 2005). Consistent leadership at multiple levels can influence the success of implementation, as well as promoting the utility of the innovation with the service users themselves (Frambach & Schillewaert, 2002; Groves, 2005). In child welfare services, agencies characterized by low rigidity (less emphasis on beauracratic “red tape”; inflexible rules and protocol) and high worker reports of feeling supported, effective, and cooperative with team members relate to higher staff morale and job satisfaction (Glisson, Green, & Williams, 2012), but research is needed to link organizational and stafflevel views to large-scale change that is characteristic of child welfare, such as practice models or moving to a statewide centralized intake system. In this context, a model for creating the change that is less “messy and organic” is needed to help frame the strategies and plan for organized roll-out across a large group of individuals and agency locations. 1.1.1. National Implementation Research Network (NIRN) In 2005, the National Implementation Research Network established a set of implementation drivers and stages based on a review of over 1000 studies and articles related to successful organizational change (Fixsen et al., 2005). The resulting NIRN framework consists of seven core implementation components or “drivers” and six implementation stages. Stages are exploration and adoption, program installation, initial implementation, full operation, innovation, and sustainability. Drivers— described as interactive processes that are integrated in the initiative to maximize their influence on staff behavior—include components such as staff selection (identifying internal qualified staff or characteristics and procedures for external hiring), consultation and coaching, and preservice training (Fixsen et al., 2005). The NIRN framework was selected to guide the implementation of child welfare systems change across the U.S. in 2008, supported by five regionally-based child welfare implementation centers that were funded by the federal Children's Bureau. State and tribal child welfare agencies implementing a change received intensive technical assistance (TA) from the regional center using the NIRN model over a period of 2 to 4 years (Armstrong et al., in press). This typically involved having one or two center staff who worked directly with the agency around implementation drivers such as obtaining leadership training for upper management or helping the agency develop a coaching model to support the planned intervention. Across the five implementation centers, there has been a predictable trajectory to installing different implementation drivers over time; for example, leadership has been a consistent focus over two years, while facilitative administration (e.g. changing policies and procedures to support the intervention) tends to become relevant in later implementation stages (Armstrong et al., in press). 1.1.2. Background on one state's effort In 2009, a state in the Rocky Mountain region of the U.S. and the Mountains and Plains Child Welfare Implementation Center (MPCWIC) began collaborative work to design and implement a statewide practice model. A practice model is a written document created by the agency to outline how the agency will function according to its mission, vision, and values. A practice model includes clear definitions and explanations of how the agency will work internally, with families, and community partners to provide child welfare services (National Child Welfare Resource Center for

30

J.S. McCrae et al. / Children and Youth Services Review 37 (2014) 28–35

Organizational Improvement, 2008). A practice model is implemented at the state-level but should permeate all levels of the organization, with an underlying theory of change, desired outcomes, practice principles, and practice standards (Barbee, Christensen, Angle, Wandersman & Cahn, 2011). These typically align with federal priorities such as services that are child-focused, family-centered, and meet core service goals of child safety, permanency and well-being (U. S. Department of Health and Human Services, Administration for Children & Families, & Children's Bureau, 2008). In 2013, at least 30 states had at least one practice model (PM) described on the Child Welfare Information Gateway (www.childwelfare.gov). Among 22 large-scale change projects supported by one of the 5 national child welfare implementation centers in 2008 to 2013, the majority (59%) focused on implementing practice models (Armstrong et al., in press). Implementing the practice model centered first on establishing agreed-upon base standards, principles, outcomes and indicators, followed by county-based Continuous Quality Improvement (CQI), peer-networking and support among the county agencies, and sharing of best practices across the state through an online Promising Practices Compendium. The initiative received implementation support from MPCWIC for 3.5 years, including having one to two implementation specialists working directly with the state child welfare division and local agencies to implement key drivers and PM components using the NIRN framework (Fixsen et al., 2005). Local child welfare agencies— nearly 60 total—were phased into the Practice Model in cohorts beginning in 2011. At the time the current research was conducted, the initiative was beginning Year 2, and the first cohort of local agencies (n = 13) had just been provided tools to implement CQI.

2.2. Sample 2.2.1. Agency staff Well over three-quarters of the staff surveyed at baseline were employed at a large child welfare agency (85%), while 11% were employed at medium-size agencies, and 5% at small agencies. The distribution by size of the 13 agencies is: small (n = 4), medium (n = 4), and large (n = 5). The quantitative staff sample was predominantly female (85%). The racial breakdown is: 7% African American, 77% White, 4% biracial or multi-racial, and 12% missing (did not report race). Nearly one-quarter of staff surveyed are Latino ethnicity (22%). Data were collected in October and November, 2011. 2.2.2. Qualitative participants In all, 22 qualitative sessions involving 52 staff were held: 6 director interviews, 5 caseworker focus groups (n = 30), 10 supervisor individual interviews, and 1 mid-level manager focus group (n = 6; in the large agency). In the large agency, caseworkers and supervisors were randomly selected to participate, whereas in the medium and smallsize agencies, all caseworkers and supervisors were invited to participate. The breakdown in the large agency was: 2 director interviews, 6 supervisor interviews, 2 caseworker focus groups (n = 16), and 1 mid-level manager focus group (n = 6). Data were collected June through October, 2012 in local agencies with sessions lasting 1.5 h. 2.2.3. State Implementation Team participants The 12-month follow-up survey was completed by 12 state-level staff assigned in pairs to work directly with each agency. One of each team member rated each agency. State staff were predominantly masters-educated (73%) with an average 14 years experience in public child welfare (range = 2 to 38 years). Data were collected in September, 2012.

2. Methods 2.3. Measures 2.1. Design This is a mixed methods study incorporating an explanatory sequential design (Creswell & Zhang, 2009). In this design, qualitative data help explain the quantitative results; here qualitative data are used to understand the construct of buy-in and its relation to implementation status. First, quantitative baseline data were collected from child welfare staff in 13 local county agencies. The agencies are the first group to implement the innovation, and were selected after submitting an application to the state child welfare division. Data collected through the baseline, online survey asked about 5 agency-level characteristics such as change readiness, and perceptions of the value of the innovation (the PM). In all, 568 staff completed the survey (58% of surveys sent). Participation rates varied considerably; in 7 agencies, 80 to 100% of staff completed the survey. Next, four of the 13 agencies were randomly selected for a qualitative case study: one large agency was selected, one medium-sized (roughly 30 staff total), and 2 small agencies (5 to 10 staff total). This was to ensure diverse agency representation by size in the qualitative data. In each agency, three sources of qualitative data were collected: (1) individual interviews with upper management, including the agency director and assistant directors, (2) individual interviews with supervisors, and (3) focus groups with caseworkers. One focus group was also held in the large agency with middle managers. Finally, concurrent to the qualitative data collection (approximately one year after the baseline survey), Implementation Specialists (12 total) who were assigned as coaches to each of the local county agencies rated their implementation status. This is the dependent variable in quantitative analyses (NIRN stages from Early Exploration to Full Implementation). One county was not rated despite numerous attempts to have the instrument completed. This agency is excluded from multivariate analyses.

2.3.1. Buy-in Four project-developed items inquired about staff's level of awareness and support for the innovation. Staff reported their level of agreement that they: (1) had seen, read, or heard about the innovation, (2) feel they have a good understanding of the innovation, (3) believe the innovation is a good fit with core needs of the agency, and (4) believe the innovation is needed. The items were summed to create a measure buy-in. The scale shows adequate reliability using Cronbach's alpha (.78). 2.3.2. Comprehensive Organizational Health Assessment (COHA) Select scales of the COHA were used to measure organizational factors hypothesized to relate to success of a large-scale organizational change effort. The full COHA is a multi-level assessment battery that is designed to measure organizational health in child welfare agencies (Butler Institute for Families, 2012). Over 200 items form 22 scales along agency, unit, and individual levels. Respondents rate items on a 6-point Likert agreement scale: (1) strongly disagree, (2) disagree, (3) disagree slightly, (4) agree slightly, (5) agree, and (6) strongly agree. Subscales measuring readiness for change, leadership, and job stress were used in the current study and are described below. Readiness for change. Elements of the Organizational Readiness for Change, Social Agency Staff version (ORC; Lehman, Greener, & Simpson, 2002) are included in the COHA battery, and the Readiness for Change subscale was used in the current study. The readiness measure consists of 24 items that factor into three subscales: change management, response to change, and learning organization. Internal consistency reliability is high (alpha = .90). 2.3.2.1. Leadership. The COHA leadership measure used in this study was developed by the Butler Institute to capture staff views of the extent of

J.S. McCrae et al. / Children and Youth Services Review 37 (2014) 28–35

distributive and adaptive leadership in the organization. Four items ask staff to rate the extent to which leaders: (1) provide visible, ongoing support for innovations and ideas, (2) view leadership roles as shared by staff and administrators, (3) encourage others to provide leadership for new projects, and (4) encourage staff to make our own decisions in work. 2.3.2.2. Job stress. An adapted version of the job stress subscale of the ORC (Lehman et al., 2002) was used in this study. Five items included “I feel a lot of stress here” and “Staff frustration is common here”. Internal consistency reliability of the job stress scale of the ORC is .83 (coefficient alpha). 2.3.3. Implementation Process Measure (IPM) The IPM was developed to measure the process of implementation in child welfare systems change (Armstrong et al., in press). The instrument inquires about the salience and level of installation (achievement) of 11 implementation drivers, and 7 implementation stages. Ratings of agencies' implementation stage were used in this study. Raters, who were state staff working closely with each agency, were provided detailed definitions of implementation stages following the NIRN framework (Armstrong et al., in press; Fixsen et al., 2005). Stages were: (1) early exploration, (2) late exploration, (3) early design and installation, (4) late design and installation, (5) early initial implementation, (6) late initial implementation, (7) early full implementation, and (8) late full implementation. Table 1 shows the definitions used by raters to assess implementation stage. 2.4. Qualitative measures A common, standard guide was used across participants with questions centered in three content areas: (1) understanding of and initial reactions to the initiative, (2) needs and experiences with implementation, and (3) agency impact. Items were included in each area, with probes to instruct the interviewer to explore further. For example, the first question for all participants was how they first found out about the initiative. Probes included whether they felt involved in decision making and what went into decision making about whether their agency would participate. The two primary buy-in items were: “what was appealing to you about the initiative?” and “what makes you feel more hesitant?”. Facilitators probed for further detail, including

Table 1 Stages of implementation used by the five child welfare implementation centers. Stage

Definition

Exploration stage

Actively considering a systems change; engaged in identifying the need for the change, the nature and scope of the intervention components of the change, the degree of awareness and support for the change, and the overall approach for designing the systems change. Actively preparing for implementation of the systems change project; including detailed design of both the intervention components and plans for their implementation, including structural and functional systems changes, and assembling the resources necessary to launch the program. Actively engaged in learning how to do the systems change project interventions, and how to support the ongoing activities of the interventions. First steps towards monitoring and supporting the use of new skills, practices, tools and strategies necessary to sustain the systems change. Actively working to make full use of the systems change interventions as part of typical functioning. New learning becomes integrated into practitioner, organizational, and community practices, policies and procedures. Staff members become skillful and the procedures and processes become normalized.

Design and installation stage

Initial implementation stage

Full implementation stage

Source: Armstrong et al., in press.

31

whether participants were bought-in to the innovation and if so, how did that develop and was this typical for them and the agency. 3. Data analysis Quantitative data were analyzed using SAS 9.3. Descriptive analyses were conducted to understand the sample and construct of interest (buy-in). Bivariate t-tests were conducted to view the relationship between staff and agency demographic characteristics and level of buyin at baseline. Multivariate logistic regression, using a cumulative logit regression model was used to view the relationship between buy-in and implementation status at 12 months, controlling for organization health at baseline. Cumulative logit models are used when the dependent variable is ordinal and consists of more than two levels. The analysis takes into account the ordering of the dependent variable. In this study, implementation raters had 8 levels to choose from, though only 4 levels were used by raters. Correlations among the independent variables of interest (3 change readiness scales, leadership, and job stress) were viewed in the model building process (Table 2). Two scales—change management and leadership—were highly correlated (above .75) and change management was removed from the model. Focus groups and interviews were recorded by a note-taker using a laptop computer. Notetakers were instructed to record dialogue verbatim. Data were analyzed using ATLAS 6.0. A three-stage process was used to code the data using a general inductive analysis approach, which combines inductive and deductive analytic methods (Thomas, 2006). The analyses are guided by the evaluation objectives, by providing a focus or relevant domain, but there are not a set of expectations about specific findings (Thomas, 2006). Coding proceeded as follows. First, one interview and one focus group were selected and each coded independently by two raters (“double-coded”). The goal was to generate a first set of upper-level codes based on the evaluation objectives and grounded in the data. The raters then met to discuss and agree on the codes and text segments. Raters were master's level research staff familiar with the project. Next, a third rater (the lead researcher) coded the same interview and focus group, using the upper-level codes, but adding a more deductive approach informed by the literature and ideas about the buy-in construct. New codes were developed and all three coders met at the end of this stage to finalize code definitions. Coding proceeded by the original two coders with meetings and regular communication about coded text and clarification. 4. Results 4.1. Survey results Table 3 presents the results of descriptive and bivariate analysis of staff characteristics and buy-in. The overall mean score on the buy-in measure was 4.23 (SD = 0.88), with a range of 1.50 to 6.00. As shown, results show that male staff report significantly higher levels of buy-in compared with female staff (M = 4.46 and 4.19, respectively; Table 2 Summary of correlations between scores used in multivariate analyses. Measure

1

2

3

4

5

6

7

1. Agency size 2. Buy-in 3. Change management 4. Learning organization 5. Response to change 6. Leadership 7. Job stress

– −.11 −.21 −.15 .07 −.20 .28

−.11 – .30 .22 −.19 .35 −.11

−.21 .30 – .51 −.31 .77 −.41

−.15 .22 .51 – −.21 .49 −.28

.07 −.19 −.31 −.21 – −.23 .26

−.20 .35 .77 .49 −.23 – −.31

.28 −.11 −.41 −.28 .26 −.31 –

32

J.S. McCrae et al. / Children and Youth Services Review 37 (2014) 28–35

Table 3 Bivariate analysis of the relationship between staff characteristics and buy-in at baseline. Characteristic Gender Male Female

Total % (n = 568)

Buy-in Mean score (SD)

14.6

4.46⁎ (0.8) 4.19 (0.9)

85.4

Race/Ethnicity African American

6.9

White

76.6

Multipe race

4.0

Missing

12.5

Ethnicity Hispanic or Latino

22.0

Not Hispanic or Latino

78.0

Agency tenure: b2 years

17.1

2 to 5 years

23.2

6 to 10 years

18.8

11 to 15 years

29.8

16 years or more

11.1

Agency size Small

4.9

Medium

10.6

Large

84.6

Position Senior management

7.5

Supervisor

14.0

Caseworker or aide

69.5

Other

9.0

4.24 (1.0) 4.25 (0.9) 3.98 (0.9) 4.23 (0.9) 4.33 (0.8) 4.20 (0.9) 4.07 (0.8) 4.22 (0.9) 4.29 (0.8) 4.18 (1.0) 4.52⁎⁎ (0.8) 4.86⁎⁎⁎ (0.6) 4.16 (0.9) 4.21 (0.9) 5.05⁎⁎⁎ (0.6) 4.62⁎⁎⁎ (0.7) 4.06 (0.9) 4.20 (1.1)

a

Direct care staff and supervisor function Intake or investigation

23.8

Ongoing and core services

34.6

Out-of-home care and adoption

15.1

Adolescent and youth

15.4

Multiple or other

11.1

Total

4.22 (0.8) 4.22 (0.8) 4.18 0.8) 4.02 (0.8) 4.54⁎ (0.8)

100.0

caseworkers and casework aides (M = 4.06; p b .001), and other staff (M = 4.20; p b .05). Further analyses using chi-square statistics revealed that senior management staff typically have more tenure (significantly correlated), but that the majority of senior management staff in this agency are female (65%), indicating that there is an independent relationship between being male and reporting higher levels of buy-in for the innovation. Among direct care staff (caseworkers and supervisors), those reporting their unit or function as “multiple or other” showed significantly higher buy-in (M = 4.54) compared with all other units (p b .05). Further analysis showed that staff in this category tended to be generalists (working across functions), which tends to occur in smaller agencies, or quality improvement and human resources staff. A multivariate model was designed to examine the relationship between buy-in and 12-month implementation status, controlling for four characteristics of organizational health at baseline (learning organization, response to change, leadership, and job stress). A third dimension of change readiness—change management—was highly correlated with leadership (.77), and was dropped from the model. Table 4 presents the results of cumulative logistic regression modeling. As shown, job stress and agency size showed significant relationship with 12-month implementation status. For each unit decrease in job stress reported by staff, the likelihood of being in a full implementation agency (the highest level) increased by 1.25 (p b .05). Small and medium-size agencies are more likely to be rated fully implemented compared with large agencies (p b .01 and p b .001, respectively).

4.2. Qualitative results County agency leaders and managers who volunteered to be the first Cohort of pilot sites voiced initial enthusiasm and excitement about the opportunity to improve practice to be more solution-focused and strengths-based. An added motivation to join the first cohort was the opportunity to help shape and guide the initiative to better fit local agency needs. These leaders indicated that they had weighed the relative advantage of the innovation against the time, effort, and resources of planning and implementation. Studies show that innovations that have clear, unambiguous advantages are more likely to be adopted (Greenhalgh et al., 2004), and leaders clearly focused perceived advantages in the areas of being able to shape the model. Supervisors and mid-level managers tended to center on the creativity afforded by joining the innovation, as indicated by this quote from a supervisor. “Being one of the first counties to participate in the initiative lends itself to the idea that the sky’s the limit.” The motivation of county leaders to volunteer to be the first cohort was also motivated in part by distrust of the state child welfare agency leaders to develop an initiative that would fit the needs of the their

a

Among supervisors and caseworkers only (n = 396). ⁎ p b .05. ⁎⁎ p b .01. ⁎⁎⁎ p b .001.

p b .05). Staff with 16 or more years of tenure also reported greater buyin compared with all other tenure-levels (p b .01 to p b .05 depending on the comparison). Staff working in small agencies reported significantly higher buy-in (M = 4.86) compared with staff in medium (M = 4.16; p b .01) and large-size agencies (M = 4.21; p b .001). Regarding position in the agency, senior management report significantly higher buy-in (M = 5.05) compared with all other staff categories (p b. 001). Supervisors also report significantly higher buy-in (M = 4.62) compared with

Table 4 Results of cumulative logit predicting 12-month implementation stage by baseline characteristics (n = 453). Characteristic (reference group)

OR

β

SE (β)

Agency size: small (all others) Agency size: medium (all others) Buy-in score Learning organization score Response to change Leadership score Job stress

.28⁎⁎ .07⁎⁎⁎

−.64 −1.32 −.17 .00 −.04 −.04 .22

.23 .22 .11 .12 .18 .09 .10

⁎ p b .05. ⁎⁎ p b .01. ⁎⁎⁎ p b. 001.

.85 1.00 .97 1.04 1.25⁎

J.S. McCrae et al. / Children and Youth Services Review 37 (2014) 28–35

33

unique county. It was important that they have a voice in the practice model values, as well as to help guide the implementation plan.

clear how the process could improve practice and outcomes for families. One supervisor explained:

“If we do it first, then we get to give input. So we don’t have to fit into a model that doesn’t fit us. We were wanting to influence the model before it was firmed up.”

“I didn’t share a whole lot with my team because I was not sure what this was going to look like here. They have enough to do, just another thing that wouldn’t make a lot of sense to them. It didn’t click with me until we started working on removal prevention. But, ‘It’s nice to see something come out of it’; A person can be on a committee for years and nothing comes of it.”

After numerous interventions throughout the years that were implemented using a “top down” approach from the state, what appealed to nearly all participants was the adaptability of the innovation, and the ability to refine and modify it to suit the needs of their county. Rogers described this concept as reinvention (1995) and spoke of the need for initiatives to have fuzzy boundaries, a hard core, and a soft periphery in order to maintain the balance between fidelity and fit. Despite their eagerness to lead the process and institute best practices, respondents also expressed skepticism that the initiative would actually lead to real and positive change for children and families. There was concern about how intensive it would be on workers and their workload, and some were worried that their county was already involved in so many other initiatives—how would they carve out time to make this succeed? Participants were unclear about state and county roles, and this made them nervous during a time when there was talk of changing to a state-administered system, as one participant expressed: “That was a difficult time, some people were pushing for us to be a state agency as opposed to the counties. I was skeptical about the roles of the state/counties and whether they’d be balanced. Seemed kind of propaganda-ish—like ‘let’s get on board with this.’ I don’t think we were really committed to it at first.” A number of participants, especially those who have worked in the child welfare field for a long time, were simply skeptical about yet another state “pet project.” There were questions about whether it would last and whether the state division would continue to support and implement the practice changes suggested by counties. One participant explained: “Most of my staff have been here 20–25 years. This isn’t the first ‘let’s do something as a state’ initiative. Sometimes there’s a lot of energy up front, then it loses steam, and it’s hard for staff to overcome. If you’ve been here for a long time, ‘it’s just another program with a different name.’” Concerning the ability of upper-level management to gain buy-in among multiple staff levels involved discussion about inclusivity— that designing the intervention components had to include frontline workers, and that communication about this should be transparent and planful. What is the staff selection process, how will it change over time, and will staff responsibilities be adapted to accommodate their participation in project planning activities? This reflects staff selection and facilitative administration implementation drivers. Front-line staff expressed that planned interventions need to emphasize child and family outcomes, and this should be reflected in the messaging, as reflected in this comment by child welfare investigation staff: “Commitment from workers will only happen if workers understand the positive impacts on their jobs and their daily work with families.” Initial skepticism was augmented by uncertainty and lack of clarity about the initiative during the exploration and design stages. The flip side of the coin of having work teams of staff involved in the planning stages, was that the initiative design was perceived as unclear and lacked definition in the beginning. Once agency-level implementation plans were developed and initial implementation began, it became

Seeing implementation happen in their own agencies was what prompted most respondents to fully buy-in to the initiative and understand how it could help the families and children they serve. Supervisors were noted as a key influence on caseworker attitudes and buy-in. “If my supervisor has buy-in and a good knowledge of the change and communicates it to us as workers, that’s a really important bridge between when the change gets decided and when it gets implemented.”

5. Discussion This state child welfare agency undertook an extensive organizational change effort to develop and implement a practice model over a three year period that included Continuous Quality Improvement (CQI), peer-to-peer learning, and developing a compendium of best practices across the state. As previously discussed, this effort was guided by the NIRN framework. Through the lens of understanding the need for attention to readiness, the project team focused on engaging all levels of the system; individuals on all levels of the organization, community stakeholders, and the organization. Meetings were held throughout the state at the county level to gain consensus on the new ideas, behaviors, and partnerships necessary to achieve the goal of developing a statewide practice model. Findings from this study indicate that this strategy was successful. High levels of buy-in were found across the organization. Senior management and supervisors had higher buy-in than workers; but, in the qualitative analysis, workers indicated that if their supervisors were invested and knowledgeable about the initiative, they would be more open to the change effort. This finding supports the diffusion of innovation theory (Rogers, 1995). Supervisors have the most influence over a worker; and, their interpersonal communication to the worker is key to achieving buy-in from the worker and ultimately implementation success. The importance of the supervisor role in the overall quality of performance and retention of workers is well-documented (Chen & Scannapieco, 2010; Kleinpeter, Pasztor, & Telles-Rogers, 2003; McCarthy, 2003; Scannapieco & Connell-Carrick, 2003; Scannapieco & ConnellCarrick, 2007). This study adds another element to the importance of the supervisor in successfully implementing system and organizational change. Through the diffusion of information to the worker, the supervisor is influencing the installation of system change. Focus needs to concentrate on the supervisor level when implementing system-wide practice or policy change. Supervisors should be part of the project team, involved in training and coaching, and the communication plan. Diffusion of information from the supervisor to the worker is an effective means of installing, implementing, and sustaining system change. Additionally, males and those with tenure of 16 years or more had higher buy-in. Further analyses revealed that senior management staff typically have more tenure, but that the majority of senior management staff in this agency are female (65%). This indicates that there is an independent relationship between being male and reporting higher levels of buy-in for the innovation. This could be because components of the innovation were perceived as more appealing by men compared with women, or because local CQI groups, the innovation's pillar, engaged men to a greater degree than women, resulting in power differentials

34

J.S. McCrae et al. / Children and Youth Services Review 37 (2014) 28–35

or perceptions of insignificance among some workers (Nygren, 2012). It is important to understand the characteristics of individuals who are less ready for change so that we can target interventions to those groups. Strategies need to focus on communicating the mission, vision, and goals of the system change to females, workers with less tenure, and workers from large agencies. More time may need to be spent in the larger counties, so that all levels of the organization may understand the innovation and its purpose. Communication strategies directed at woman may require a more interactive approach, which is more female centric, versus print and electronic communication. Future research around gender differences in readiness assessment will be an important area for exploration. Successful implementation requires a child welfare organization to progress through stages of implementation as outlined earlier (Fixsen et al., 2005). Advancement through the stages, from exploration, to design, to installation, and finally implementation is thought to be driven by many factors, one being the level of buy-in. Results in this study indicate that buy-in was not a driving factor in implementation, but rather manageable levels of job stress appeared to hasten the innovation. Workers in high-stress organizations may have been more resistant to adopting the innovation because of concerns about workload and their ability to follow through with their non-negotiable work tasks, such as seeing families. The innovation could also have been perceived as optional or “distal” by some workers, especially in the short time period of the study. Employees weigh the potential personal costs of change when making adoption decisions (Wright, Christensen, & Isett, 2013) and this study supports that there may have been costs associated with the change that outweighed participation and influenced progress. In another study of nurses involved in organizational change, the change process resulted in increased administrative stress, which later influenced levels of nursing-related stress and job satisfaction (Teo, Pick, Newton, Yeung, & Chang, 2013). However, effective coping strategies were found to mediate the relationship between job stress and job satisfaction. Nurses who used problem-focused coping such as asking for advice from co-workers reported reduced stress and higher levels of job satisfaction. In child welfare agencies, recognizing the administrative stress that is associated with change and supporting workers' positive coping during organizational change could influence implementation outcomes. Participatory decision making, open communication and clear information are also consistent themes in motivating workers and adopting change in public service organizations (Teo et al., 2013; Wright et al., 2013). These are goals that may be harder to achieve in larger organizations. In the current study, using a phased implementation approach may have been more successful in smaller, less stressed counties. Although the larger counties were not progressing as quickly as the smaller counties, they were moving through the stages. Larger counties experienced more complex, far reaching challenges than smaller or medium counties. Strategies addressing adaptive challenges in these larger organizations need to be identified throughout the process and adjustments made to the implementation plan.

6. Limitations The study sites are agencies that stepped forward first to participate in the large-scale change effort, which may have influenced the results. Later analysis of baseline characteristics of the second agency cohort showed higher levels of buy-in and change readiness among these agencies, and lower levels of job stress compared with first-cohort agencies (Butler Institute, 2013). This could mean that the innovation has “spread”, and that there is greater clarity and communication about the innovation and implementation steps in later years of the initiative. Data concerning the implementation status of second-cohort agencies were not collected due to the lack of available follow-up time in relation to the project ending.

Current results are useful in the first induction of large-scale change that occurs through phased implementation. 6.1. Missing data Nearly 60% of staff in the participating agencies completed the online survey, but had all data been available, the results may be different. Completion rates were highest in small and medium-size agencies, indicating that patterns related to the study findings in larger agencies may not be fully representative. Finally, the buy-in measure is specific to details of the particular change effort, which is recommended, but may not reflect all aspects of support or opinions about the innovation. There are no established measures of buy-in to address large-scale systems change, with the closest approximation being Aaron's Evidence-based Practice Attitude Scale (EBPAS; Aarons, 2004), but this measure was not appropriate given that the innovation implemented was not an evidence-based practice and was more formative in nature. 7. Conclusions The field of implementation science in child welfare is emerging, and gradually, should evolve to having well-tested models of change that reduce dynamics that may contribute to negative work environments and staff turnover. This study demonstrates the need to address stress among staff as a key barrier to successfully implementing change. Stress levels in this study did not increase over time, which could be perceived as positive given the extent of organizational change occurring, but nonetheless acted as challenging to implementation and should be included in future work around organizational and practice change in child welfare. References Aarons, G. A. (2004). Mental health provider attitudes toward adoption of evidence-based practice: The Evidence-based Practice Attitude Scale (EBPAS). Mental Health Services Research, 6, 61–74. Aarons, G. A., & Palinkas, L. A. (2007). Implementation of evidence-based practice in child welfare: Service provider perspectives. Administration and Policy in Mental Health, 34, 411–419. Armstrong, M. I., McCrae, J. S., Graef, M. I., Richards, T., Lambert, D., & Bright, C. L. (2013). Development and initial findings of an implementation process measure for child welfare system change. Journal of Public Child Welfare (in press). Barbee, A. P., Christensen, D., Angle, B., Wandersman, A., & Cahn, K. (2011). Successful adoption and implementation of a comprehensive casework practice model in a public child welfare agency: Application of the Getting to Outcomes (GTO) model. Children and Youth Services Review, 33, 622–633. Bouckenooghe, D. (2010). Positioning change recipients’ attitudes toward change in the organizational change literature. Journal of Applied Behavioral Science, 46, 500–531. Butler Institute for Families (2012). Comprehensive organizational health assessment. [Online survey]. Unpublished instrument. Denver, CO: University of Denver, Graduate School of Social Work, Butler Institute for Families. Chen, S. U., & Scannapieco, M. (2010). The influence of job satisfaction on child welfare worker’s desire to stay: An examination of the interactions effect of self-efficacy and supportive supervision. Children and Youth Services Review, 32, 482–486. Creswell, J. W., & Zhang, W. (2009). The application of mixed methods designs to trauma research. Journal of Traumatic Stress, 22, 612–621. Fixsen, D. L., Naoom, S. F., Blasé, K. A., Friedman, R. M., & Wallace, F. (2005). Implementation research: A synthesis of the literature. Tampa, FL: University of South Florida, Louis de la Parte Florida Mental Health Institute, The National Implementation Research Network (FMHI Publication #231). Frambach, R., & Schillewaert, N. (2002). Organizational innovation adoption: A multi-level framework of antecedents and opportunities for future research. Journal of Business Research, 55, 163–176. Glisson, C., Green, P., & Williams, N. J. (2012). Assessing the Organizational Social Context (OSC) of child welfare systems: Implications for research and practice. Child Abuse & Neglect, 36, 621–632. Greenhalgh, T., Robert, G., Macfarlane, F., Bate, P., & Kyriakidou, O. (2004). Diffusion of innovations in service organizations: Systematic review and recommendations. The Milbank Quarterly, 82, 581–629. Groves, K. (2005). Linking leadership skills, follower attitudes, and contextual variables via an integrated model of charismatic leadership. Journal of Management, 31, 255–277.

J.S. McCrae et al. / Children and Youth Services Review 37 (2014) 28–35 Kleinpeter, C. B., Pasztor, E. M., & Telles-Rogers, T. (2003). The impact of training on worker performance and retention: Perceptions of child welfare supervisors. Professional Development: The International Journal of Continuing Social Work Education, 6, 39–49. Lehman, W. E. K., Greener, J. M., & Simpson, D.D. (2002). Assessing organizational readiness for change. Journal of Substance Abuse Treatment, 22, 197–209. Lewin, K. (1951). Field theory in social science: Selected theoretical papers. New York: Harper. McCarthy, M. L. (2003). The relationship between supervision and casework retention in county-based child welfare systems. (Doctoral dissertation, Memorial University of Newfoundland, 2003). Dissertation Abstracts International, 65, 1119. National Child Welfare Resource Center for Organizational Improvement (2008). An introduction to the practice model framework: A working document series. (Available: http://tatis.muskie.usm.maine.edu/pubs/pubdetailWtemp.asp?PUB_ID=B060067) Nygren, K. G. (2012). Narratives of ICT and organizational change in public administration. Gender, Work and Organization, 19, 615–630. Powell Davies, G., Williams, A.M., Larsen, K., Perkins, D., Roland, M., & Harris, M. F. (2012). Coordinating primary health care: An analysis of the outcomes of a systematic review. Medical Journal of Australia, 188, S65–S68. Proctor, E., Silmere, H., Raghavan, R., Hovmand, P., Aarons, G., Bunger, A., et al. (2010). Outcomes for implementation research: Conceptual distinctions, measurement challenges, and research agenda. Administration and Policy in Mental Health, 38, 65–76. Proctor, E., Silmere, H., Raghavan, R., Hovmand, P., Aarons, G., Bunger, A., et al. (2011). Outcomes for implementation research: Conceptual distinctions,

35

measurement challenges, and research agenda. Administration and Policy in Mental Health, 38, 65–76. Rogers, E. M. (1995). Diffusion of innovations. New York: Free Press. Scannapieco, M., & Connell-Carrick, K. (2003). Do collaborations with schools of social work make a difference for the field of child welfare: Practice, retention and curriculum. Journal of Human Behavior in the Social Environment, 7, 35–51. Scannapieco, M., & Connell-Carrick, K. (2007). Child Welfare Workforce: The state of the workforce and strategies to improve retention. Child Welfare, 86, 31–52. Teo, S. T. T., Pick, D., Newton, C. J., Yeung, M. E., & Chang, E. (2013). Organisational change stressors and nursing job satisfaction: the mediating effect of coping strategies. Journal of Nursing Management, 21, 878–887. Thomas, D. R. (2006). A general inductive approach for analyzing qualitative evaluation data. American Journal of Evaluation, 27, 237–246. U. S. Department of Health and Human Services, Administration for Children and Families, & Children’s Bureau (2008). Spotlight on systems change in child welfare. Children’s Bureau Express, 9. Weiner, B. J. (2009). A theory of organizational readiness for change. Implementation Science, 4, 67–75. Weiner, B. J., Amick, H., & Lee, S. D. (2008). Review: Conceptualization and measurement of organizational readiness for change: A review of the literature in health services research and other fields. Medical Care Research and Review, 65, 379–436. Wright, B. E., Christensen, R. K., & Isett, K. R. (2013). Motivated to adapt? The role of public service motivation as employees face organizational change. Public Administration Review, 73, 738–747.