Changes in the service delivery model and ... - Wiley Online Library

IMHJ (Wiley)

LEFT

BATCH

top of AH A

R

T

I

C

L

E

CHANGES IN THE SERVICE DELIVERY MODEL AND HOME VISITORS’ JOB SATISFACTION AND TURNOVER IN AN EARLY HEAD START PROGRAM SUKHDEEP GILL, MARK T. GREENBERG, AND ARCEL VAZQUEZ

The Pennsylvania State University ABSTRACT: Study of the processes through which interventions achieve the intended outcomes is as important as the study of the intervention outcomes themselves. In this article, the process of implementing an Early Head Start program is described and related to changes in the home visitors’ job satisfaction and turnover. Initially, the biggest challenge for the implementing agency was maintaining undisrupted service delivery whereas the new administration team, when hired, confronted the task of rejuvenating the program. This led to an overall restructuring of the home-based model of service delivery. Results show that, as predicted, home visitors were least satisfied with their roles, work atmosphere, and job conditions during the program make-over phase. The staff turnover rate was the highest in the program’s history during this period (39%). However, these trends were reversed as the program stabilized due to the changes. These results are presented and discussed in light of the program implementation issues and changes to the service delivery model. RESUMEN: El estudio del proceso por medio del cual las intervenciones logran los esperados resultados

es tan importante como el estudio de los mismos resultados de la intervencioń. En este artıćulo, se describe el proceso de implementacioń de un programa “Early Head Start,” y el mismo esta´ relacionado con los cambios en las responsabilidades de quienes llevan a cabo las visitas a casa y la consecuente rotacioń en tales posiciones. Inicialmente, el mayor reto que enfrentaba la agencia encargada de la implementacioń fue el de mantener la prestacioń del servicio sin interrupcioń, en tanto que el nuevo equipo administrativo, cuando se contrataba uno, confrontaba la tarea de rejuvenecer el programa. Esto llevo´ a una completa restructuracioń del modelo de prestaciones de servicio a domicilio. Los resultados muestran que, tal como se predijo, los que visitaban las casas se sintieron menos satisfechos con el papel que tenıán, el ambiente de trabajo y las condiciones del puesto durante la fase de cambio del programa. El porcentaje ma´s alto de rotacioń de personal en la historia del programa ocurrio´ durante este perıódo (39%). Sin embargo, esta tendencia cambio´ tan pronto como el programa logro´ estabilidad debido a la restructuracioń del mismo. Estos resultados se presentan y discuten a la luz de los asuntos relacionados con la implementacioń del programa y los cambios al modelo de prestaciones de servicios a domicilio. RE´SUME´: L’e´tude des processus a` travers lesquels les interventions atteignent le but de´sire´ est aussi importante que l’e´tude des re´sultats de l’intervention eux-meˆmes. Dans cet article, le processus d’application d’un programme Early Head Start (le EHS est un programme de pre´vention et d’aide

This study was supported by a grant from the Dorothy Rider Pool Health Care Trust (Grant #490). Direct correspondence to: Sukhdeep Gill, Prevention Research Center, The Pennsylvania State University, S 109 Henderson, University Park, PA 16802; phone: 814-863-6344; fax: 814-865-2530; e-mail: [email protected]. INFANT MENTAL HEALTH JOURNAL, Vol. 23(1– 2), 182– 196 (2002) 䊚 2002 Michigan Association for Infant Mental Health

182

short standard base of drop

IMHJ (Wiley)

RIGHT

BATCH

EHS Program Evaluation

●

183

gouvernementale a` la petite enfance de´favoriseé aux Etats-Unis d’Ame´rique) est dećrit et lie´ aux changements ayant trait a` la satisfaction avec leur travail des employe´s charge´s de faire les visites a` domicile et a` la rotation du personnel. Initialement, le plus grand des de´fis pour l’agence chargeé de l’application du programme e´tait de maintenir un service non interrompu, alors que la nouvelle e´quipe administrative, une fois engageé, avait pour taˆche de re´geńe´rer le programme. Cela menait a` une restructuration geńe´rale du mode`le d’offre de service. Les re´sultats montrent que, comme pre´vu, les visiteurs de domiciles e´taient les moins satisfaits de leurs roˆles, de l’atmosphe`re de travail et des conditions de travail durant la phase de transformation du programme. Le taux de rotation du personnel fut le plus haut de l’histoire des programmes durant cette pe´riode (39%). Cependant, les tendances se sont renverseés au fur et a` mesure que le programme s’est stabilise´ apre`s les changements. Ces re´sultats sont pre´sente´s et discute´s a` la lumie`re des proble`mes pose´s par l’application du programme et des changements effectue´s au mode`le d’offre de service.

top of rh base of rh cap height base of text

ZUSAMMENFASSUNG: Studien u¨ber den Prozeß der Art der Intervention, die den beabsichtigen Effekt hat,

sind von genauso großer Bedeutung wie Studien u¨ber die Ergebnisse der Intervention selbst. Dieser Artikel beschreibt den Prozeß der Einfu¨hrung eines Fru¨hfo¨rderungsprogramms (EHS) und bezieht ihn auf die Zufriedenheit im Beruf und die Personalfluktuation der Hausbesucher. Urspru¨nglich war es fu¨r den Verein, der das Programm einfu¨hrte die gro¨sste Herausforderung die Betreuung ohne Unterbrechung anzubieten, wohingegen das neue Team, nach seiner Anstellung die Aufgabe hatte die Leistungen zu verju¨ngen. Dies fu¨hrte zu einer Restrukturierung der Art des Angebots im Bereich der Hausbesuche. Ergebnisse zeigen, dass, wie vorausgesagt, die Hausbesucher wa¨hrend der Umstellungsphase am wenigsten Zufriedenheit in bezug auf ihre Rolle, dem Betriebsklima und den Bedingungen der Arbeit hatten. Der Personalwechsel war wa¨hrend dieser Phase (39%) der ho¨chste in der Geschichte des Programms. Diese Trends wurden jedoch nach der Stabilisierung der Vera¨nderungen des Programms umgekehrt. Die Ergebnisse werden im Licht der Themen der Einfu¨hrung und der Vera¨nderung des Programms pra¨sentiert und diskutiert.

*

*

*

Evaluators have emphasized the role of process evaluation in ascertaining what is inside the “black box” of early preventive intervention programs. Inquiry into program processes provides the stakeholders with insight on the aspects of interventions that work and those that do not. Inquiry also helps to highlight what actually happens during the process of intervention implementation (McCoy & Reynolds, 1998). Process evaluations not only aid in interpreting program outcomes, but also serve an important purpose in identifying the key elements of successful interventions. By tracing the life course of intervention programs — the ways in

short standard

IMHJ (Wiley)

184

●

LEFT

BATCH

S. Gill, M.T. Greenberg, and A. Vazquez

which policies, services, structures, and program expectations change over time — studies of program implementation can provide much needed information regarding what programs actually do (Weiss, 1987), may offer insight regarding program effectiveness during periods of change, and provide guidance to others interested in adapting the program to their setting (Durlak, 1998). The purpose of this article is twofold. The first purpose is to document the events leading up to changes in the model of service delivery of an Early Head Start (EHS) program (1) to illustrate the nature of challenges experienced by programs during the early formative stages of program implementation, and (2) to provide information regarding the effective and problematic features of a home visitation model. The second purpose is (1) to explore the home visitation staff’s reactions to changes in the model of service delivery, and (2) to investigate the impact of changes in program implementation on home visitors’ job satisfaction and turnover. We begin with a brief description of the issues in program implementation and the home visitation model. We then summarize the history of the program and present results from interviews with home visitors and administrators that led to changes in the service delivery model during the first two years of its operation. An outline of the process of change and a brief description of the new model of service delivery are presented in the next section. We submit that such significant program changes are likely to have an impact on the home visitors and the families being served by the program. The long-term results of these changes remain to be seen. There may, however, be certain short-term consequences that act to moderate longterm program outcomes, and therefore need to be considered. In this article, we focus on the effect of program changes on home visitors’ levels of job satisfaction and turnover. These results and their implications are discussed in the final section.


PROGRAM IMPLEMENTATION

To be effective, a program needs to be stable, have strong leadership, and have clearly defined goals as well as efficient strategies to implement those goals. In addition, a system for monitoring program implementation and staff development, which helps create feedback loops to inform the program, is particularly important during the initial phases to ensure an effective service delivery system to facilitate achieving intended program outcomes (Durlak, 1998). Monitoring and documenting program implementation processes is essential (Holden & Reynolds, 1997; Ramey & Ramey, 1998; Scheirer, 1994) because programs are rarely implemented as planned (Chen, 1998; Powell, 1993; Weiss, 1987). Process evaluation, then, not only provides information on the internal dynamics and operations of the program, but also provides insight into unanticipated changes in the program that could very well have pronounced effects on program outcomes. Outcome evaluations that are mounted without any consideration of program processes make it difficult to interpret the results of the interventions. Lessons learned from process evaluations are not only important for the specific programs to which they pertain, but are of immense value for other similar programs. Sharing information about program implementation can help others avoid duplication of errors, thereby, saving time and resources. Dissemination of such information can provide insight into what is needed to implement a new program and what to expect during the initial phases of the program. It can also offer support for programs going through changes (McGraw et al., 1994), especially during the early stages of implementation. Efficient monitoring of program processes should be dynamic in nature. Collecting data on individual statuses at predetermined fixed time points does not always provide an adequate basis for studying change in longitudinal research (Bryk & Weisberg, 1977; Bryk & Rauden-

short standard

IMHJ (Wiley)

RIGHT

BATCH


●

185

bush, 1987). It is important to study programs when crucial events take place to understand fluctuations, disruptions, or changes in program implementation that could impact the effectiveness of the program eventually. It is equally important for programs to use the feedback information from process evaluations in a constructive and timely manner; doing so ensures that program changes are implemented early enough for the intervention to be effective. Implementation of such changes, however, is likely to result in periods of temporary distrubances in the homeostatic equilibrium that may have consequences for the staff and, in turn, the families being served by the program. Therefore, documentation of the changes in the implementation of a home visitation model of service delivery was one of the two goals of this evaluation study. The other was the exploration of the effects of these changes on the home visitors.


HOME VISITATION

The concept of home visitation to provide guidance and model proper ways of behavior to urban poor families (Boyer, 1978) has a long history (Olds, Gomby, & McCroskey, 2000). It has become a prominent model of service delivery in early preventive intervention programs with children and families during the past 3 decades (Gomby, Culross, & Behrman, 1999; Weiss, 1993; Weissbreg & Greenberg, 1998). Through teaching and role modeling, home visitors work with families to promote healthy development of children, to improve family functioning, and to connect families to social services and networks within their communities. Forming close, personal, and trusting relationships with families, over an extended period of time, is at the heart of the home visitation model (Kitzman, Yoos, Cole, Korfmacher, & Hanks, 1997; Olds, Kitzman, Cole, & Robinson, 1997; Robinson, Emde, & Korfmacher, 1997). Because the intervention takes place within the context of everyday family living, it allows the home visitor to be an active observer of family processes. The home visitor gains valuable insight into the physical and social conditions in which families live, the key players in family decision making, and their interaction patterns, family values and beliefs, and the availability of resources. Such information enables the home visitor to tailor the services to each family’s unique needs and goals. A home visitor is thus the key link in the provision of services to clients in an efficient and effective manner. In family-centered comprehensive interventions, home visitors develop individualized service delivery plans that are responsive to the unique constellation of age groups, parent – child – other interaction patterns, family makeup, and needs and goals of each client in their caseload without compromising the program theory and goals. Therefore, the professional training, knowledge, and skills that home visitors bring to a program, as well as their understanding of the service delivery system are likely to impact the quality of support and consultation they provide to the program families (Wasik, 1993; Weissberg & Greenberg, 1998). In addition, aspects of their work environment may affect their job satisfaction and staff turnover rate, which may subsequently impact the quality of service delivery. Although some attention has recently been focused on staffing issues in home visitation programs (Wasik, 1993), it has primarily been concerned with recruitment and training issues. Research on home visitors’ experiences at work is still sparse. We submit that in addition to professional background and on-the-job training, program implementation, ongoing support, and a feeling of program ownership contribute to the job satisfaction of the home visiting staff. Specifically, we hypothesized that changes in program implementation would have consequences for the home visitors’ job satisfaction. This would, in turn, impact the program by influencing the quality of service delivery directly or through staff turnover.

short standard

IMHJ (Wiley)

186

●

LEFT

BATCH


PROGRAM HISTORY


The first wave of 143 EHS programs, an initiative of the Administration on Children, Youth, and Families (ACYF) to enhance health and development of infants and young children, to strengthen families, and to promote connections and networks within communities through comprehensive service delivery to low-income families, was funded in 1995 (Department of Health and Human Services, 2001). During the subsequent years, several additional agencies successfully applied for funding to start EHS programs in their communities. These grants were approved for an initial period of five years, with possibilities of renewal based on evidence of effective service delivery. ACYF developed detailed guidelines for the implementation and monitoring of these programs to ensure compliance with EHS performance standards. Many agencies adopted program evaluations to fulfill these requirements and to collect sound and defensible evidence of program effectiveness. The EHS program described in this study was one of the projects that received Wave 2 funding in 1996. The first group of families was enrolled in August 1997. In October1998, the agency received additional funds from a local foundation in support of this initiative. Process and an outcome evaluations of the program were commissioned at this time. The program maintains an enrollment of 75 children. Home visitation is offered to pregnant women and families with children under the age of 3. In addition to receiving comprehensive services in the areas of child development, health, safety, parenting, parents’ life course development, and community linkages through weekly home visits, families have opportunities to participate in at least two group socializations per month. Also, parents are encouraged to serve on parent committees and the policy council that governs the EHS/HS program. This program serves a heterogeneous population; currently, the participants include approximately 32% Caucasian, 57% Latino, and 11% African American families. The original program delivery system was based on a model of specialization. That is, a team of two specialists — a “Family Specialist,” specializing in family needs, human services, and community networks, and an “Education Specialist,” specializing in child development and parenting issues — were assigned to each family. Education Specialists provided weekly visits of approximately two-hour duration to six to eight families, whereas Family Specialists provided visits every two weeks for about two hours to 10 – 13 families. A program director, program manager, and two supervisors managed the program. The administrative body and the Executive Director of the larger agency oversaw the EHS program.

PROGRAM CHANGES

Interviews with the administrative staff and the home visitors at Time 1 (October 1998) and Time 2 (October 1999) as well as the program documents provided information regarding program changes and the conditions leading up to changes in the model of service delivery. Thirteen home visitors (eight Education Specialists; five Family Specialists) and six administrators participated in the Time 1 interviews; 14 home visitors (seven Education Specialists; seven Family Specialists) and six administrators were interviewed at Time 2. Semistructured interview schedules were developed in consultation with the program administrators to ensure that the interview protocols were congruent with staffs’ job responsibilities, program goals, and objectives of the evaluation. The interviews lasted between 60 – 90 minutes. Copies of the interview protocols can be obtained from the first author. A chronological account of the major events in the life course of the program is presented in Table 1. Two types of changes occurred during the early stages of the program: (1) administrative changes, and (2) changes in the model of service delivery.

short standard

IMHJ (Wiley)

RIGHT

BATCH


●

187


Administrative Changes

Table 1 shows that, with the exception of one supervisor, there was a complete administrative staff turnover during the first two years of the program. As reported by a member of the administrative team, staff turnover at the administrative level led to a lack of leadership during the initial formative stages of program implementation (Staff Interviews, October 1998). Lack of direction during the first year of operation resulted in the absence of a clearly defined program agenda and implementation plan. These changes at the administrative level led to a feeling of instability and lack of clarity about roles among the home visitors. In December 1998, an interim team was pulled together from two other programs of the agency (Head Start and another home visitation program) to keep the EHS service delivery from being disrupted. This team managed the EHS program for a period of 5 – 6 months, and gradually turned the operations over to the newly hired EHS program director, program manager, and family specialist supervisor by mid-1999. Changes in the Model of Service Delivery

As the new administrative team examined the program operations, it focused attention on aspects of the program that required change. For example, one of the administrators noted that when she joined the program, “there were no written procedures or policies, no curriculum plans, and no system for record keeping as a result of which the direct services staff did not understand . . . what was expected of them” (Staff Interviews, October 1999). Under the new leadership, an agency-level internal assessment was completed. During the same time, the results from the October 1998 staff interviews were shared with the EHS staff as well as the agency representatives who had been involved in the administration of the program since the TABLE 1. Chronological History of the Early Head Start Program Year

Project Events

1996 1997

Early Head Start grant received Program Director hired First group of families enrolled Local funding awarded Independent evaluation funded Program Manager hired Director left Family Specialist Supervisor (FSS) hired Education Specialist Supervisor (ESS) hired Program Manager left Family Specialist Supervisor left Interim project management team assigned Project Manager hired Project Director hired New program team in place Family Specialist Supervisor hired Feedback on staff interviews shared with the program Internal project review completed Change in the service delivery model implemented New model of service delivery implemented Enrollment Family Specialist hired

1998

1999

2000

short standard

IMHJ (Wiley)

188

●

LEFT

BATCH


inception of the evaluation. Together, all these factors led to changes in the service delivery model that are described later in the article. Following is a summary of findings from the home visitor interviews regarding the initial program delivery model.

Problems with the existing model of service delivery. Interviews with the home visitors identified a number of barriers arising from the existing model in which two specialists worked with each family as a team. Barriers stemmed from differences in role perceptions and caseloads. In addition, differences in the successful completion of home visits, lack of clarity regarding the essential components of a home visit, and lack of information about the principles and guidelines of the service delivery model also contributed to the problems. Education Specialists and Family Specialists were expected to work with each family as a team, but had separate roles and responsibilities. The Education Specialists focused on the care and development of the child by strengthening parenting skills. They worked with the primary caregiver (primarily the mother, and father where present) and the target child. Their responsibilities included providing information to develop parents’ knowledge and skills in parenting, serving as role models, and working directly with children to promote their optimal development. The Family Specialists, on the other hand, focused on family goals. They were responsible for working with family members to develop Individualized Family Partnership Agreements (IFPA), and time lines for working on the goals defined by the family, and connecting the families with other resources and agencies in the community to help parents achieve their goals. Although these roles seemed clear on paper, the home visitors found them to be unclear in practice. One point of confusion, for example, had to do with determining whose role it was to help parents get their child immunized. Some believed that it was an area directly related to child development, and thus within the Education Specialist’s purview. Others thought it was a part of networking with other agencies in the community, and hence a job for the Family Specialist. Also, there were no guidelines for service delivery to pregnant women. As a consequence, home visitors were unclear about what their respective duties were when working with pregnant women. Because there was no child to work directly with, the Education Specialists did not see a role in it for themselves. The Family Specialists did not know much about dealing with the prenatal period and were not sure that it was their duty either. Consequently, the program did not enroll any pregnant women during the first year of service delivery. Another example of blurred roles and responsibilities had to do with a mismatch of professional background and role assignment. For example, one home visitor who reported having applied for and been interviewed for the Family Specialist position, found herself in the Education Specialist’s role. Some of her Family Specialist partners appreciated her shouldering additional responsibilities involving networking, but others were frustrated with her frequent crossing of boundaries. The existing model of service delivery also caused problems in the completion of home visits as reported by program administrators and home visitors. First, some of the home visitors were unable to complete the requisite number of home visits or failed to engage the family. These home visitors reported an inability to schedule meetings despite their repeated efforts to get in touch with the family. However, the administrators noted that their teammate who was working with the same family faced no such problems. It seemed that some families preferred working with one specialist more than the other. These families met with their preferred specialist on a regular basis and devoted their time and energy to working with him/her while they declined meetings with the other specialist. Second, there was confusion as to whether both partners could go on a home visit together


short standard

IMHJ (Wiley)

RIGHT

BATCH


●

189

and count one two-hour visit towards each home visitor’s visitation requirement. Finally, it was unclear what constituted a “complete” home visit. The program administrators reported that the home visitors used the concept of home visit very loosely, counting any contact in or outside the home, meeting at a social place, or taking a family to a doctor’s appointment, as a home visit. As a result, there was great variation in what was accomplished during these visits as far as different components of service delivery were concerned. The accumulation of perceived difficulty with the current model of service delivery was reflected in the communication between the home visitation team members. Many found their efforts to communicate and keep in touch with their teammates frustrating. They reported problems with coordination and difficulty in following-up on decisions made during their team meetings. Several home visitors reported that although their aim was to hold a team meeting once each week, their schedules did not always allow them to meet. However, they maintained contact by talking to each other in the passing or leaving phone messages. Another communication problem faced by home visitors had to do with differences in caseloads between Education Specialists and Family Specialists. The Family Specialists worked with as many as three to four different partners, making it more difficult for them to communicate with their teammates effectively and in a timely fashion. Conversely, those who worked well with each other visited their families together. They coordinated their tasks so that while the Family Specialist worked with the parent, the Education Specialist provided activities to the child. The home visitors who went on home visits together reported being supportive of each other, finding it safer to visit families during the evening hours, and having a very good rapport with their teammate. The program administrators reported that having two home visitors working with each family made it easier to continue undisrupted service delivery in the event of staff turnover. It also facilitated transfer of cases to newly hired home visitors who received detailed information on the characteristics and needs of families in their caseload as well as support in forming a positive relationship with them. These features were seen as significant merits of the team approach of home visitation. The process of change. As a result of the above feedback, coupled with an internal agency review indicating a low number of completed home visits, high family turnover rates, and role confusion in service delivery, the program administration decided to revise the service delivery model in November 1999. Although these changes were comprehensive and included changes in the structure and functions of the administrators, home visitors’ roles, as well as the process of service delivery, in this article we only address changes in the home visitation model. Once the program administrators had concluded that the existing model was not working effectively, the decision to bring about changes was made quickly. It was decided to dissolve all of the existing Education Specialist and the Family Specialist positions and advertise a new position that would combine these two roles. The program gave the current specialists the first option to reapply. The applications were screened and all, but two, of the Education Specialists and Family Specialists who were interested in working within the revised model of service delivery were rehired. The process of rehiring the existing staff was completed over an 8-day period. The program administrators felt that it would be the most efficient way to make the transition to the new model of service delivery, and would cause minimal service disruption to the program families. It may be noted that the program families were also involved in this process in several ways. First, a group of parents who served on the Policy Council (responsible for approving all the EHS program administration decisions) knew and agreed to the proposed changes. More importantly, keeping all the program families informed of the changes was made a priority. The program administrators sent three letters to the program families at different points in this


short standard

IMHJ (Wiley)

190

●

LEFT

BATCH


process. The first letter told them about the plans, the second one announced their new home visitor, and the third invited them to share their concerns about the changes, if any. In addition, a staff member from the program made phone calls to each program family and talked to them about the changes and addressed any concerns they had about how these changes would impact them. The administrators completed the hiring process by the first week of December. Following the hiring process, a month was taken to complete the reorientation and reassignment of cases, as needed. The new home visitors, now called “Child Development Partners,” went through a 2-week intensive reorientation before they picked up home visitation in full force on January 3, 2000. According to the program administrators, efforts were made to ensure that the case assignments were kept the same for the rehired staff to make the transition most seamless for the program families.


The new model. The new model of service delivery differed from the old model in two ways. First, it was based on having a single home visitor be responsible for all the clients in their caseload. The home visitor, called the “Child Development Partner-Home,” would provide services in all the program content areas during two-hour weekly visits, and would have a lower caseload of six to eight families compared to the previous caseload of 8 – 13 families (depending upon their Education or Family Specialist roles). According to the official communication regarding these changes, the home visitors were responsible for providing comprehensive services related to early childhood education, health and immunization, safety, nutrition, family development (including connecting families with social networks), and life skills management. Second, four Child Development Partners were designated as “Center Partners.” They would take the lead in encouraging parent participation in the program and devote most of their time organizing and conducting parent – child activities, parent meetings and other social events for the program participants. A new position of “Enrollment Family Partner” was created to focus attention on enrollment, community awareness of the program, and program advocacy. Before these changes, the home visitors carried most of these responsibilities. The program administrators believed that the new model would help to avoid role confusion, promote the home visitor’s sense of ownership and responsibility for comprehensive service delivery, improve home visitor – family relationship and bonding, and increase chances for group participation for the program families. Moreover, the program families would not have to find time to schedule visits with two home visitors. Altogether, this would be a significant step towards effective program implementation.

HOME VISITORS’ PERCEPTIONS OF PROGRAM CHANGES

The above-mentioned changes in the model of service delivery were a crucial event in program development that needed to be recorded even though this was not a part of the initial timetable for planned staff interviews. The program administrators, however, agreed that it would be very useful to capture the home visitors’ perceptions regarding the changes, especially, because they did not participate in the planning for these changes. These interviews would inform the program whether the home visitors agreed that these changes were desirable for efficient service delivery. Also, it would help to explore the impact of these changes on the home visitors themselves. Thus, a short open-ended survey was mailed to all the home visitors in February 2000 (Time 3) to record their perceptions of changes in service delivery.

short standard

IMHJ (Wiley)

RIGHT

BATCH


●

191

Perceptions Regarding Changes in the Service Delivery Model


As stated earlier, nine of the 17 home visitors had been with the programs through these changes. In response to a question regarding recent changes in the model of service delivery, six home visitors said that they liked the changes, one disliked them, one was unsure, and one did not respond. The home visitors agreed that these changes would result in clearer roles and lower caseloads for them, especially because the new administrative structure would take away the responsibilities of recruitment and initial paperwork from the home visitors. As for the service delivery, they believed that these changes would lead to better organization and improved services to the program families as indicated by a higher number of completed home visits, better partnership, and a closer relationship with the program families. Having “Center Partners” to focus on the opportunities for group socializations would lead to better parent involvement in the program. On the down side, they noted that it might be difficult to provide the full range of services within the stipulated time for a home visit. Also, they were concerned that they might not be able to connect families with as many social networks as was possible when one home visitor paid exclusive attention to that aspect. Impact of Program Changes on the Home Visitors

After a brief description of the data collection methodology and respondent characteristics, data are presented that address two research questions: (1) were there any changes in the home visitors’ job satisfaction during the period of program changes? and (2) did the staff turnover rates fluctuate in response to the program changes? Data collection. At the end of the qualitative interviews (see section on program changes), the interviewer asked home visitors to rate their overall job satisfaction on a 10-item scale. For example, “On a scale of 1 – 5, how satisfied are you with your workload?” A visual bar-card representing the five categories from “least satisfied” to “most satisfied” was presented to encourage uniform responses. The job satisfaction scale was divided into three thematic areas for more detailed analyses. These subscales included items on satisfaction with interpersonal factors (four items), roles and responsibilities (three items), and salary, benefits, and promotion (three items). Job satisfaction data were collected at four points: October 1998 (Time 1), October 1999 (Time 2; shortly before changes in the service delivery model), February 2000 (Time 3; shortly after changes in the service delivery model), and October 2000 (Time 4) at the time of the annual staff interviews. It may be noted that the program supervisors/director did not have access to the interview or survey data. Home visitor characteristics. During the period under investigation (October 1998 – December 2000), the program hired a total number of 28 home visitors. At the beginning of the process evaluation (October 1998), there were 13 home visitors (eight Education Specialists; five Family Specialists); four were hired within the next 6 months. As stated earlier, only 9 of these 17 home visitors stayed with the program throughout the process of program changes. Four of the home visitor positions turned over two or more times during this time. Sixty-five percent of all home visitors had worked for this EHS program for more than a year. Approximately half of the home visitors (53%) came into the program with a degree in Early Childhood Education or Human Development, whereas the rest had degrees/certificates in Nursing, Dietetics, Occupational Therapy, Social Work, or Criminal Justice. Their work

short standard

IMHJ (Wiley)

192

●

LEFT

BATCH


experience was as diverse as their professional training. One-third had been teachers in early childhood programs (35%), another one-third had worked with older children, while the remaining one-third had worked in public health, agriculture, industry, or with mentally retarded adults. None had a background or experience in home visitation.


Job satisfaction. Mean differences in the home visitors’ job satisfaction are presented in Figure 1. There were significant differences between the means for overall job satisfaction among Time 1, Time 2, Time 3, and Time 4, F(3,47) ⫽ 4.71, p⬍.05. Although the overall job satisfaction scores were the lowest at Time 3 (just after the program changes), these were not statistically different from the two previous time points. Further, multiple comparisons using Tukey’s post hoc test indicated that home visitors’ job satisfaction at Time 4 was significantly higher than at any of the previous time points. It may be noted that the mean scores showed a slight upward trend when the agency hired the EHS administration team in 1999 (Figure 1) that provided guidance and direction to the program. Further analyses showed that two of the three subscales of job satisfaction — satisfaction with the interpersonal subscale (communication, support, and work atmosphere; M ⫽ 3.25, 3.45, 2.81, 3.83 for Times 1, 2, 3, and 4, respectively) and salary, benefits and promotion subscale (M ⫽ 2.49, 2.93, 2.59, 3.22 for Times 1, 2, 3, and 4, respectively) — followed the same trend as the overall job satisfaction (see Figure 1). However, home visitors’ satisfaction on the latter subscale was the lowest at all the four times under investigation. Item-level analysis indicated that they were less satisfied with their salary and promotion opportunities than with their benefits, however. In fact, they mentioned that the excellent job benefits made other jobs less attractive even if the pay was better. Home visitors’ satisfaction on the roles and responsibilities subscale declined from Time 1 through Time 3 (M ⫽ 3.67, 3.43, 2.85, respectively) but reversed at Time 4, when they reported being most satisfied (M ⫽ 3.77), F(3,47) ⫽ 3.29, p ⬍ .05. Also, Tukey’s post hoc comparisons indicated that ratings of satisfaction on these subscales differed significantly from Time 3 to Time 4. Staff turnover. We traced staff turnover rates from October 1998 to December 2000. The results showed that there were two periods of high staff turnover since October 1998 (see Figure 2). Twenty-five percent of the home visitors left the program during the first 3-month period; this was at a time when two of the three members of the administration left the program. During the period when the interim management team took over the day-to-day program operations in December 1999 (see Table 1) and the new EHS program management team took charge, the staff turnover rates remained quite low. The highest staff turnover (39%), as expected, occurred during the changes in the service delivery model (see Figure 2). Out of a total number of eight home visitors who left the program during this period, one left prior to these changes and two left within 4 months of these changes, one did not apply for the new position, two applied but were not hired, one went back to college, and one left due to family circum-

FIGURE 1.

Job satisfaction during different phases of program implementation.

short standard

IMHJ (Wiley)

RIGHT

BATCH


●

193


FIGURE 2.

Staff turnover by 3-month period intervals.

stances. These analyses show that the staff turnover did not occur solely as a consequence of the program changes. It is worthwhile to note that in the period after the program changes, the staff turnover has been very low.

LIMITATIONS

One of the key limitations of the present study is its sample size. Because sample size has a direct influence on the power to detect group differences, this sample size has most likely led to an underestimation of the findings. Despite a small sample, we found significant differences in levels of job satisfaction at Time 4. Also, only those home visitors who were rehired and thus stayed with the program filled out the survey on job satisfaction at Time 3. This could have introduced a selection bias. However, because only two of the home visitors lost their jobs, the staff reports are likely to be fairly representative of the entire staff.

DISCUSSION

The Early Head Start program initiative allows the program grantees to choose a model of service delivery to achieve the intended outcomes for children, families, staff, and community development. Therefore, agencies offering EHS programs have adopted models that are homebased, center-based, or a combination of the two (Kisker, Love, & Raikes, 1999; Weissberg & Greenberg, 1998). The EHS program discussed in this article adopted a team approach in which two home visitors provided home-based services to program participant; later, the service delivery model was changed in an effort to improve the quality of services to its clients. In this article, we have documented the challenges faced by the program during this early phase of program implementation. We found that the staff turnover at the administration and home visitation levels was one of the most critical issues encountered by the program during the period under report. Two periods of high turnover rates overlapped the unstable periods in program implementation. The first period coincided with the initial start-up phase when the program lacked leadership, which resulted in role ambiguities and a lack of structure. Frequent changes in the upper level of administration left much to the interpretation of the individuals dealing with the day-to-day operations. During this phase, the highest priority of the program was to maintain the stipulated enrollment and services to the enrolled families. In the absence of leadership, the home visitors made their own decisions, took additional responsibilities, and counted upon their teammates to provide support as well as guidance in case of problems. Once the new administration team came on board, they started to look at the program more holistically in an attempt to bring all program components up to speed. In so doing, the roles and responsibilities of the home visitors changed drastically in a short period of time. The program administrators decided to change the model of service delivery before the Time 2 interviews were conducted. However, analyses of Time 2 interviews (conducted before

short standard

IMHJ (Wiley)

194

●

LEFT

BATCH


the home visitors were informed about the impending changes in the service delivery model) indicated that the barriers reported by the home visitors during the first year did not fade away. Rather, a higher number of home visitors reported difficulties with the team approach at Time 2. For example, at Time 1, 69% of the home visitors reported that they coordinated their activities with their teammates, and 85% reported holding regular meetings to coordinate teamwork. By Time 2, only 40% of the home visitors said that they coordinated their activities with their teammates, and merely 20% reported meeting on a regular basis. In addition, families found it stressful to continue scheduling visits with two home visitors, especially because the new welfare-to-work reform required mothers to seek employment outside the home. These changes made it necessary for the home visitors to schedule their visits in the evenings and sometimes during the weekends. Considering these factors, changes in the service delivery model seemed to be a step in the right direction. The program administrators believed that these changes would have positive consequences not only for the program families, but also for the staff. These would help avoid role confusion between home visitors, make the program more straightforward for the families, and eliminate conflicts arising from interpersonal dynamics, which, in turn, would lead to more efficient and effective service delivery. The second period of high staff turnover followed the above-mentioned changes in the program. Although the administrators expected this to happen, losing a sizable proportion of the staff at the same time placed additional demands on the staff members who stayed with the program. However, the program coped with this situation by hiring four home visitors within a month after the high turnover period. Also, the service delivery was temporarily curtailed due to program restructuring, which gave members an opportunity to pick up speed once these new positions were filled. As noted above, the home visitors’ level of job satisfaction showed a downward trend following these changes. Noteworthy is the positive change in the level of home visitors’ job satisfaction at Time 4, 8 months after the program changes. A corresponding low staff turnover affirmed the program’s decision to change the service delivery model. Further, when asked about their opinions regarding changes in the service delivery model at Time 3, a majority of the home visitors endorsed the program administrators’ views. For example, home visitors said that the recent changes would lead to better communication with program families, lesser role confusion, and lower caseloads. It is plausible, however, that those home visitors who stayed with the program through these changes also were more enthusiastic and optimistic (Buchbinder, Duggan, McFarlane, Fuddy, Windham, & Sia, 1999). In this regard, further investigation of these factors over time will be fruitful. The changes in the home visitation model described in this article raise a conceptual question about the model of service delivery in home visitation programs. Is the model based on a single home visitor more effective than the team approach? Home visitation programs are faced with the challenge of developing the most effective models of service delivery. On the positive side, having a single home visitor helps build a strong bond with the program participants, eliminates role confusion, and makes case management simple. Conversely, staff turnover in this case would result in complete discontinuity of service and hurt the continuation of a trusting relationship between the home visitor and the family. In addition, in this model, a single home visitor is expected to provide the full range of services described earlier in the article. Providing quality services requires the home visitors to be highly trained in child development as well as family issues. As pointed out by Wasik (1993), “In today’s atmosphere, home visitors need increased knowledge about family and child functioning, better consultation and counseling skills, and more ease with other skilled community workers than ever before” (p. 44). In the absence of comprehensive training, home visitors may neglect the areas in which they do not have expertise or confidence.


short standard

IMHJ (Wiley)

RIGHT

BATCH


●

195

Developing effective models of service delivery is tough in the absence of highly trained staff. Need for adequately trained personnel can only be fulfilled if the academic institutions respond to the expertise needed in the social services arena and integrate these elements into their educational agenda. The burgeoning demand for well-trained professionals in the field of early preventive intervention is likely to continue to grow in the near future given the current trends in the field of early preventive interventions. Crossdisciplinary training may help to provide resources and expertise required to offer such training. Finally, the issue of the role of the evaluator in responsive program evaluations is worth mentioning. To what extent should the evaluation steer program implementation? Experts have discussed the merits and drawbacks of having evaluators serve as program consultants as well as evaluators of program outcomes (Fine, Thornburg, & Ispa, 2000). In the present evaluation, the service providers and the evaluators explicitly agreed upon the role and purpose of the evaluation. The agency responsible for EHS program delivery did not wish the evaluation to spearhead any changes in the program nor influence program decisions. Rather, the goal of the evaluation was to keep the program “informed” by capturing the program processes and providing feedback from the evaluation data in a timely fashion. Therefore, the evaluators worked with the program closely to complete the above-mentioned tasks and provided information on staff development and other program processes, but decisions regarding the nature, manner, or extent of changes remained with the program. It should be noted that it is possible for the evaluators to also be program improvement consultants, especially when the focus is on process as well as outcome evaluation. This could result in joint problem solving, and might enhance the quality of the collaboration between researchers and program administrators. It could also raise questions about the integrity of the evaluation design, however. Thus, in the field of responsive program evaluations, the scope of evaluation continues to be an area of negotiation for the benefit of the program.


REFERENCES Boyer, P. (1978). Urban masses and moral order in America, 1800–1900. Cambridge, MA: Harvard University Press. Bryk, A.S., & Raudenbush, S.W. (1987). Application of hierarchical linear models to assessing change. Psychological Bulletin, 101, 147– 158. Bryk, A.S., & Weisberg, H.I. (1977). Use of the nonequivalent control group design when subjects are growing. Psychological Bulletin, 84, 950–962. Buchbinder, S.B., Duggan, A.K., McFarlane, E., Fuddy, L., Windham, A., & Sia, C. (1999, March). The impact of home visitor turnover on remaining co-workers. In The 11th Annual Research Conference Proceedings, a System of Care for Children’s Mental Health: Expanding the Research Base (pp. 253– 258). Chen, H. (1998). Theory-driven evaluations. In H.J. Wallberg (Series Ed.) & A.J. Reynolds (Vol. Ed.), Advances in educational productivity: Vol. 7. Evaluation research for educational productivity (pp. 15– 34). Greenwich, CT: JAI Press. Department of Health and Human Services. (2001). Building their futures: How Early Head Start programs are enhancing the lives of infants and toddlers in low-income families— summary report. Administration on Children, Youth and Families: Head Start Bureau. Durlak, J.A. (1998). Why program implementation is important. Journal of Prevention and Intervention in the Community, 17(2), 5– 18. Fine, M.A., Thornburg, K., & Ispa, J. (2000, June). When program evaluators are also program improvement consultants: Methodological, ethical, and logistical challenges. Poster session presented at Head Start’s Fifth National Conference, Washington, DC.

short standard

IMHJ (Wiley)

196

●

LEFT

BATCH


Gomby, D.S., Culross, P.L., & Behrman, R.E. (1999). Home visiting: Recent program evaluations— Analysis and recommendations. The Future of Children, 9(1), 4–26.


Holden, K.C., & Reynolds, A.J. (1997). Process evaluation of W-2: What it is, why it is useful, and how to do it? Evaluation comprehensive welfare reforms: A conference (pp. 139–156). Special Report No. 69. Madison, WI: Institute for Research on Poverty, University of Wisconsin–Madison. Kisker, E.E., Love, J.M., & Raikes, H. (1999). Leading the way: Characteristics and early experiences of selected Early Head Start Programs. Vol. 1. Cross-site perspectives. The Commissioner’s Office of Research and Evaluation and the Head Start Bureau. Kitzman, H., Yoos, H.L., Cole, R., Korfmacher, J., & Hanks, C. (1997). Prenatal and early childhood home-visitation program processes: A case illustration. Journal of Community Psychology, 25(1), 27–45. McCoy, A.R., & Reynolds, A.J. (1998). Evaluating implementation. In H.J. Wallberg (Series Ed.) & A.J. Reynolds (Vol. Ed.), Advances in educational productivity: Vol. 7. Evaluation research for educational productivity (pp. 117– 133). Greenwich, CT: JAI Press. McGraw, S.A., Stone, E.J., Osganian, S.K., Elder, J.P., Perry, C.L., Johnson, C.C., Parcel, G.S., Webber, L.S., & Luepker, R.V. (1994). Design of process evaluation within the child and adolescent trial for cardiovascular health (CATCH). Health Education Quarterly (supplement 2), S5–S26. Olds, D., Gomby, D., & McCroskey, J. (2000). Home visitation: Does it work? Children’s Institute International Forum [On-line] (pp. 1– 5). Available: www.childrensinstitute.org. Olds, D., Kitzman, H, Cole, R., & Robinson, J. (1997). Theoretical foundations of a program of home visitation for pregnant women and parents of young children. Journal of Community Psychology, 25(1), 9– 25. Powell, D.R. (1993). Inside home visiting programs. The Future of Children, 3(3), 23–38. Ramey, C.T., & Ramey, S.L. (1998). Early intervention and early experience. American Psychologist, 53(2), 109– 120. Robinson, J.L., Emde, R.N., & Korfmacher, J. (1997). Integrating emotional regulation perspective in a program of prenatal and early childhood home visitation. Journal of Community Psychology, 25(1), 59–75. Scheirer, M.A. (1994). Designing and using process evaluation. In J.S. Wholey, H.P. Hatry, & K.E. Newcomer (Eds.), Handbook of practical program evaluation (pp. 40–68). San Francisco: JosseyBass. Wasik, B.H. (1993). Staffing issues for home visiting programs. The Future of Children, 3(3), 141–157. Weiss, C.H. (1987). Evaluating social programs: What have we learned? Society, 25, 40–45. Weiss, H.B. (1993). Home visits: Necessary but not sufficient. The Future of Children, 3(3), 114–128. Weissberg, R.P., & Greenberg, M.T. (1998). School and community competence-enhancement and prevention programs. In W. Damon (Series Ed.), I.E. Sigel, & K.A. Renninger (Vol. Eds.), Handbook of child psychology: Vol. 4. Child psychology in practice (5th ed., pp. 877–954). New York: John Wiley & Sons.

short standard