Opening the Black Box: Using Process ... - Wiley Online Library

American Journal of Community Psychology, Vol. 27, No. 5 1999

Opening the Black Box: Using Process Evaluation Measures to Assess Implementation and Theory Building1 Tracy W. Karachi,2 Robert D. Abbott, Richard F. Catalano, Kevin P. Haggerty, and Charles B. Fleming Social Development Research Group, University of Washington, School of Social Work

The past decade has seen increasing recognition in prevention science of the need to move away from a black box approach to intervention evaluation and toward an approach that can elaborate on the mechanisms through which changes in the outcomes operate (Chen & Rossi, 1989; Durlak & Wells, 1997; Spoth et al., 1995). An approach that examines issues of program implementation is particularly critical in the design of efficacy studies of school-based preventive interventions. Numerous preventive intervention strategies are now delivered within the schools, often by regular classroom teachers. The extent to which teachers faithfully deliver a particular curriculum or incorporate instructional strategies emphasized by an intervention is a critical question for the overall project evaluation. This article illustrates the utilization of process measures from a multicomponent school-based prevention program to examine implementation of a teaching staff development intervention, and the program's underlying theoretical basis. Given the nested study design, the analyses utilize hierarchical linear models (Bryk & Raudenbush, 1992) to examine changes in teaching strategies by condition and investigate the hypothesized relationships between teaching practices and student behaviors based on the program's theoretical framework. Results suggest that teaching practices in two of the six intervention focus areas were positively impacted in the first 18 months of the project. Findings also support 1

Supported by Grant #RO1 DA08093 from the National Institute on Drug Abuse. The authors gratefully acknowledge Danielle Gangnes, Danielle Abbott, John Brown, Lindsay Dobrzynski, Cindy Impola, Linda Barber, Renee Bellamy, Kathy Nelson, Mary Casey-Goldstein, and staff at each of the participating schools for their support and cooperation for this project. 2 Requests for reprints should be sent to Tracy W. Karachi, Social Development Research Group, Box 358734, University of Washington, Seattle, Washington 98195. 711 0091-0562/99/1000-0711$16.00/0 © 1999 Plenum Publishing Corporation

712

Karachi, Abbott, Catalano, Haggerty, and Fleming

the relationships between teachers' instructional practices and students' behavior. KEY WORDS: program implementation; school-based prevention.

The field of prevention science continues to be challenged by questions about whether a program works, why, and under what circumstances. It has become increasingly important in intervention research to move beyond a "black box" approach to program evaluation (McLaughlin, 1987; Patton, 1979) and to examine, not only outcomes but also implementation fidelity and to differentiate between implementation failure and program failure. Many examples of intervention studies have reported no effects when, in fact, the lack of results stemmed from implementation failure (i.e., Type III error) rather than shortcomings in the intervention (Dobson & Cook, 1980; Kassebaum, Ward, & Wilner, 1971; Quay, 1979). While promising results of school-based social influence programs have been shown under efficacy trial conditions, considerable variability is reported across effectiveness trial studies (Goodstadt, 1988; Schaps et al., 1986). Furthermore, different results from similar prevention programs raise concerns that differential implementation may account for the variability (Pentz et al., 1990). For example, evaluations of educational programs in which teachers utilize classroom management and instructional practices have often focused on intervention and control group comparisons while assuming a dichotomous categorization between intervention and control groups (Kerr, Kent, & Lam, 1985). This perspective assumes that all intervention students receive comparable treatment. In reality, large variations may likely characterize adoption and implementation of practices in the experimental group. Furthermore, control teachers, who more appropriately should be termed "treatment as usual" teachers, may also routinely use the practices that are the target of the intervention and so expose control children to targeted practices. Thus, the degree to which teachers in both conditions implement targeted practices must be examined (Hawkins et al., 1991). Published material on process evaluation involving prevention programs is limited (e.g., Altman, 1986; McGraw et al., 1989; Norman et al., 1990; Pentz et al., 1990). Battistich and colleagues found that the strongest program effects (significant reductions in drug use and delinquency) were associated with students in schools where a greater degree of program implementation was demonstrated (Battistich et al., 1996). Rohrbach, Graham, and Hansen (1993) examined issues related to diffusion of a psychosocial-based substance abuse prevention program and found that integrity of program delivery was positively associated with immediate program outcomes. Implementers reported fewer years of teaching experience and

Implementation and Theory Building

713

strong self-efficacy, enthusiasm, preparedness, teaching methods compatibility, and principals' encouragement than did non-implementers. These two examples illustrate the need to incorporate examination of program implementation into the overall evaluation design to get beyond the black box and enhance our understanding of the outcomes. Preventive interventions, especially those in efficacy trials, would do well to include in their design a process evaluation that documents and analyzes the operationalization of a program to interpret the outcomes of the study. This issue is particularly salient for bridging the gap between prevention science and prevention practice. As researchers disseminate results from efficacy studies, it is critical for effectiveness trials and going to scale with these programs that practitioners and policy makers who are adopting the technology understand what conditions and circumstances are necessary to produce similar outcomes. There is growing support for the premise that program evaluations are more useful when grounded in a theoretical context (Finney & Moos, 1989). The theoretical context elucidates explanation or interpretation of the mechanisms through which the intervention effects occur. A program evaluation enhances its utility by examining the theoretical basis of the program and the intervening and contextual factors that mediate the relationship between the program and the ultimate outcome. Thus, information generated from this type of evaluation can improve intervention conceptualization while it contributes to theory development and validation. One of the challenges faced by prevention science (and noted by Spoth et al., 1996) is translating etiological models that specify global constructs and linking these constructs to behaviorally specific intervention targets. Finney and Moos (1989) agree that a critical issue in applying theories to evaluations is the degree to which the theory clearly specifies intervening processes or mechanisms that link the program activities and outcomes. Selecting or designing a program and operationalizing its key elements is the essential first step in designing an assessment of program implementation (Leithwood & Montgomery, 1980; Lipps & Grant, 1990). This paper illustrates the development and utilization of process measures from a study in progress that incorporates a theoretically integrated process evaluation into its overall evaluation design. Raising Healthy Children (RHC) is a comprehensive elementary school-based prevention program aimed at adolescent problem behaviors such as substance abuse and delinquency. Based on a risk-reduction and protective-factor enhancement framework (Hawkins, Catalano, & Miller, 1992), RHC targets specific risk and protective factors. Furthermore, it is theoretically guided by the social development model (Catalano & Hawkins, 1996; Catalano, Kosterman, Hawkins, Newcomb, & Abbott, 1996; Farrington & Hawkins, 1991;

714

Harachi, Abbott, Catalano, Haggerty, and Fleming

Hawkins & Weis, 1985). The program consists of strategies focused on reducing risk and enhancing protection for school teaching staff, students, and their families. The program provides teachers workshops in classroom management and instructional strategies designed to increase students' opportunities for involvement in the classroom, their social competency, and their commitment and attachment to school while reducing the risk factors of academic failure and early antisocial behavior. This staff development component is a universal intervention offered to all teachers at the experimental schools. The program also includes a universal family component that consists of parenting workshops available to any interested families and a selective intervention aimed at higher-risk families, home-based services. Additionally, the student component of the program provides opportunities for summer camp and an after-school study club. (See Haggerty, Catalano, Karachi, & Abbott, 1998, for a detailed project description.) The staff development component of the RHC program focuses on instructional strategies that have been shown in mainstream classrooms to be effective in reducing academic risks and early aggressiveness among elementary school-aged children (Gottfredson, 1990; Hawkins, Doueck, & Lishner, 1988) and to enhance the desired protective factors (Hawkins, Catalano, Morrison, et al., 1992). These strategies include (1) proactive classroom management, (2) cooperative learning methods, (3) strategies to enhance student motivation, (4) student involvement and participation, (5) reading strategies, and (6) interpersonal and problem-solving skills training. Proactive classroom management is strongly supported by studies of teacher effectiveness that link high academic achievement and motivation with clear, authoritative leadership and direction on the teacher's part (Brophy & Good, 1986; Doyle, 1986). Studies have shown that proactive classroom management contributes to increasing time on task (Walberg, 1988), and increases students' attachment and commitment to school (Hawkins, Catalano, Morrison, et al., 1992; Hawkins, Doueck, & Lishner, 1988). Cooperative learning has been found to positively influence social and academic learning and intergroup relationships among students (Aronson, Bridgeman, & Geffner, 1978; Slavin, 1990). Mastery of learning tasks, motivation, positive student attitudes toward teachers and schools, and self-concept are greater in cooperative classrooms than in competitive or individualistic ones (Johnson & Johnson, 1980). When students are motivated and actively involved in learning activities and teachers monitor and assess students' comprehension and progress, higher achievement is the result (Brophy, 1987). Reading is provided as a content emphasis for students because it is the subject most strongly related to verbal


715

skills and later academic success (Brophy, 1990; Slavin, 1990), and is a critical focus area in the primary school period (Fielding & Pearson, 1994; Routman, 1991). Last, learning social and emotional skills can have a positive effect, not only on students' interactions with others but on their attitudes toward school and toward responsible norm-following behavior, their interactions with teachers and other adults, and their academic achievement (Aronson et al., 1978; Rotheram, 1982; Wentzel, 1991), There is also evidence that learning positive social and emotional behavior helps to reduce negative and antisocial behavior (Caplan & Weissberg, 1989; Dusenbury & Falco, 1997; Patterson, 1986; Weissberg et al., 1981). The extent to which experimental teachers incorporate the instructional strategies emphasized by staff development is a critical variable that must be examined during the evaluation of the project overall. The process evaluation for this study is designed to collect information about who receives the staff development intervention and at what level of exposure and to assess to what extent instructional practices are a part of the teaching environment in classrooms of either condition (McGraw et al., 1989). Furthermore, the process evaluation is designed to examine whether there are greater changes, in terms of adoption and implementation of the target practices, among the experimental teachers exposed to the staff development intervention than among the control teachers. Finally, the theoretical relationship between implementation of target instructional practices and student outcomes is examined. In natural environments, many of the project's target instructional strategies are used by teachers. Thus utilizing a dichotomous experimental-control teacher comparison to measure the relationship between practice and student outcomes is not appropriate. Rather, the analysis of the theoretical questions about the proximal impact of the instructional practices on student behaviors will use measures of teacher practices and student outcomes, regardless of condition. The paper illustrates the development and utilization of process measures—in this instance, a classroom observation system—to examine implementation of a staff development intervention and of the program's underlying theoretical basis. The balance of this paper describes the observation system that was developed to assess implementation of the project's instructional strategies and results of analyses, that reflect two distinct objectives of the process evaluation: (1) sensitivity of the observational measures to changes in teaching strategies and (2) an examination of the relationships between teaching practices and student behaviors hypothesized in the project's theoretical framework.

716


METHODS Project Design The RHC project begun in October 1993 includes children in 10 suburban public elementary schools. The schools were randomly assigned to intervention or "treatment as usual" control conditions. The study panel consists of first- and second-grade students in these schools. Parents of 76% of the eligible first- and second-grade students actively consented to participate (n = 1,040). The study had several sources of data that were collected at multiple time points. The two sources reported here are classroom observations and student behavior checklists. Observations of teachers in the classroom are collected annually in the Fall (Winter in Year 1) and Springtime. Teachers complete student behavior checklists at the same intervals. Observation System Instrument Description Before the intervention started, the research team reviewed the content of the staff development program. Operational definitions of the targeted teaching strategies were developed and were used to construct the observation instruments, which consisted of a continuous coding form and a classroom checklist. The continuous coding form is divided into a teacherfocused section and a student-focused section. Both the coding form and the checklist contain instructional strategies and behaviors that are organized by the six focus areas of the project's staff development: 1. Proactive Classroom Management (e.g., appropriate behavior management, clear procedures, dead time, off-task students) 2. Motivation (e.g., praise from the teacher and goal setting by students) 3. Student Involvement (e.g., teachers' check for understanding, and ratings of students' engagement in the learning task) 4. Social Skills Reinforcement (e.g., behaviors of the teacher as well as reinforcers in the classroom environment) 5. Reading (e.g., includes time spent on reading) 6. Cooperative Learning (e.g., opportunities for students to work cooperatively). The RHC Classroom Observation System Manual (for detailed de-


717

scription see Karachi et al., 1993) describes in detail each of the variables that comprise the six focus areas. Each variable is illustrated with many examples of the specific practice or behavior and, when appropriate, the distinguishing attributes of the variable. For example, under the variable of Praise, positive recognition is divided into three types: (1) praise that emphasizes the students' internal attribution, (2) praise that gives external attribution for specific behavior, and (3) praise for nonspecific behavior. Staff development encourages teachers to use positive recognition that is behaviorally specific and emphasizes students' internal attribution. Therefore, this type of praise is given the highest coding value and praise for nonspecific behavior the lowest. On the continuous coding form, the observer records instances of the teaching strategies and behaviors in 90-second intervals. During the first 60 seconds, the focus is on coding teaching strategies and teacher behavior. In the next 30 seconds, the focus is on coding classroomwide student behaviors and those of a randomly selected subset of students. The coding form consists of 20 90-second intervals for a total of 30 minutes per observation. The checklist is not part of the timed observation but is completed at the end of the 30-minute observation period. Observer Training Observers are provided 40 hours of intensive training before data collection. They are given the observation manual for at-home study before the start of training. Next, the field supervisor provides group instruction and a review of the content. Practice is also given in the timing of the coding form and procedures for randomly selecting students for the studentfocused segment. Observers review videotaped segments of classroom activities and then have opportunities to rate practice videotapes. These practice segments allow the observers to become familiar with the timing sequence of the coding form and to fine-tune their knowledge of rating each of the categories. A portion of the training time is devoted to critical issues such as limiting one's presence in the classroom and administrative procedures for the field period. The last phase of training consists of observers' coding in live-practice classrooms along with the observation supervisor. Administration Participants include experimental teachers and control teachers in the 10 project schools. Active consent was gathered from teachers before the

718


start of the initial field period. To provide a consistent curriculum area, observations were conducted during reading or language arts classes. At the time of consent, teachers provided a schedule of their reading and language arts periods for the coming term. Although teachers knew in advance in which curriculum period observations would occur, they were not told on which particular days. During each data collection period, teachers were observed on three occasions within a 2-week period.

Reliability Results To examine the correlation of scores among observers, a comparison was made of raters on their coding of 30-minute videotape test segments before the start of the field period and over its course. Beginning in Year 2, observations were conducted with the supervisor and each observer during each field period paired to assess interrater reliability under field conditions. Table I reports the range of agreement between observers rating test videotapes and the range of agreement scores between the field supervisor and individual observers in the field. Both the teacher- and student-focused categories had multiple dimensions; thus, multiple scored variables constitute the correlations across observers. The probability of chance agreement is quite low. Most correlations are high: in particular, scores in the second year demonstrate increased stability and consistency of rating, reflecting improvements in the training and supervision process.

Table I. Range of Interobserver Correlations

Winter Year 1 Test Tape A Test Tape B Spring Year 1 Test Tape C Test Tape D Fall Year 2 Test Tape E Test Tape F Field Interrater Spring Year 2 Test Tape G Test Tape H Field Interrater

Overall Corrections

Teacher Focus Categories

Student Focus Categories

74-.93 .V7-.94

.60-.95 .66-.94

.75-.90 .58-.85

.88- .98 .69-.96

.87-.99 .60-.94

.71-.97 .65-.91

.93-.99 .93-.97 .90-.99

.93-.99 .91-.97 .88-.99

.87-.95 .88-.92 .79-.99

.97-.99 .89-.98 .89-.99

.96-.99 .79-.99 .91-.99

.85-.97 .80-.96 .83-1.00

719


Summary Measures Average scores for each observation variable were computed using the three observations of each teacher. Summary positive and summary negative scale scores were created for each of the six focus areas: (1) proactive classroom management, (2) motivation, (3) student involvement, (4) cooperative learning, (5) reading, and, (6) social skills reinforcement. Observation items were categorized as either positively contributing to the high-quality implementation of the targeted teaching strategies or detracting from implementation. For example, under the focus area of Proactive Classroom Management, when teachers gave a set of directions that the majority of the class were able to follow, this was included in the positive summary measure of proactive classroom management. Conversely, when teachers provided directions that more than half the class were unable to follow, this fact was included in the negative summary measure. Additionally, a global positive scale and a global negative scale were created by adding together the positive or negative summary scales for each of the six focus areas. Standardized z scores were created for each positive or negative summary scale before summation so that the contribution of each scale to the global scale was equivalent (Table II). Other Measures Teacher Checklist of Students' Behavior The second set of analyses reported here utilized Year 2 data from the teacher-rated student behavior checklists which were completed on study panel students. Completion rates for the behavior checklists were Table II. Scales Means and Standard Deviations for Teacher Survey of Students

Social competency Mean

SD School commitment Mean

SD Antisocial behavior Mean

SD

Fall 1994

Spring 1995

.04 .02

.04 .01

3.32

3.35

.60

.60

1.10

1.23

.31

.30

720


96,7% in the Fall and 95.4% in the Spring; however, some students had moved out of project schools so their teachers were not observed. Additionally, some students did not have the same teacher throughout the school year. Only data for students who attended project schools and had the same teacher were utilized in the second set of analyses (n = 689). Three scales from the behavior checklists were utilized in the analyses. Social competency was assessed by a scale comprised of nine individual items taken from the Teacher Observation of Classroom Adaptation Revised (TOCA-R) (Werthamer-Larsson, Kellam, & Oveson-McGregor, 1990) and the Walker-McConnell Scale of Social Competence and School Adjustment (Walker & McConnell, 1988). Examples of the items include "Cooperates with peers in group activities" and "Resolves peer problems on his/her own." Since response options differ between items, the items were standardized before being scaled. The social competency scale has an alpha (a) coefficient of 0.94 at each of the time points. Students' commitment to school was assessed by a scale (a = 0.88 at both time points) of two items: "Student tries hard in school" and "Student wants to do well in school." This scale was taken from the Seattle Social Development Project (Hawkins, Catalano, Morrison, et al., 1992). The measure to assess antisocial behavior consists of 10 items taken from the TOCA-R (Werthamer-Larsson, Kellam, & Oveson-McGregor, 1990) and Child Behavior Checklist-Teacher Report (Achenbach, 1991). Scores on this scale have a coefficients of 0.91 and 0.93 for the two time points. Some items the students were rated on at the present time or within the past month, were "Stubborn," "Threatens people," and "Argues a lot." Table III provides the means and standard deviations for the scales used in these analyses.

Analysis This article reports results on two separate analyses. The first examines changes in teaching practices by condition after the first 18 months of intervention implementation. The second investigates the hypothesized relationships between teaching practices and student behaviors based on the program's theoretical framework using classroom observation data and teachers' reports of student behavior. Hierarchical linear modeling was used for both analyses (Bryk & Raudenbush, 1992). This technique has several features that address limitations of traditional linear model analysis. In relation to this study, the technique overcomes lack of independence among observations for each subject in a repeated-measures design (Bryk & Raudenbush, 1987) and takes into account the nested structure of the data (students within classrooms within schools; Arnold, 1992).


721

Table HI. Means and Standard Deviations for Summary Scalesa Fall 1994 n = 57 Proactive Classroom Management-Positive Mean SD Proactive Classroom Management-Negative Mean SD Social Skills Reinforcement-Positive Mean SD Involvement-Positive Mean SD Involvement-Negative Mean SD Motivation-Positive Mean SD Motivation-Negative Mean SD Reading-Positive Mean SD Reading-Negative Mean SD Cooperative Learning-Positive Mean SD Cooperative Learning-Negative Mean SD

Spring 1995 n = 57

7.53 1.80

7.58 2.03

-1.05 1.21

-1.22 1.21

.77 .92

.56 .92

2.54 .89

2.44 1.33

-.43 .53

-.62 .60

.69 .69

.78 .96

-2.44 1.19

-2.68 1.42

1.87 .89

1.86 .93

-.54 .61

-.57 .62

2.07 .35

2.13 .37

-1.93 .35

-1.87 .37

a

There was no occurrence of items in the Social Skills Reinforcement-Negative Scale so it was not reported in the table.

Examination of Teacher Practice Change by Condition Teaching practices of experimental and control teachers were compared over the first 2 years of the project. Teachers who had four observation points over this time were included in the analyses (n = 42). The sample of teachers per school ranged from one to eight. Using growth curve analyses, hierarchical linear models were estimated for each of the positive and negative summary and global scales. A three-level model was constructed with repeated classroom observation scores at Level 1, the teacher identifier as Level 2 predictor of the intercept and slope of the repeated


722

Fig. 1. Equations to model growth in teacher practices by condition.

measures, and the school as Level 3. Fall and Spring observations from Years 1 and 2 were predicted by time coded as -3 (Winter, Year 1), -2 (Spring, Year 1), -1 (Fall, Year 2), and 0 (Spring, Year 2) in Level 1. An initial model for each outcome was run to examine the amount of variance left to be explained. Results of the initial model suggest that it is appropriate to proceed to entering other predictors into the model. Therefore, condition assignment (experimental = 1, control = 0) was entered as a predictor of the intercept and slope coefficients of the Level 3 equation (see Fig. 1). Results reported in Table IV suggest that two of the teaching practice focus areas were significantly affected by the intervention. Experimental condition affected the average level of positive involvement practices in the end of Spring of Year 2 and the change in involvement practices over Table IV. Coefficients and Significance Tests for the Individual Growth Models for Teaching Practice Summary Scales by Condition Coefficient Positive Involvement Intercept value Condition Slope value Condition Positive Social Skills Reinforcements Intercept value Condition Slope value Condition

T Ratio

p Value

2.000 0.705 -0.182 0.433

7.508 2.272 -1.314 2.691

.001 .052 .225 .028

0.287 0.745 -0.079 0.231

1.073 2.352 -0.686 1.700

.315 .046 .512 .127

723


Fig. 2.

the four time points. Figures 2 and 3 show these results graphically and represent predicted estimates derived from substituting the coefficients. At the first measurement point, experimental teachers had lower observed levels of positive student involvement practices than control teachers, but in subsequent measurement points experimental teachers showed growth

Fig. 3.

724


in involvement practices while the control teachers' use of these practices declined. Experimental condition affected the average level of positive social skills reinforcement practices at the Spring of Year 2 and had a trend level effect on the change in these practices over the four time points (see Fig. 3). At baseline, the experimental and control teachers had similar levels of positive social skills reinforcement but experimental teachers had much growth while control teachers' use of these skills declined over time. This resulted in a higher level of use of positive social skills reinforcement among experimental teachers than controls at the end of Year 2.

Relationship Between Teaching Practices and Student Behaviors The social development model suggests that student behaviors will be affected by instructional practices in the classroom. In particular, instructional practices are hypothesized to directly affect perceived opportunities for involvement, students' skills for involvement, and perceived rewards for involvement. To investigate the theoretical relationships between teaching practices and student behaviors based on the program's theoretical framework, data for those students attending project schools during the second year of the project and whose classroom teacher was observed were utilized in these second analyses. Given the limitations of available data, only the constructs and relationships shown in Fig. 4 were investigated. Thus, it is a partial examination of the relationships in the social development model. Hierarchical linear modeling was used to examine the relationship between teacher practices observed in the Fall of Year 2 and differences in student behavior between Fall and Spring of that year. Separate models were created to predict three student behaviors; students' social compe-

Fig. 4.

725


Fig. 5.

tency, bonding to school, and antisocial behavior (Fig. 5). Individual students' difference scores from Fall to Spring of Year 2 were entered in at Level 1. In Level 2, each model included the Fall Year 2 teacher's observation score on each positive and negative summary scale score as predictors of the intercept coefficient of the Level 1 equation. Level 3 reflects the nesting of classrooms within schools, although no school level predictors were entered. The parameter estimate of the teacher summary score was modeled as fixed, since no significant differences were found between schools on each of the teacher summary predictors. Table V displays the values of the coefficients and significance tests for each of the estimated parameters. Table V. Coefficients and Significance Tests for Fall-to-Spring Changes in Student Behaviors Predicted by Teachers' Practices

Social Competency Intercept value Positive involvement teaching practice Intercept value Proactive classroom management teaching practice Intercept value Negative involvement teaching practice School Commitment Intercept value Proactive classroom management teaching practice Antisocial Behavior Intercept value Global positive teaching practice

Coefficient

T Ratio

-.1928 0.0716 -0.4200 0.0534

-2.231 2.199 -3.576 3.570

.052 .033 .007 .001

0.0480 0.1417

1.272 2.396

.236 .021

-.1387 .0242

-1.427 1.973

.187 .054

0.0591 -0.0078

3.963 -2.094

.004 .042

p Value

726


These results support the relationships hypothesized in the social development model. The effect of teachers' instructional skills is directly related to students' skills for interaction or operationalized here, as degree of social competency. In particular, positive student involvement teaching practices significantly predicted Fall-to-Spring difference scores in students' social competency. Predicted values based on the equations given in Fig. 5 were calculated for students in classrooms where the teacher demonstrated an average amount of positive involvement practices. Additionally, predicted values were calculated for teachers at 1 SD higher and lower in the amount of practice. Students in classrooms whose teachers demonstrated positive involvement practice 1 SD higher showed the greatest gains in social competency from Fall to Spring. Conversely, students in classrooms whose teachers were rated 1 SD lower than the mean in involvement practices declined in social competency over the course of the year. Proactive classroom management was significantly positively related to students' gains in social competency. Establishing a clear set of routines and consequences in classrooms was associated with positive gains in students' social competency over the course of the school year. Conversely, using negative practices that reduced opportunities for students' involvement was associated with decreases in social competency. Another part of the social development model suggests that the effects of teachers' instructional skills on school bonding are mediated by perceived rewards. There was no measure of perceived rewards to test the mediation; however, the relationship between proactive classroom management and school bonding was found to approach significance. Thus, teachers' demonstrating proactive classroom management techniques seemed to promote both students' social competency and bonding to school. Finally, we examined whether any of the instructional practices were related to the outcome of antisocial behavior. The global measure of positive instructional practices was significantly associated with decreases in antisocial behavior.

DISCUSSION The results from these analyses illustrate how process measures can be utilized to move beyond a black box evaluation design. The RHC classroom observation system shows promise as a tool for assessing the degree of implementation of the staff development intervention. The observation system, in combination with other process measures (e.g., attendance records at staff development workshops and a checklist of content covered

Implemenlation and Theory Building

727

in the staff development sessions) provide implementation data that will be useful in the overall examination of program effectiveness. The first question these analyses addressed was whether teachers in the experimental condition increased their use of targeted instructional strategies relative to what comparison teachers did. The results suggest that experimental teachers adopted some of the instructional practices promoted by the project by the end of 18 months' intervention. Experimental teachers demonstrated better implementation than control teachers of practices that promote student involvement and practices that promote social skills reinforcement. These findings suggest short-term effects for the intervention on positive teaching practices. Social skills instruction was given as the teachers' first workshop. Perhaps as a result of coaching and the extended time available to incorporate social skills reinforcement into their instructional practices, use of this skill increased over the four time points among experimental group teachers. On the other hand, the planned effect of each of the staff development sessions was to increase opportunities for involvement in the classroom. For example, in the proactive classroom management session, teachers are encouraged to check for understanding that involves the majority of the class, and in the reading staff development session they are encouraged to use methods such as paired reading, which promote involvement. Thus, enhancement of students' involvement is woven throughout the staff development sessions. The steady growth in this area among experimental teachers, as compared with the steady decline among control teachers, seems to be consonant with this intent. Alternatively, these practices may be easier to incorporate into established teaching practices or may be more acceptable to teachers. Further analysis is planned to track incorporation of other promoted teaching practices over time. The failure to realize significant reductions in the negative teaching practices through the intervention suggests the need for further examination of its potential to affect negative teaching practices. Knowing the origins of the differences in changes in positive (versus negative) teaching practices may be important for identifying possible implementation issues that need strengthening. If teachers who received staff development are increasing their positive teaching practices but not reducing negative ones, further examination of what factors contribute to this differential adoption is in order. Such an examination can provide information that then may be utilized by program personnel to make appropriate modifications or course corrections in the program to produce maximal implementation. Additionally, analyses that examine predictors associated with change—either greater adoption of positive practices or extinguishing negative practices— will be useful. Rohrbach, Graham, and Hansen (1993) found particular characteristics of teachers and of the school environment that were associ-

728


ated with greater implementation. This type of analysis could also inform program delivery in the future, in terms of identifying teachers who may require additional assistance or support in adopting the targeted instructional practices. Furthermore, the results of the second set of analyses underscore the importance of reducing or extinguishing the negative teaching practices. Teachers' practices that limited opportunities for student involvement were associated with decreases in social competency. However, these analyses also demonstrated that the global positive practice scale was related to antisocial behavior. Thus, both enhancing positive practices and extinguishing negative ones appear to be important. The results of the second analysis illustrate how process measures can be utilized to examine the theoretical basis of a preventive intervention. Teacher practices were significantly related to three student behaviors, as hypothesized by the social development model. Instructional practices that provide opportunities for involvement in the classroom enhance students' social competency. Additionally, students in classrooms where teachers manage proactively are more likely to gain social competency over the course of a school year. Proactive classroom managing was also associated with increases in students' school bonding, suggesting that these classroom environments nurture their attachment or commitment to learning. Last, students whose teachers provided more positive instructional practices appeared to exhibit decreases in antisocial behaviors over the course of the school year. While these measures are showing some promise of being sensitive to change and appear to be related to students' attitudes and behaviors in theoretically predicted ways, the data are still preliminary. We have examined only static predictors of changes in students' behaviors. Clearly, over the elementary school period students are exposed to multiple teachers, and our models must incorporate this information. For example, how the instructional skills of multiple teachers contribute to social competency or antisocial behaviors is still unknown. Is the best model of teaching practice exposure a sum of different teachers' scores the appropriate measure or one that incorporates practice as a year to year influence? Further, replication and extension of these findings in future years will provide more insights into the system's usefulness as an implementation and theorytesting measure.

REFERENCES Achenbach, T. M. (1991). Manual for the teacher's report form and 1991 profile. Burlington, VT: University of Vermont, Department of Psychiatry.


729

Altman, D. G. (1986). A framework for evaluation of community based heart disease prevention programs. Social Science and Medicine, 22, 479-487. Arnold, C. L. (1992). An introduction to hierarchical linear models. Measurement and Evaluation in Counseling and Development, 25, 58-90. Aronson, E., Bridgeman, D. L., & Geffner, R. (1978). Interdependent interactions and prosocial behavior. Journal of Research and Development in Education, 12, 16-27. Battistich, V., Schaps, E., Watson, M., & Solomon, D. (1996). Prevention effects of the Child Development Project: Early findings from an ongoing multisite demonstration trial. Special Issue: Preventing adolescent substance abuse. Journal of Adolescent Research, 11, 12-35. Brophy, J. (1987). Synthesis of research on strategies for motivating students to learn. Educational Leadership, 45, 40-48. Brophy, J. (1990). Teaching social studies for understanding and higher-order applications. Elementary School Journal, 90, 351-417. Brophy, J., & Good, T. L. (1986). Teacher behavior and student achievement. In M. C. Wittrock (Ed.), Handbook of research on training (pp. 328-375). New York: Macmillan. Bryk, A. S., & Raudenbush, S. W. (1987). Application of hierarchical linear models to assessing change. Psychological Bulletin, 1011, 147-158. Bryk, A. S., and Raudenbush, S. W. (1992). Hierarchical linear models: Applications and data analysis methods. Thousand Oaks, CA: Sage Publications. Caplan, M. Z., & Weissberg, R. P. (1989). Promoting social competence in early adolescence: Developmental considerations. In B. H. Schneider, G. Attili, J. Nadel, & R. P. Weissberg (Eds.), Social competence in developmental perspective (pp. 371-385). Boston: Kluwer Academic. Catalano, R. F., & Hawkins, J. D. (1996). The social development model: A theory of antisocial behavior. In J. D. Hawkins (Ed.), Delinquency and crime: Current theories (pp. 149-197). New York: Cambridge University Press. Catalano, R. F., Kosterman, R., Hawkins, J. D., Newcomb, M. D., & Abbott, R. D. (Spring, 1996). Modeling the etiology of adolescent substance use: A test of the social development model. Journal of Drug Issues: Empirical Validity of Theories of Drug Abuse, 26,429-455. Chen, H., & Rossi, P. H. (1989). Issues in the theory-driven perspective. Evaluation and Program Planning, 12, 299-306. Dobson, L. D., & Cook, T. J. (1980). Avoiding type III error in program evaluation: Results from a field experiment. Evaluation and Program Planning, 3, 269-376. Doyle, W. (1986). Classroom organization and management. In M. E. Wittrock (Ed.), Handbook of research on teaching, 392-431. New York: Macmillan. Durlak, J. A., & Wells, A. M. (1997). Primary prevention mental health programs for children and adolescents: A meta-analytic review. Special Issue: Meta-analysis of primary prevention programs. American Journal of Community Psychology, 25, 115-152. Dusenbury, L., & Falco, M. (1997). School-based drug abuse prevention strategies: From research to policy and practice. In R. P. Weissberg, T. P. Gullotta , R. L. Hampton, B. A. Ryan, & G. R. Adams (eds.), Healthy children 2010: Enhancing children's wellness. (pp. 47-75). Thousand Oaks, CA: Sage Publications. Farrington, D. P., & Hawkins, J. D. (1991). Predicting participation, early onset, and later persistence in officially recorded offending. Criminal Behavior and Mental Health, 1,1-33. Fielding, L. G., & Pearson, D. P. (1994). Reading comprehension: What works? Educational Leadership, 1, 62-68. Finney, J. W., & Moos, R. H. (1989). Theory and method in treatment evaluation. Evaluation and Program Planning, 12, 307-316. Goodstadt, M. J. (1988). School based drug education in North America: What is wrong? What can be done? Journal of School Health, 56, 278-281. Gottfredson, D. C. (1990). Changing school structures to benefit high-risk youths. In P. E. Leone (Ed.), Understanding troubled and troubling youth (pp. 246-271). Newbury Park, CA: Sage Publications. Haggerty, K. P., Catalano, R. F., Karachi, T. W., & Abbott, R. D. (1998). Preventing adolescent

730


problem behaviors: A comprehensive intervention description. Criminologie, 31(1), 25-47. Karachi, T. W., Haggerty, K. P., Catalano, R. F., Cummings, C., Gangnes, D., & Abbott, R. A. (1993). RHC classroom observation system manual. Unpublished instrument. Hawkins, J. D., Abbott, R. A., Catalano, R. F., & Gillmore, M. R. (1991). Assessing effectiveness of drug abuse prevention: Long-term effects and replication. In C. Leukfeld & W. Bukoski (Eds.), Drug abuse prevention research: Methodological issues. NIDA Research Monograph 107, DHHS Publication No. 91-1761:195-212. Washington, DC: U.S. Government Printing Office. Hawkins, J. D., Catalano, R. F., & Miller, J. Y. (1992a). Risk and protective factors for alcohol and other drug problems in adolescence and early adulthood: Implications for substance abuse prevention. Psychological Bulletin, 54, 661-664. Hawkins, J. D., Catalano, R. F., Morrison, D. M., O'Donnell, J., Abbott, R. D., & Day, L. E. (1992b). The Seattle Social Development Project: Effects of the first four years on protective factors and problem behaviors. In J. McCord & R. Tremblay (Eds.), The prevention of antisocial behavior in children (pp. 139-161). New York: Guilford. Hawkins, J. D., Doueck, H. J., & Lishner, D. M. (1988). Changing teaching practices in mainstream classes to improve bonding and behavior of low achievers. American Educational Research Journal, 25, 31-50. Hawkins, J. D., & Weis, J. G. (1985). The social development model: An integrated approach to delinquency prevention. Journal of Primary Prevention, 6, 73-97. Johnson, D. W., & Johnson, R. T. (1980). Effects of cooperative, competitive, and individualistic learning experiences on cross-ethnic interaction and friendships. Journal of Social Psychology, 118, 47-58. Kassebaum, G., Ward, D. A., & Wilner, D. M. (1971). Prison treatment and parole survival. New York: John Wiley. Kerr, D. M., Kent, L., & Lam, T. C. M. (1985). Measuring program implementation with a classroom observation instrument. The interactive teaching map. Evaluation Review, 9, 461-482. Leithwood, K. A., & Montgomery, D. J. (1980). Evaluating program implementation. Evaluation Review, 4, 193-214. Lipps, G., & Grant, P. R. (1990). A participatory method of assessing program implementation. Evaluation Review, 14, 427-434. McGraw S. A., McKinlay, S. M., McClements, L., Lasater, T. M., Assaf, A., & Carleton, R. A. (1989). Methods in program evaluation: The process evaluation system of the Pawtucket Heart Health Program. Evaluation Review, 13, 459-483. McLaughlin, M. W. (1987). Implementation realities and evaluation design. Evaluation Studies Review Annual, 12, 73-97. Norman, S. A., Greenberg, R., Marconi, K., Novelli, W., Felix, M., Schechter, C., Stolley, P., & Stunkard, A. (1990). A process evaluation of a two-year community, cardiovascular risk reduction program: What was done and who knew about it? Health Education Research, 5, 87-97. Patterson, G. R. (1986). Performance models for antisocial boys. American Psychologist, 41, 432-444. Patton, M. Q. (1979). Evaluation of program implementation. Evaluation Studies Review Annual, 4, 318-345. Pentz, M. A., Trebow, E. A., Hansen, W. B., MacKinnon, D. P., Dwyer, J. H., & Johnson, C. A. (1990). Effects of program implementation on adolescent drug use behavior. The Midwestern Prevention Project (MPP). Evaluation Review, 14, 264-289. Quay, H. C. (1979). The three faces of evaluation: What can be expected to work. In L. Sechrest, S. G. West, M. A. Phillips, R. Redner, & W. Yeaton (Eds.), Evaluation studies review annual, (Vol. 4). Beverly Hills, CA: Sage Publications. Rohrbach, L. A., Graham, J. W., & Hansen, W. B. (1993). Diffusion of a school-based substance abuse prevention program: Predictors of program implementation. Preventive Medicine, 22, 237-260.


731

Rotheram, M. J. (1982). Social skills training for underachievers, disruptive, and exceptional children. Psychology in the Schools, 19, 532-539. Routman, R. (1991). Invitations. Portsmouth, NH: Heinemann. Schaps, E., Moskowitz, J. M., Malvin, J. H., & Schaeffer, G. A. (1986). Evaluation of seven school-based prevention programs: A final report on the Napa project. International Journal of the Addictions, 21, 1081-1112. Slavin, R. E. (1990). Cooperative learning theory, research and practice. Englewood Cliffs, NJ: Prentice Hall. Spoth, R., & Redmond, C. (1996). Illustrating a framework for prevention research: Project Family studies of rural family participation and outcomes. In R. Peters & R. McMahon (Eds.), Childhood disorders, substance abuse, and delinquency: Prevention and early intervention approaches (pp. 299-328). Newbury Park, CA: Sage. Spoth, R., Redmond, C., Haggerty, K., & Ward, T. (1995). A controlled outcome study examining individual difference and attendance effects. Journal of Primary Prevention, 57, 449-464. Walberg, H. J. (1988). Synthesis of research on time and learning. Educational Leadership, 45, 76-85. Walker, H. M., & McConnell, S. R. (1988). The Walker McConnell Scale of social competence and school adjustment. Austin, TX: Pro-Ed. Weissberg, R. P., Gesten, E. L., Carnrike, C. L., Toro, P. A. Rapkin, B. D., Davidson, E., & Cowen, E. L. (1981). Social problem-solving skills training: A competence building intervention with second- to fourth-grade students. American Journal of Community Psychology, 9, 411-423. Wentzel, K. R. (1991). Social competence at school: Relation between social responsibility and academic achievement. Review of Educational Research, 61, 1-24. Werthamer-Larsson, L., Kellam, S. G., & Ovesen-McGregor, K. E. (1990). Teacher interview: Teacher Observation of Classroom Adaptation—Revised (TOCA-R). In S. G. Kellam (Ed.), Johns Hopkins prevention training manual. Baltimore: Johns Hopkins University.