Psychometric Properties and Reference Point Data for the ... - CiteSeerX

BEHAVIOR THERAPY

31,695-712,2000

Psychometric Properties and Reference Point Data for the Revised Edition of the School Observation Coding System JENIFER R. JACOBS

STEPHEN R. BOG~S SHEILA M . EYBERG DANIEL EDWARDS PATRICIA DURNING JANE G. QUERIDO

Universi~ of Florida CHERYL

B. M C N E I L

West Virginia Unii,ersity BEVERLY W. FUNDERBURK

Universi~ of Oklahoma Health Science Center The psychometric properties of a new observation coding system for children's disruptive classroom behavior were evaluated. The Revised Edition of the School Observation Coding System (REDSOCS) was used to observe 51 young children clinic-referred for conduct-disordered behavior and 182 nonreferred children from the classrooms of the referred children. Reference point data for the REDSOCS categories with preschoolers were obtained from the sample of nonreferred children. Interobserver reliability and concurrent validity of the three REDSOCS categories with teacher rating scales of oppositional behavior and hyperactivity were demonstrated. Initial evidence of convergent and discriminant validity was established through correlations of the REDSOCS categories with the subscales of the Revised Conners Teacher Rating Scale. Differences in REDSOCS scores between the nonreferred children and children referred for school behavior problems provided eviJenifer Jacobs is now at University of California at San Diego and Children's Hospital of San Diego; Daniel Edwards is now at Multi Systemic Therapy Services, Charleston. This work was supported by USPHS Grant MH-46726 from the National Institute of Mental Health. Address correspondence to Sheila M. Eyberg, Ph.D., Department of Clinical and Health Psychology, P.O. Box 100165, University of Florida, Gainesville, FL 32610; e-mail: seyberg@ hp.ufl.edu. 695 005-7894/00/0695~71251.00/0 Copyright2000 by Associationfor Advancementof BehaviorTherapy All rightsfor reproductionin any formreserved.

696

JACOBS ET AL. dence of discriminative validity for the REDSOCS categories. The REDSOCS was also found to classify correctly the majority of referred children according to the presence or absence of school behavior problems. The results suggest that the REDSOCS is a promising instrument for measuring disruptive classroom behavior in preschoolers.

Disruptive behavior in the school setting constitutes a significant problem. Approximately 10% to 15% of school-age children meet DSM-IV (American Psychiatric Association, 1994) criteria for oppositional defiant disorder (ODD) or attention-deficit/hyperactivity disorder (ADHD), or both (Kazdin, Siegel, & Bass, 1990), and as many as 22% of children in a given classroom exhibit problem behaviors such as hyperactivity, inattention, or aggression (Costello, 1989). There is typically an expansion of settings in which conduct problems occur over time, from the home to settings such as the school and the broader community (Eyberg, Schuhmann, & Rey, 1998). Educators, researchers, and clinicians need instruments that permit assessment of these school problems. In addition to the identification of children with significant disruptive behavior, instruments are needed that can monitor interventions in the classroom. Rating scales have been used for many years as the primary tool for the assessment of behavior problem children (Abikoff, Gittleman-Klein, & Klein, 1977; Eyberg & Pincus, 1999). Although behavior rating scales have several advantages, including low cost, ease of administration, administration versatility, and face validity, these instruments have been shown to be influenced by potentially biasing factors, including the respondent's global perceptions of the child and the respondent's mood (Edelbrock, 1988; Patterson, 1982). Direct observation is the most objective means of evaluating classroom disruptive behaviors and has been found to be the most ecologically valid means of identifying children with externalizing disorders in the classroom (Barkley, 1991). The principle disadvantage of direct observation is its relative cost. Multiple gating has been advocated as a cost-effective, stepwise screening mechanism for the identification of children at risk (Loeber, 1990). In this procedure, the first stage consists of a relatively inexpensive mass screening within a specified population, such as school children. Those children who pass the criteria for selection into the second stage continue through the screening process and undergo direct observation of the target classroom behavior. The multiple-gating procedure is ideal when resources are limited and the goal is to identify a high-risk group for treatment. The additive use of direct observation has been shown to enhance the predictive accuracy of a single screening gate using ratings of child behavior (Charlebois, Leblanc, Gagnon, & Larivee, 1994). In addition to its utility in identifying children for treatment eligibility, classroom observation is an important assessment method for use in monitoring changes that occur during treatment and evaluating treatment outcome. It facilitates objectivity during the repeated administrations necessary to assess the progress and effectiveness of treatment and the clinical significance of changes in targeted behavior (Eyberg, Boggs, & Algina, 1995).

REDSOCS

697

A number of psychometrically sound coding systems have been developed to assess children's school behavior. The Child Behavior Checklist-Direct Observation Form (DOF; Achenbach, 1986) was designed to record observations of children in classrooms and other group situations. The observer writes a narrative description of the child's ongoing behavior over a 10 min period, scores on-task behavior at the end of each minute, and then scores 96 problem items based on the 10 min of observation. The manual recommends that observations also be obtained on two control children on the same occasion and in the same situation. However, the system does not specify a method to gather the information on multiple children concurrently. The Behavioral Coding System (BCS), developed by Harris and Reid (1981), is an interval coding system designed to measure coercive behavior and aggression in classroom and playground settings. It yields data on eight behavioral categories, but offers information on a limited range of child disruptive classroom behaviors. The system does not assess academically related behavior, such as on-task behavior. Abikoff et al. (1977) developed a classroom observation code designed to discriminate between hyperactive and normal children. This observational system uses a modified time sampling procedure and consists of 14 observation categories. Notably, the average training reported for observers to reach performance criterion is 50 hours. It is important that researchers using observation systems consider not only the psychometric properties of individual systems, but also the attainability of the results. The original School Observation Coding System (SOCS; McNeil, Eyberg, Eisenstadt, Newcomb, & Funderburk, 1991) was designed to assess a young child's behavior in an age-appropriate classroom setting (e.g., preschool, Head Start, kindergarten classrooms) and sorts behavior within three behavioral domains: (a) appropriate versus inappropriate behavior; (b) compliant versus noncompliant behavior versus no command given; and (c) on-task versus off-task behavior versus not applicable. The first two domains were derived from the coding system used by Forehand and colleagues (Breiner & Forehand, 1981; Forehand et al., 1979), which was an interval sampling system in which the presence or absence of behavior in a 10 s interval was determined. The third, on-task, domain was based on the definition of academically engaged time proposed by Walker, Shinn, O'Neill, and Ramsey (1987). Walker and colleagues assessed academically engaged time via direct observation using a duration recording method by trained observers during two 15min reading or math periods. The SOCS used a modified time sampling system that combined the frequency coding of compliant and appropriate behavior with the duration coding of on-task behavior so that scores could be obtained on the three important types of disruptive behavior in young children. However, the original SOCS, like the Forehand et al. (1979) coding system, was not time-efficient. In these systems, during each 45-min observation session, each of three target children in the classroom was observed sequentially for alternating 10-s

698

JACOBS ET AL.

intervals followed by a 5- to 10-s marking time. Therefore, Funderburk and colleagues (1998) modified the coding system used by McNeil et al. (1991) to reduce marking time while maintaining the real-time coding of negative behavior occurrences. In this system, the nonoccurrences of disruptive behavior were marked at the end of each 10-s observation interval, with pauses in observation only at the end of each 60-s period to rotate focus to the next child. This change substantially decreased coder time per child. The REDSOCS is a further revision of the SOCS. It retains the Funderburk et al. (1998) time sampling system, which allows for coding any number of children in a classroom during an observation session, and its category code definitions have been refined. The REDSOCS yields three scores for each observed child: Percent Inappropriate Behavior (number of 10-s intervals in which the child exhibits an inappropriate behavior divided by the total number of intervals); Percent Noncompliant Behavior (number of intervals in which a teacher command is given and not obeyed by the child divided by the total number of intervals in which a teacher command is given); and Percent Off-task Behavior (number of intervals in which the child is off task divided by the total number of intervals in which the child is expected to be focused on a task). Definitions of the REDSOCS classroom behavior codes that characterize the three categories of disruptive behavior are tabled in the Appendix. Using the earlier versions of this system, McNeil et al. (1991) and Funderburk et al. (1998) found satisfactory interobserver agreement for the behavior codes. The purpose of this study was to examine the psychometric properties of the REDSOCS and to present reference point data. We predicted that the three disruptive behavior categories of the REDSOCS would demonstrate high interobserver agreement and would correlate positively with the Intensity Scale of the Sutter-Eyberg Student Behavior Inventory (SESBI) and the Total Score of the Revised Conners Teacher Rating Scale (CTRS-28), which are well-validated paper-and-pencil screening measures of attentional and conduct problems in the classroom. We predicted that the REDSOCS categories would also demonstrate convergent and discriminant validity in their relations to the individual subscales of the CTRS-28. Specifically, we expected the Inappropriate and Noncompliant Behavior categories of the REDSOCS, but not the Off-task Behavior category, to relate to the Conduct Scale. We expected the Off-task Behavior category of the REDSOCS to relate more strongly to the Inattention Scale of the CTRS-28 than to the Conduct Scale. No specific predictions were made regarding relations among the REDSOCS categories and the Hyperactivity and Hyperkinesis scales of the CTRS-28. We hypothesized that the clinic-referred children with reported school problems would differ from the nonreferred classroom control children on all three disruptive behavior categories of the REDSOCS, demonstrating the discriminative validity of the REDSOCS, whereas the clinicreferred children without reported school behavior problems would not differ from the control children. Finally, we expected the REDSOCS to demonstrate

REDSOCS

699

high classification rates for referred children identified according to the presence of school behavior problems based on either parent or teacher report.

Method Participants The participants were 233 preschool and kindergarten children in northcentral Florida. Fifty-one children were participants in a pretreatment assessment of clinic-referred children with identified behavior problems at home (Schuhmann, Foote, Eyberg, Boggs, & Algina, 1998). They were between the ages of 3 years, 0 months, and 6 years, 11 months, and met criteria for the DSM-III-R diagnosis of ODD. Children with major sensory or developmental impairments, such as blindness, autism, or mental retardation, were excluded from the clinic-referred sample. Using parents' dichotomous responses to the question, "Does your child have behavior problems at school?" from a demographic questionnaire completed at the intake evaluation, the clinic-referred children were classified into two groups: 34 children with reported behavior problems at school and 17 children without reported behavior problems at school. The clinic-referred children with school behavior problems included 29 boys and 5 girls, with a mean age of 4.7 years. The clinic-referred children without school behavior problems included 10 boys and 7 girls, with a mean age of 4.1 years. The demographic characteristics of the referred children are summarized in Table 1. The nonreferred children were randomly selected from the classrooms of

TABLE 1 DEMOGRAPHIC CHARACTERISTICS Clinic-referred With Reported School Problems (n = 34) Mean age (years)

4.7 (0.95)

Clinic-referred Without Reported School Problems (n = 17) 4.1 (0.97)

Sex (n = 51) Boys Girls

29 5

10 7

Mean SES

30.35 (13.01)

28.76 (13.01)

Race (n = 51) Caucasian Other

27 7

15 2

7 1

1 0

Medication (n = 9) Ritalin Thorazine Note.

SES = Socioeconomic status measured by the Hollingshead (1975) Four Factor Index.

700

JACOBS ET AL.

the referred children. During the pretreatment assessment, three randomly selected children from the classroom of each referred child, matched in age and sex to the referred child, were identified as nonreferred control children. To increase the sample size for the present study, nonreferred classroom control children who were obtained during the posttreatment assessment of the referred children were also included if they had not been a part of the referred child's pretreatment assessment. For this reason, the referred and nonreferred samples were not precisely matched. Further, because the majority of referred children were boys, the nonreferred sample included a comparable overrepresentation of boys. The nonreferred sample included 130 boys and 52 girls, with a mean age of 4.8 years. Other demographic and descriptive data were not collected on the nonreferred sample, although because they were drawn from the classrooms of the referred children, their socioeconomic status (SES) was assumed to be relatively similar to the referred children. Groups were not significantly different in age, F(2,229) = 2.43, p --< .05, or sex, F(2,229) = 2.99, p --< .05. The observers were graduate research assistants trained for approximately 4 hours per week, over the course of 5 weeks, to at least 80% interobserver agreement on each of the eight REDSOCS codes used to determine the three disruptive behavior categories. Coder training involved initial didactic instruction in observation coding procedures and code definitions, followed by 2 to 3 weeks of practice coding of videotaped classroom behavior. Once observers met 80% interobserver agreement on all categories during a 10-rain tape segment, they began live practice coding, in pairs, for approximately 2 weeks in the university laboratory preschool, until 80% interobserver agreement was attained in that setting. Measures REDSOCS. The REDSOCS is an interval coding system for recording disruptive classroom behaviors of preschool- and elementary-age children. Behaviors that are coded include inappropriate behavior, noncompliant behavior, and off-task behavior. The system allows for children to be coded alternately during an observation session and results in 10 rain of observation time per child during each of 3 sessions conducted within a 2-week period. Although 4 children were observed in each classroom in this study, any number of children may be observed using this s y s t e m - - f r o m 1 child to the entire classroom. Observational data are collected in 10-s intervals, and children are observed in l-rain rotations throughout each observation session. SESBI. The SESBI (Eyberg, 1992) is a 36-item teacher rating scale of conduct problem behaviors at school for children between the ages of 2 and 16. The Intensity Scale measures the frequency of children's problem behaviors on a 7-point scale (1 = never to 7 = always). The Problem Scale measures the degree to which the child's behaviors are problematic for the teacher on a yes-no scale. Cronbach's alpha was found to be .97 for the Intensity Scale and .96 for the Problem Scale (Bums & Owen, 1990). High internal consis-

REDSOCS

701

tency, temporal stability over a 7-day period, and interteacher agreement have been reported for the two scales (Funderburk & Eyberg, 1989), and convergent and discriminant validity have been demonstrated (Schaughency, Seeley, Talarico, & Jackson, 1990) with the CBCL-DOF. CTRS-28. The CTRS-28 (Goyette, Conners, & Ulrich, 1978) is a shorter version of the original 39-item Conners Teacher Rating Scale (Conners, 1969). Both versions of the teacher rating scale were designed to identify hyperactive children. The CTRS-28 asks teachers to indicate the degree to which a child exhibits each of the 28 listed symptoms on a 4-point scale (0 = not at all to 3 = very much). The CTRS-28 has been standardized for children ages 3 to 17 (Goyette et al.). Interparent agreement ranges from .46 on the Psychosomatic factor to .57 on the Conduct Problem factor. Parent-teacher agreement ranges from .33 for the Conduct Problem factor to .49 for the Hyperkinesis factor. High stability coefficients have been reported for the factor scores (Edelbrock, Greenbaum, & Conover, 1985), and the CTRS-28 has shown significant correlations with the original Conners Parent Rating Scale (Goyette et al.). Procedure

During a pretreatment assessment, consent to contact the school was obtained from the parent of each referred child. The study coordinator then contacted the director or principal of the school of each clinic-referred child to obtain permission to contact the child's teacher and collect data on the clinic-referred child and three classroom control children. The Institutional Review Board determined that informed consent was not required for the nonreferred control children because they were identified to study personnel only by their first names. After the principal's or director's permission was obtained, the graduate student observers were informed of the age and sex of the clinic-referred, target child and the name of the school and teacher to contact. The observers contacted the teacher to schedule the observations and to ask for a list of the first names of all children in the classroom who were the same age and sex as the target child. From the list of names, the study coordinator chose 4 children, including the target child and 3 randomly selected control children. In this way, the observers were kept uninformed as to which of the 4 children was the clinic-referred, target child. During a structured learning situation, the 4 children were observed in the classroom for 1 min at a time in an alternating fashion until each child had been observed for a total of 10 min per day. Classroom behavioral observations were conducted on three different days within a 2-week period. On 1 of the 3 days of observation, two observers simultaneously coded each child's behavior using a dualjack tape recorder that played 10-s timed prompts and signaled the observers to focus on the next child. Reliability was calculated across all 4 children on these visits. The teachers were paid $20 to complete the SESBI and the CTRS-28 on each of the 4 children who were observed. Scores for the control and referred samples on these measures are shown in Table 2.

702

JACOBS ET AL.

TABLE 2 MEAN SCORES ON TEACHER RATING SCALES FOR REFERRED AND NONREFERRED SAMPLES Clinic-referredWith School Problems (n = 34) Scale SESBI Intensity Problem CTRS-28 Conduct Hyperactivity Inattention Hyperkinesis Total

Note.

Clinic-referred Without School Problems (n = 17)

Nonreferred Classroom Controls (n = 182)

M

(SD)

M

(SD)

M

(SD)

149.88 a 15.54 ~

(42.68) ( 12.21 )

93.31 b 5.00 ~

(30.76) (6.95)

87.60 b 4.79 b

(43.1 l) (7.50)

12.37 a 14.26" 11.33" 17.89 ~ 45.07 ~

(6.12) (4.65) (6.61) (7.05) ( 16.85 )

5.86 b 8.64 b 8.21 ",h 8.71 b 24.86 b

(5.74) (6.12) (4.42) (6.60) ( 13.40)

4.78 b 6.00 ~ 5.77 b 7.07 b 19.75 b

(5.18) (5.65) (5.05) (6.56) ( 15.95)

SESBI = Sutter-Eyberg Student Behavior Inventory. CTRS-28 = Revised Conners Teacher Rating Scale. ab G r o u p s with different superscripts differ significantly at p < .05.

Results Descriptive Data Prior to analysis of the REDSOCS data, the scores from the teacher rating scales were examined to provide support from the teacher's perspective for the criterion validity of the groups of referred children defined as having or not having school behavior problems based only on parent report. Table 2 shows that for the 34 children in the clinic-referred sample with parentreported school behavior problems, their raw score on the SESBI Intensity Scale ranged from 67 to 246, with a mean score of 150 (SD = 43). Thus, this group shows high variability around a mean score in the clinically significant range for preschoolers (Funderburk & Eyberg, 1989). Raw scores on the CTRS-28 Total Scale were similar and ranged from 12 to 84, with a mean of 45 (SD = 17). For the 17 children in the clinic-referred sample without parent-reported school behavior problems, the SESBI Intensity Scale raw scores ranged from 52 to 163, with a mean score of 93 (SD = 31), which is in the normal range of scores (Funderburk & Eyberg, 1989). Raw scores on the CTRS-28 Total Scale ranged from 7 to 51, with a mean of 25 (SD = ! 3). Similarly, in the nonreferred sample, the scores showed wide variability, with raw score means in the normal range (SESBI M = 88, SD = 43; CTRS-28 M = 20, SD = 16). Differences on the two teacher rating scales between each of the referred groups and the nonreferred group of children were examined with one-way analyses of variance (ANOVAs). Significant differences emerged on both the

REDSOCS

703

SESBI Intensity Scale, F ( 2 , 2 2 9 ) = 24.30, p < .01, and the CTRS-28 Total Scale, F(2,229) = 29.42,p < .01. Scheffe post hoc analyses showed that, on both teacher-rating scales, the referred children with reported school behavior problems differed significantly from the nonreferred children. There were no differences on either scale between the referred children without parent-reported school problems and the nonreferred children. Table 2 also shows the comparisons on the subscales of the SESBI and CTRS-28 among the three groups.

Reliability Interrater agreement for the REDSOCS, as calculated by percent agreement on occurrences for each category code, ranged from 70% for the off-task behavior code to 99% for the appropriate behavior code (see Table 3). Kappa coefficients (Cohen's kappa; Fleiss, 1981) for all codes ranged from .78 for the noncompliant behavior code to .94 for the no-task-given code (see Table 3).

Convergent and Discriminant Validity Correlations of the REDSOCS categories with teachers' ratings on the CTRS-28 and SESBI scales were used to evaluate the convergent and discriminant validity of the REDSOCS. It is important to note that only the 182 children in the nonreferred sample were included in these analyses. These children had been randomly selected from demographically similar classmates, and their scores on the behavior measures were normally distributed. Inclusion of the referred children, selected specifically because of their behavior problems and arbitrarily representing 25% of the total sample of classroom children observed, would produce a bivariate distribution of behavior scores, not representative of the typical classroom. Correlational analyses assume a normal distribution of scores, and the inclusion of distinct groups in analyses of convergent and discriminant validity produce artificially inflated validity coefficients. As shown in Table 4, all categories of the REDSOCS correlated significantly with the raw scores of the CTRS-28 Total Scale and the SESBI IntenTABLE 3 INTEROBSERVERRELIABILITYor REDSOCS CATEGORYCODES Category Codes Appropriate behavior Inappropriate behavior Compliant behavior Noncompliant behavior No command given On-task behavior Off-task behavior No task expected

Percent Agreement

Kappa

99% 74% 75% 78% 98% 93% 70% 86%

.85 .83 .88 .78 .90 .86 .80 .94

Note. Reliability averaged across 42 reliability sessions.

704

JACOBS ET AL.

TABLE 4 CORRELATIONS BETWEEN R E D S O C S CATEGORYSCORES AND TEACHER RATING SCALE SCORES REDSOCS Behavior Category Scale

Inappropriate

Noncompliant

Off-task

SESB1 intensity Problem

.38* .23*

.26* .23*

.26* .20

CTRS-28 Conduct Hyperactivity Inattention Hyperkinesis Total

.30* .39* .25" .41 * .34*

.22* .26* .25" .29* .25*

.16 .21 .29" .29* .22*

Note.

REDSOCS = Revised Edition of the School Observation System. SESBI - SutterEyberg Student Behavior Inventory. CTRS-28 = Revised Conners Teacher Rating Scale. * p < .002.

sity Scale. For these analyses, the experimentwise error rate was controlled using the Dunn-Bonferroni correction, and only correlations at p < .002 were considered significant. We found significant positive correlations between the Conduct Scale of the CTRS-28 and both the Inappropriate Behavior category, r(l 81) = .30, p < .001, and the Noncompliant Behavior category, r(181) = .22, p < .001, but not the Off-task Behavior category of the REDSOCS, r(181) = .16, p = .023. We also found significant positive correlations between the Hyperactivity Scale of the CTRS-28 and both the Inappropriate Behavior category, r(181) = .39, p < .001, and the Noncompliant Behavior category, r(181) = .26, p < .001, but not with the Off-task Behavior category of the REDSOCS, r( 181) = .21, p < .005. Finally, we found significant positive correlations between the Inattention Scale of the CTRS-28 and all three categories of the REDSOCS, with the highest correlations occurring for the Off-task Behavior category, r(181) = .29, p < .001. Examination of the Hyperkinesis Scale of the CTRS-28 revealed significant positive correlations with all three categories of the REDSOCS, with the highest correlations occurring for the Inappropriate Behavior category, r( 181) = .41, p < .001.

Discriminative Validity An ANOVA was conducted to examine differences in REDSOCS scores and membership in the groups of referred children with and without reported school behavior problems and the nonreferred classroom control children. A significant group membership effect was found for the Inappropriate Behavior category, F(2,229) = 8.28,p < .01, the Noncompliant Behavior category, F ( 2 , 2 2 9 ) = 3.39, p < .05, and the Off-task Behavior category, F(2,229) =

REDSOCS

705

TABLE 5 PERCENT OCCURRENCE OF REDSOCS SCHOOL BEHAVIOR PROBLEM CATEGORIES IN REFERRED AND NONREFERRED SAMPLES

Clinic-referred With School Problems (n=34)

Clinic-referred Without School Problems (n= 17)

Nonreferred Classroom Controls (n= 182)

REDSOCS Category

M

SD

M

SD

M

SD

Percent inappropriate Percent noncomply Percent off task

7.4a 27.3a 22. Ia

1.3 5.3 2.9

4.7a,b 13.7a,b 13.3a,b

1.2 4.8 3.1

4.0b 17.1b 16.0b

0.3 1.6 1.1

Note.

REDSOCS= Revised Edition of the SchoolObservation System. abGroups with different superscripts differ significantly at p < .05.

3.18, p < .05. Mean scores for each group are shown in Table 5. Scheffe post hoc analyses showed that, for each REDSOCS category, the nonreferred sample differed significantly from the referred sample with reported school behavior problems, but not from the referred sample without reported school behavior problems. Among the referred children, a logistic regression was performed to examine how well these children could be classified by the three REDSOCS categories into those with and without parent-reported school behavior problems. The overall classification accuracy rate was 70%, with 80% of the referred children with school behavior problems correctly classified and 53% of the referred children without school behavior problems correctly classified. Individual REDSOCS categories did not contribute significantly to the model. However, the entire model accounted for 20% of the variance (Nagelkerke R Square = .20). In addition, a second logistic regression was performed to examine how well children could be classified by the three REDSOCS categories when they were grouped according to the presence of significant teacher-reported behavior problems at school. Children were grouped according to their SESBI score, and the SESBI cutoff score of 151 was used to divide the referred children into two groups. The overall classification rate was 71%, with 74% of the referred children with significant levels of teacherreported school behavior problems correctly classified and 68% of the referred children with normal levels of teacher-reported behavior problems at school correctly classified. As shown in Table 6, the Off-task category was the only individual REDSOCS category that was significant in the model. The entire model accounted for 46% of the variance. Reference Point Data

Among the nonreferred control children, the mean percent occurrence of disruptive classroom behavior was calculated to provide a reference point for

706

JACOBS ET AL.

TABLE 6 LOGISTIC REGRESSION CLASSIFYINGREFERRED CHILDRENWITH AND WITHOUT SIGNIFICANT TEACHER-REPORTED SCHOOLBEHAVIORPROBLEMSUSING R E D S O C S CATEGORIES REDSOCS Categories

B

SE B

Wald

Off-task Noncompliance Inappropriate Behavior Nagelkerke R 2

.13 .03 .01 .46**

.06 .02 .l 3

5.06* 2.27 0.14

*p < .05. **p < .01.

typical classroom behavior in young children. Results showed that Inappropriate Behavior occurred in an average of 4% of the observation intervals, Off-task Behavior occurred in an average of 16% of the observation intervals in which children were expected to be on task, and Noncompliant Behavior occurred in an average of 17% of the observation intervals in which there was an opportunity to comply.

Discussion Examination of the psychometric properties of this revision of the School Observation Coding System (Funderburk et al., 1998; McNeil et al., 1991) with preschool and kindergarten children supports its suitability for use with young children in a variety of classroom settings. Our results indicate that the behavioral codes defining noncompliant, inappropriate, and off-task behavior in the REDSOCS can be coded reliably and can distinguish referred children with school behavior problems from other referred and nonreferred children. To assess the discriminative validity of the REDSOCS categories, the scores for the nonreferred sample were compared with the scores for the clinicreferred samples of children with and without parent-reported school behavior problems. Early-onset conduct-disordered behavior normally begins in a single setting such as the home and expands to a greater number of social situations, including the school (Eyberg et al., 1998). For this reason, preschoolers referred for treatment of severely disruptive behavior may or may not present disruptive behavior problems in the classroom. In the present study, we defined our school behavior problem group by parent report of school problems at the time of treatment referral. This parent-report criterion was validated by teacher rating scale data of oppositional and hyperactive behaviors in the classroom. Specifically, the clinic-referred children with parentreported school behavior problems differed from the classroom control children on both the SESBI and the CTRS-28, whereas the clinic-referred children without reported school behavior problems were not distinguishable from the classroom controls. The REDSOCS categories demonstrated discriminative validity as pre-

REDSOCS

707

dicted. The clinic-referred children with reported school behavior problems showed significantly more disruptive classroom behavior than their nonreferred classroom peers on all categories of the REDSOCS, whereas the clinic-referred children without reported school hehavior problems showed no significant differences from their nonreferred classmates on any REDSOCS category. Also, the REDSOCS scores correctly classified the majority of clinic-referred children for the presence or absence of school behavior problems at intake whether children were grouped according to parent or teacher report of school behavior. Notably, the REDSOCS categories were especially powerful in classifying groups based on teacher rating scales of child behavior. These findings provide compelling evidence to suggest that a 30-min sample of child behavior is adequate to discern clinically meaningful behavioral differences using REDSOCS. As noted by Abikoff, Gittleman, and Klein (1980), however, discrimination between groups may lessen upon replication, and it will be important to replicate this initial demonstration of discriminative validity within a second sample of children. To study the sensitivity of the REDSOCS further, we examined the relations among the behavioral categories and teacher ratings of oppositional and attentional problems for the nonreferred children. Within this sample, all categories of the REDSOCS were positively and significantly related to both the Intensity Scale of the SESBI and the Total Score of the CTRS-28. These findings suggest that meaningful distinctions in classroom behaviors associated with the disruptive behavior disorders can also be validly measured within the nonclinical range of REDSOCS scores. Measures that demonstrate sensitivity within the normal range may be particularly valuable for assessing treatment outcome (Eyberg & Pincus, 1999). We also examined the convergent and discriminant validity of the individual categories of the REDSOCS. We found that both the Inappropriate and Noncompliant Behavior categories, but not the Off-task Behavior category, were correlated with the Conduct Scale of the CTRS-28, as predicted. The Conduct Scale contains teacher-rated items assessing descriptors of behaviors such as "acts smart" "uncooperative" and "denies mistakes/blames others," which are reflected in both the Inappropriate and Noncompliant Behavior categories of the REDSOCS. These associations across measurement methods are strong evidence of the convergent validity of these categories, which together capture the defining behaviors of young children with ODD. Conversely, the lack of significant relations between the Off-task Behavior category of the REDSOCS and the CTRS-28 Conduct scale demonstrates the discriminant validity of the Off-task Behavior category. The Off-task Behavior category of the REDSOCS correlated positively with the Inattention Scale of the CTRS-28, as predicted. Items on the Inattention scale, such as "restless" "squirmy" and "always on the g o " represent observable behaviors similar to those coded as off-task in REDSOCS, and the significant association between these measures demonstrates convergent validity of the Off-task Behavior category. The positive association

708

JACOBS ET AL.

also underscores the ability of the REDSOCS to assess attentional problems in preschool children and its potential for use in early identification and monitoring of the range of disruptive behavior that may emerge during the preschool period. Correlations between the SESBI Problem Scale and the REDSOCS categories provide insight into the types of behavior that teachers find most annoying. The Problem Scale was positively associated with both the Inappropriate Behavior category, which measures behaviors such as whining, yelling, and tantruming, and the Noncompliant Behavior category, but was not associated with the Off-task Behavior category. This pattern suggests that although teachers find inappropriate behavior and noncompliance more consistently annoying, they find off-task behavior less problematic for them. It may be that teachers perceive a lack of intentionality in off-task behavior, which they experience as less bothersome. The data provided on the randomly selected, nonreferred young children from 51 separate preschool and kindergarten classes in 44 different schools in north-central Florida provides a reference point for observations of young children, but should not substitute for within-classroom controls in the individual case. The high variability in teacher rating scale data has called into question the meaningfulness of broad-based normative school behavior data. Perhaps even more so than rating scale data, observational data are dependent on contextual events, and children's school behavior is highly situationspecific. The classroom environments of preschoolers are notably different among different classrooms, different schools, from day to day, and from one activity to another in the same classroom. More highly structured classrooms may provide more opportunity for the occurrence of off-task behavior, the number and type of commands issued by a teacher may affect the compliance rate, and a teacher's limit-setting and consistency in follow-through may affect the rate of inappropriate behavior. For these reasons, experimental control children from within the classroom are important in establishing a child's behavioral deviation. By using alternate interval observations, REDSOCS permits the concurrent observation of classroom control children, and allows an ecologically valid picture of the child to emerge. In summary, the initial evidence of reliability and construct validity demonstrated in this sample of preschoolers suggests that the REDSOCS is a promising system for use in the second stage of identifying children for treatment of school behavior problems. Further, its sensitivity to behavior problem variability within the normal range and its ability to discriminate between conduct-disordered and normally disruptive school behavior suggest that it may be useful for targeting children's classroom behaviors that require intervention and for evaluating the effects of behavioral interventions. However, it will be important to evaluate the stability of the REDSOCS over time without intervention. Based on the strong stability of the earlier version of the system reported by McNeil and colleagues (1991), we expect that the REDSOCS will also demonstrate stability over time. Another important next step

REDSOCS

709

for the REDSOCS is to assess its psychometric properties with older, schoolage samples of children.

Appendix Revised Edition of the School Observation Coding System Category Code Definitions Category 1: Noncompliant Behavior Compliant. The target child obeys, begins to obey, or attempts to obey within 5 s of a direct or indirect teacher command. The command can be directed toward the target child individually or to a group of children that includes the target child. If a command is given during the last half of a 10 s coding interval, the observer will continue to watch for child compliance to that command for 5 s and will score the outcome in the interval in which the command is given. Noncompliant. The target child makes no movement toward obeying a direct or indirect teacher command during a 5-s period following the command. No Command Given. Coded when no command was issued to the child during the 10-s observation interval.

Category 2: Inappropriate Behavior Appropriate Behavior. The absence of all Inappropriate Behavior for the entire 10 s interval.

Inappropriate Behavior. The following categories of behavior are scored as Inappropriate/Oppositional Behavior because they may be annoying or disruptive to the target child, the teacher, or other children: • Whining: Coherent words uttered by the child in a slurring, nasal, highpitched voice. • Crying: Inarticulate utterances of distress (e.g., audible weeping) that may or may not be accompanied by tears. • Yelling: Loud screeching, screaming, or shouting. The sound must be loud enough so that it is clearly above the intensity of normal indoor conversation. Yelling is not coded during outdoor recess observations. • Destructiveness: Behaviors during which the child damages or destroys an object or threatens to damage an object. Do not code destructiveness if it is appropriate within the context of the play situation (e.g., ramming cars in a car crash). • Aggressive Behavior: Examples include fighting, kicking, slapping, hitting, grabbing an object roughly from another person, or threatening to do any of the preceding. • Negativism: Verbal or nonverbal behavior expressing a negative attitude. Negativism may be scored when the child makes a neutral comment that is delivered in a tone of voice that conveys an attitude of "don't bother

710

•

•

•

•

•

•

J A C O B S ET AL.

me." Negativism may be expressed in a derogatory, uncomplimentary, or angry manner. Also included are exaggerated defeatist statements such as "I give u p " contradictions of another person, and teasing or mocking behaviors or verbalizations. "Pouting" facial expressions are included in this category. Self-Stimulation: Repetitive physical movements (involving only the child's body and not other objects) that may be harmful and that interfere with the child's ability to attend or complete a task. Examples include head banging, thumb sucking, and masturbation. Demanding Attention: Includes inappropriate verbal or nonverbal requests for attention from the teacher or other students (e.g., "Call on me! Call on me! Call on me!"). Examples include tugging on the teacher's sleeve, tapping a neighbor on the shoulder, waving arms in the air, and passing notes to another child. Disruptive Behavior: Any physically active or repetitive behavior that is or may become disruptive to others or interfere with the target child's ability to attend or complete a task. Examples include kicking a child's chair repeatedly, drumming on a table loudly, clowning, making funny noises, teasing, or spinning a pencil on a desk. Talking Out of Order: Any talking when the class has been instructed to be silent unless called on to speak. This includes situations in which a "classroom rule" exists that silence is to be maintained (i.e., the teacher does not have to give the instruction explicitly--the expectation for silence is sufficient). Examples include whispering to a neighbor, answering a question directed to someone else, calling out to another child, and talking, singing, or humming to oneself. Being Out of Area: Coded when the target child, without permission, leaves the area to which she or he is assigned. Examples include standing up when the rest of the class is seated, leaving the desk, approaching the teacher without permission, or playing with a toy that is not in the child's assigned work area the child is supposed to be in. The behavior must be appropriate for the context or classroom norms (e.g., in some classrooms children are allowed to walk to the teacher's desk to obtain help with an assignment). Cheating: Child borrows another child's work when such behavior is clearly not allowed. Examples include looking at another child's paper during a spelling quiz and copying another child's work.

Category 3: Off'task Behavior On task. The child is considered to be on task if she or he is (a) attending to the material and the task, (b) making appropriate motor responses (e.g., writing, computing, pasting), or (c) asking for assistance (where appropriate) in an acceptable manner. Interacting with the teacher or classmates about academic matters or listening to directions or instructions are considered to be on-task behaviors. To be coded on task the child must remain on task for the entire 10-s interval.

REDSOCS

711

Offtask. Coded if at any point during the 10-s interval the child engages in a behavior that does not meet the definition of on-task behavior. Examples of off-task behavior include failing to attend to or work on the assigned task, breaking classroom rules (getting out of seat, talking out, disturbing others, etc.), resting head on desk passively when there is a task to complete, and staring blankly away from the task or materials (i.e., daydreaming). If the child is in time-out during the observation interval, she or he is automatically coded as off task. Not Applicable. Coded when there is no readily identifiable task that the child is expected to perform. Examples of Not Applicable activities include free play time and unstructured recess time.

References Abikoff, H., Gittleman, R., & Klein, D. (1980). Classroom observation code for hyperactive children: A replication of validity. Journal of Consulting and Clinical Psychology, 48, 555-565. Abikoff, H., Gittleman-Klein, R., & Klein, D. (1977). Validation of a classroom observation code for hyperactive children. Journal of Consulting and Clinical Psychology, 45,772783. Achenbach, T. M. (1986). Manual for the Child Behavior Checklist/4-18 & 1991 Profile. Burlington, VT: Department of Psychiatry, University of Vermont. American Psychiatric Association. (1994). Diagnostic and statistical manual of mental disorders (4th ed.). Washington, DC: Author. Barkley, R. (1991). Diagnosis and assessment of attention deficit-hyperactivity disorder. Comprehensive Mental Health Care, 1,27-43. Breiner, J., & Forehand, R. (1981). An assessment of the effects of parent training on clinic referred children's school behavior. Behavioral Assessment, 3, 31-42. Burns, G., & Owen, S. (1990). Disruptive behavior in the classroom: Initial standardization data on a new teacher rating scale. Journal of Abnormal Child Psychology, 18, 515-525. Charlebois, P., LeBlanc, M., Gagnon, C., & Larivee, S. (1994). Methodological issues in multiple-gating screening procedures for antisocial behaviors in elementary students. Remedial and Special Education, 15, 44-54. Conners, C. K. (1969). A teacher rating scale for use in drug studies with children. American Journal of Psychiatry, 126,884-888. Costello, E. J. (1989). Development in child psychiatric epidemiology: Introduction. Journal of the American Academy of Child and Adolescent Psychiatry, 28,836-841. Edelbrock, C. S. (1988). Informant reports. In E. S. Shapiro & T. R. Kratochwill (Eds.), Behavioral assessment in schools: Conceptual foundations and practical applications (pp. 351383). New York: Guilford Press. Edelbrock, C., Greenbaum, R., & Conover, N. C. (1985). Reliability and concurrent relations between the teacher version of the Child Behavior Profile and the Conners Revised Teacher Rating Scale. Journal of Abnormal Child Psychology, 13,295-303. Eyberg, S. M. (1992). Parent and teacher behavior inventories for the assessment of conduct problem behaviors in children. In L. VandeCreek, S. Knapp, & T. L. Jackson (Eds.), Innovations in clinicalpractice: A source book (Vol. 11). Sarasota, FL: Professional Resource Press. Eyberg, S. M., Boggs, S. R., & Algina, J. (1995). Parent-Child Interaction Therapy: A psychosocial model for the treatment of young children with conduct problem behavior and their families. Psychopharmacology Bulletin, 31, 83-91.

712

JACOBS ET AL.

Eyberg, S. M., & Pincus, D. (1999). Eyberg Child Behavior Inventory and Sutter-Eyberg Student Behavior hn,entorv: Professional Manual. Odessa, FL: Psychological Assessment Resources. Eyberg, S. M., Schuhmann, E. M., & Rey, J. (1998). Child and adolescent psychotherapy research: Developmental issues. Journal of Abnormal Child Psychology, 26, 71-82. Fleiss, J. L. (1981 ). Statistical methodsjbr rates and proportions. New York: Wiley. Forehand, R., Sturgis, E., McMahon, R., Aguar, D., Green, K., Wells, K., & Breiner, J. (1979). Parent behavioral training to modify child noncompliance: Treatment generalization across time and from home to school. Behavior Modification, 3, 3-25. Funderburk, B. W., & Eyberg, S. M. (1989). Psychometric characteristics of the Sutter-Eyberg Student Behavior Inventory: A school behavior rating scale for use with preschool chiIdren. Behavioral Assessment, 11,297-313. Funderburk, B. W., Eyberg, S. M., Newcomb, K., McNeil, C. B., Hembree-Kigin, T., & Capage, L. (1998). Parent-child interaction therapy with behavior problem children: Maintenance of treatment effects in the school setting. Child and Family Behavior Therapy, 20, 17-38. Goyette, C. H., Conners, C. K., & Ulrich, R. F. (1978). Normative data on Revised Conners Parent and Teacher Rating Scales. Journal of Abnormal Child Psychology, 6, 221-236. Harris, A. H., & Reid, J. B. (1981). The consistency of a class of coercive child behaviors across school settings for individual subjects. Journal of Abnormal Child Psychology, 9,219-227. Hollingshead, A. B. (1975). Four factor index of social status. Unpublished manuscript, Yale University, New Haven, CT. Kazdin, A. E., Siegel, T. C., & Bass, D. (1990). Drawing on clinical practice to inform research on child and adolescent psychotherapy: Survey of practitioners. Professional Psychology: Research and Practice, 21, 189-198. Loeber, R. (1990). Development and risk factors of juvenile antisocial behavior and delinquency. Clinical Psychology Review, 10, 1-41. McNeil, C. A., Eyberg, S. M., Eisenstadt, T. H , Newcomb, B. K., & Funderburk, B. W. (1991). Parent-child interaction therapy with behavior problem children: Generalization of treatment effects to the school setting. Journal of Clinical Child Psychology, 20, 140-151. Patterson, G. R. (1982). Coercive family processes. Eugene, OR: Castalia Press. Schaughency, E. A., Seeley, J., Talarico, B. N., & Jackson, M. J. ( 1990, November). Multitraitmultimethod evaluation of the Sutter-Eyberg Student Behavior Inventory with a clinic referred sample. Paper presented at the annual meeting of the Association for Advancement of Behavior Therapy, San Francisco. Schuhmann, E., Foote, R., Eyberg, S. M., Boggs, S. R., & Algina, J. (1998). Efficacy of parentchild interaction therapy: Interim report of a randomized trial with short-term maintenance. Journal of Clinical Child Psychology, 2 7, 34-45. Walker, H. M., Shinn, M. R., O'Neill, R. E., & Ramsey, E. (1987). A longitudinal assessment of the development of antisocial behavior in boys: Rationale, methodology, and first year results. Remedial and Special Education, 8, 7-16. RECEIVED: February 24, 1999 ACCEPTED: June 28, 2000