Research and Evaluation into the Curry/Samara Model Of Curriculum ...

Research and Evaluation into the Curry/Samara Model Of Curriculum, Instruction, and Assessment.

by

Michael L. Connell, Ph.D. University of Houston Curriculum and Instruction CUIN 5872 Houston, TX 77069-5872 [email protected]

REPORT 11/04/2003 Prepared in cooperation with The Aldine Independent School District and The Curriculum Project

Table of Contents

Introduction and Overview

3

Population and Data Sources

4

A-B Comparisons: Findings and Discussion

5

Student-Problem Time Series Analysis Findings and Discussion

21

Student-Problem Growth Curve Analysis Findings and Discussion

39

Conclusions

44

Appendices A) Demographics of schools used

47

B) Time Series Analysis in Chart Form

50

C) Student-Problem Growth Curve Analysis

55

Appendix B & C are omitted from this PDF file to decrease memory requirements, but are available upon request, please call 800.867.9067 for more information.

2

Introduction and Overview

The research presented in this report addresses two foundational areas concerning the Curry/Samara Model of Curriculum, Instruction, and Assessment. The first may be classified as providing empirical, replicable and quantitative evidence that adoption and use of the Curry/Samara Model enables positive impacts upon student classroom performance. The second, and in many ways more interesting of these areas, is an attempt to describe some of the positive benefits accruing from continuing use of the Curry/Samara Model over time - particularly as reflected in long-term patterns of student achievement and curriculum implementation. For this report two parallel elementary schools were carefully selected from within Aldine Independent School District (AISD)- one of which (Control Elementary1) was described by district administrators and personnel as not significantly participating in the Curry/Samara Model and the other (Treatment Elementary) which was classified as participating at a high level. These schools served as the basis for both strands of the analysis. The primary research concerns were addressed via a two-part research and evaluation scheme involving two mutually supporting research methodologies - Analysis of Variance (ANOVA) and Student-Problem Analysis (S-P Chart). The use of the

1

This report has modified the actual school names to ensure confidentiality as requested by Aldine Independent School District.

3

Analysis of Variance methodology allows statistically significant claims to be made regarding observable differences between these two groups as reflected in their student performance scores. The use of the Student-Problem Chart allows in-depth investigation of differences in student understanding, as measured by consistency of item response, and objective mastery - as well as providing insight into the degree to which teachers were able to successfully align their instruction to the objectives evaluated. Together these two methodologies allow for comprehensive A vs. B comparisons to be made and provide a series of pedagogical and curricular insights into the nature of the differences observed addressing the identified research concerns.

Populations and Data Sources The schools chosen for participation in this study were carefully chosen to match on a number of important variables including gender, Title 1A status, educational “atrisk” classification, etc. Although in educational research a perfect match is rarely possible, these schools were found to be comparable in most areas. Furthermore, in several areas such as number of students classified as being educationally at-risk and number of students classified as limited English proficient (LEP), the control school was initially in a more favorable position. A series of frequency charts providing matching demographics of these schools is provided in Appendix A. Data on student mastery from the state mandated end of year Texas Assessment of Academic Skills (TAAS) for 1997, 1998, 1999, 2000, 2001, and 2002 were used for each

4

analysis2. The data provided by the district was in its raw form as directly received from the state ensuring its integrity. From this raw data a Statistical Package for the Social Sciences (SPSS) file was constructed for each year, mapping each element of the data line vector with their appropriate variables and labels, as described in the document Data File Format with Student Item Analysis3. These six data sets were then used as the source for demographic analysis and school selection, as well as the Analysis of Variance studies, and Student Problem Chart development.

A-B Comparisons Findings and Discussion To address the first research area, concerning empirical, replicable and quantitative evidence supporting adoption and use of the Curry/Samara Model, a series of Analysis of Variance (ANOVA) studies were conducted. These ANOVA studies compare student data by grade, on the number of TAAS objectives mastered each year in the core content areas of reading, writing, and mathematics. This data was drawn from two parallel elementary schools - one of which was identified as not significantly participating in the Curry/Samara Model (control) while the other was classified by district administrators as participating at a high level (treatment). This section will now report each of the parallel series of Analysis of Variance for the years 1997, 1998, 1999, 2000, 2001, and 2002. In these analyses, and all subsequent plots, the number of objectives mastered in the area of Writing is indicated by

2

3

A special thanks is offered to Area Superintendent Margarita Byrum and Monica Uphoff, administrative assistant of data analysis for their help and support in providing the six years of TAAS data utilized in this report. http://www.tea.state.tx.us/student.assessment/reporting/formats/

5

W_#MAST, the number of objectives mastered in the area of Reading is indicated by R_#MAST, and the number of objectives mastered in the area of Mathematics is indicated by M_#MAST.

1997 ANOVA of Core TAAS Objectives

Descriptives

N W_#MAST

Control Treatment Total R_#_MAST Control Treatment Total M_#_MAST Control Treatment Total

110 114 224 225 243 468 225 243 468

Mean 2.09 1.85 1.97 3.71 3.70 3.70 7.924 7.597 7.754

Std. Deviation Std. Error 1.246 .119 1.459 .137 1.361 .091 2.383 .159 2.772 .178 2.590 .120 4.0925 .2728 5.2075 .3341 4.7024 .2174

95% Confidence Interval for Mean Lower Bound Upper Bound Minimum 1.86 2.33 0 1.58 2.12 0 1.79 2.15 0 3.40 4.02 0 3.35 4.05 0 3.47 3.94 0 7.387 8.462 .0 6.939 8.255 .0 7.327 8.181 .0

Maximum 3 3 3 6 6 6 12 12 12

ANOVA

W_#MAST

R_#_MAST

M_#_MAST

Between Groups Within Groups Total Between Groups Within Groups Total Between Groups Within Groups Total

Sum of Squares 3.225 409.556 412.781 .029 3131.687 3131.716 12.549 10314.193 10326.741

df 1 222 223 1 466 467 1 466 467

Mean Square 3.225 1.845

F 1.748

Sig. .187

.029 6.720

.004

.948

12.549 22.133

.567

.452

It should be noted that although none of these observed differences in student mastery of objectives are statistically significant, in each case the control school is scoring slightly better. This advantage at the time of the initial data would certainly lend credence to the argument that any future growth in student performance could be attributed to the continuing implementation and use of the Curry/Samara Model. At a 6

bare minimum it illustrates the beginning comparability of the two schools chosen for use in this study. The following means plots illustrate these observed differences.

1997 Means Plots 2.2

2.1

2.0

1.9

1.8 Control

CAMPUS#

Treatment Mean of W_#MAST

4.4

4.2

4.0

3.8

3.6

3.4

3.2

3.0 Control

CAMPUS#

Treatment Mean of R_#_MAST

7

8.0

7.9

7.8

7.7

7.6

7.5 Control

CAMPUS#

Treatment Mean of M_#_MAST

These graphs clearly show that at the time the baseline for this research was drawn there was not an unfair advantage given to the treatment group by selection of the schools. Indeed, the treatment school is at a noticeable disadvantage in each area measured. As earlier indicated this is also reflected in the number of students classified as at-risk and Limited English Proficient (LEP). When one considers the initial disadvantages demonstrated by the student performance at the treatment school it makes the later growth illustrated in the following ANOVA's all the more impressive.

8


Descriptives 95% Confidence Interval for Mean N W_#MAST

R_#_MAST

M_#_MAST

Control

113

Mean 2.10

Std. Deviation 1.172

Std. Error .110

Lower Bound 1.88

Upper Bound 2.32

Minimum 0

Maximum 3

Treatment

146

2.18

1.258

.104

1.97

2.38

0

3

Total

259

2.14

1.220

.076

1.99

2.29

0

3

Control

228

4.20

1.974

.131

3.94

4.46

0

6

Treatment

271

4.60

2.309

.140

4.33

4.88

0

6

Total

499

4.42

2.169

.097

4.23

4.61

0

6

Control

228

8.689

3.4302

.2272

8.241

9.136

.0

12

Treatment

271

9.192

4.2470

.2580

8.684

9.700

.0

12

Total

499

8.962

3.8994

.1746

8.619

9.305

.0

12

ANOVA

W_#MAST

R_#_MAST

M_#_MAST


Sum of Squares .415 383.299 383.714 19.784 2323.679 2343.463 31.364 7540.912 7572.277

df 1 257 258 1 497 498 1 497 498

Mean Square .415 1.491

F .278

Sig. .598

19.784 4.675

4.232

.040

31.364 15.173

2.067

.151

By 1998 the baseline situation presented in the previous ANOVA has been significantly changed to the benefit of the treatment school. Although these results are only statistically significant in the case of reading (p ≤ .04), it shows a marked turnaround from the baseline. This difference is made very clear when examining the means plots which follow.

9


2.18

2.16

2.14

2.12

2.10

2.08

Control

Treatment

Mean of W_#MAST CAMPUS#

4.7

4.6

4.5

4.4

4.3

4.2

4.1

Control

CAMPUS#

Treatment

Mean of R_#_MAST

9.3

9.2

9.1

9.0

8.9

8.8

8.7 8.6

Control

CAMPUS#

Treatment

Mean of M_#_MAST

10

A visual inspection of the slope of these plots when compared with the previous year provides for a striking illustration of the improvement measured. Although statistical significance has yet to be achieved in each content area, the impact of the Curry/Samara Model can already be seen on student improvement as measured by the TAAS. As we will show in the subsequent analyses not only will significant statistical significance be achieved, but at a level rarely seen in educational research.

11


Descriptives

N W_#MAST

Control Treatment

R_#_MAST

Total Control Treatment

M_#_MAST

Total Control Treatment

Total

116 141 257 245 269 514 245 269 514

Mean 2.00 2.11 2.06 3.96 4.50 4.24 7.690 9.405 8.588

Std. Deviation 1.292 1.302 1.296 2.453 2.304 2.390 4.3014 3.7620 4.1145

Std. Error .120 .110 .081 .157 .141 .105 .2748 .2294 .1815

95% Confidence Interval for Mean Lower Bound Upper Bound 1.76 2.24 1.89 2.32 1.90 2.22 3.65 4.26 4.23 4.78 4.03 4.45 7.149 8.231 8.954 9.857 8.231 8.944

Minimum 0 0 0 0 0 0 .0 .0 .0

Maximum 3 3 3 6 6 6 12 12 12

ANOVA

W_#MAST

R_#_MAST

M_#_MAST


Sum of Squares .720 429.404 430.125 38.330 2891.755 2930.086 377.303 8307.257 8684.560

df 1 255 256 1 512 513 1 512 513

Mean Square .720 1.684

F .428

Sig. .514

38.330 5.648

6.787

.009

377.303 16.225

23.254

.000

By the third year student performance at the treatment school has reached statistical significance in the core content areas of reading and mathematics. Furthermore, these results go far beyond the levels typically associated with educational significance (p ≤ .05). For the case of reading we see that the probability of this difference being due to chance is (p ≤ .009) and for mathematics (p ≤ .0005). To find significance at this level is more typical of medical research than educational or social research.

12

The following mean plots illustrate this continuing, and now in two cases statistically significant, growth in student achievement on the part of the treatment school.


2.10

2.08

2.06

2.04

2.02

2.00

1.98 Control

CAMPUS#

Treatment

Mean of W_#MAST

4.6

4.5

4.4

4.3

4.2

4.1

4.0

3.9 Control

CAMPUS#

Treatment

Mean of R_#_MAST

13

10.0

9.5

9.0

8.5

8.0

7.5 Control

CAMPUS#

Treatment

Mean of M_#_MAST

14

2000 ANOVA of Core TAAS Objectives Descriptives

N W_#MAST


137 134 271 277 269 546 277 269 546

Mean 1.90 2.41 2.15 4.16 5.00 4.58 7.957 9.870 8.899

Std. Deviation Std. Error 1.190 .102 1.028 .089 1.140 .069 2.347 .141 1.833 .112 2.148 .092 4.0973 .2462 3.1412 .1915 3.7776 .1617


Maximum 3 3 3 6 6 6 12 12 12

ANOVA

W_#MAST

R_#_MAST

M_#_MAST


Sum of Squares 17.802 332.995 350.797 95.732 2419.690 2515.421 499.533 7277.926 7777.460

df 1 269 270 1 544 545 1 544 545

Mean Square 17.802 1.238

F 14.381

Sig. .000

95.732 4.448

21.523

.000

499.533 13.379

37.338

.000

By the fourth year the superior student performance at the treatment school has resulted in the achievement of statistical significance in all three areas of writing, reading and mathematics. Furthermore, these results again go far beyond the levels typically associated with educational significance (in this case p ≤ .0005). Given the initial inequities between the two schools in favor of the control group, to have achieved such amazingly significant results is truly impressive. The mean plots illustrate this now three-way statistically significant growth in student achievement.

15


2.4

2.3

2.2

2.1

2.0

1.9

1.8 Control

Treatment

Mean of W_#MAST

CAMPUS# 5.2

5.0

4.8

4.6

4.4

4.2

4.0 Control

Treatment

Mean of R_#_MAST

CAMPUS# 10.0

9.5

9.0

8.5

8.0

7.5 Control

CAMPUS#

Treatment

Mean of M_#_MAST

16

2001 ANOVA of Core TAAS Objectives Descriptives

N W_#MAST


133 151 284 280 266 546 280 266 546

Mean 2.21 2.15 2.18 4.40 5.04 4.71 7.211 9.729 8.438

Std. Deviation Std. Error .922 .080 .969 .079 .946 .056 1.775 .106 1.633 .100 1.736 .074 2.9625 .1770 1.8208 .1116 2.7735 .1187


Maximum 3 3 3 6 6 6 12 12 12

ANOVA

W_#MAST

R_#_MAST

M_#_MAST


Sum of Squares .297 252.900 253.197 56.110 1585.745 1641.855 865.304 3327.079 4192.383

df 1 282 283 1 544 545 1 544 545

Mean Square .297 .897

F .331

Sig. .565

56.110 2.915

19.249

.000

865.304 6.116

141.483

.000

This year marks an anomaly from the marked improvement trend for the treatment school’s writing performance. Although still strong, the treatment school no longer enjoys a statistically higher level than that of the control. The associated probability that this event could be due to chance however, is quite high (p ≤ .565). And, as will be shown in the final 2002 analysis, there is much evidence to indicate that this is a one time random fluctuation. There is very little probability however, that the differences observed in the areas of reading and mathematics can be ascribed to chance (p