Research and Evaluation into the Curry/Samara Model Of Curriculum, Instruction, and Assessment.
by
Michael L. Connell, Ph.D. University of Houston Curriculum and Instruction CUIN 5872 Houston, TX 77069-5872
[email protected]
REPORT 11/04/2003 Prepared in cooperation with The Aldine Independent School District and The Curriculum Project
Table of Contents
Introduction and Overview
3
Population and Data Sources
4
A-B Comparisons: Findings and Discussion
5
Student-Problem Time Series Analysis Findings and Discussion
21
Student-Problem Growth Curve Analysis Findings and Discussion
39
Conclusions
44
Appendices A) Demographics of schools used
47
B) Time Series Analysis in Chart Form
50
C) Student-Problem Growth Curve Analysis
55
Appendix B & C are omitted from this PDF file to decrease memory requirements, but are available upon request, please call 800.867.9067 for more information.
2
Introduction and Overview
The research presented in this report addresses two foundational areas concerning the Curry/Samara Model of Curriculum, Instruction, and Assessment. The first may be classified as providing empirical, replicable and quantitative evidence that adoption and use of the Curry/Samara Model enables positive impacts upon student classroom performance. The second, and in many ways more interesting of these areas, is an attempt to describe some of the positive benefits accruing from continuing use of the Curry/Samara Model over time - particularly as reflected in long-term patterns of student achievement and curriculum implementation. For this report two parallel elementary schools were carefully selected from within Aldine Independent School District (AISD)- one of which (Control Elementary1) was described by district administrators and personnel as not significantly participating in the Curry/Samara Model and the other (Treatment Elementary) which was classified as participating at a high level. These schools served as the basis for both strands of the analysis. The primary research concerns were addressed via a two-part research and evaluation scheme involving two mutually supporting research methodologies - Analysis of Variance (ANOVA) and Student-Problem Analysis (S-P Chart). The use of the
1
This report has modified the actual school names to ensure confidentiality as requested by Aldine Independent School District.
3
Analysis of Variance methodology allows statistically significant claims to be made regarding observable differences between these two groups as reflected in their student performance scores. The use of the Student-Problem Chart allows in-depth investigation of differences in student understanding, as measured by consistency of item response, and objective mastery - as well as providing insight into the degree to which teachers were able to successfully align their instruction to the objectives evaluated. Together these two methodologies allow for comprehensive A vs. B comparisons to be made and provide a series of pedagogical and curricular insights into the nature of the differences observed addressing the identified research concerns.
Populations and Data Sources The schools chosen for participation in this study were carefully chosen to match on a number of important variables including gender, Title 1A status, educational “atrisk” classification, etc. Although in educational research a perfect match is rarely possible, these schools were found to be comparable in most areas. Furthermore, in several areas such as number of students classified as being educationally at-risk and number of students classified as limited English proficient (LEP), the control school was initially in a more favorable position. A series of frequency charts providing matching demographics of these schools is provided in Appendix A. Data on student mastery from the state mandated end of year Texas Assessment of Academic Skills (TAAS) for 1997, 1998, 1999, 2000, 2001, and 2002 were used for each
4
analysis2. The data provided by the district was in its raw form as directly received from the state ensuring its integrity. From this raw data a Statistical Package for the Social Sciences (SPSS) file was constructed for each year, mapping each element of the data line vector with their appropriate variables and labels, as described in the document Data File Format with Student Item Analysis3. These six data sets were then used as the source for demographic analysis and school selection, as well as the Analysis of Variance studies, and Student Problem Chart development.
A-B Comparisons Findings and Discussion To address the first research area, concerning empirical, replicable and quantitative evidence supporting adoption and use of the Curry/Samara Model, a series of Analysis of Variance (ANOVA) studies were conducted. These ANOVA studies compare student data by grade, on the number of TAAS objectives mastered each year in the core content areas of reading, writing, and mathematics. This data was drawn from two parallel elementary schools - one of which was identified as not significantly participating in the Curry/Samara Model (control) while the other was classified by district administrators as participating at a high level (treatment). This section will now report each of the parallel series of Analysis of Variance for the years 1997, 1998, 1999, 2000, 2001, and 2002. In these analyses, and all subsequent plots, the number of objectives mastered in the area of Writing is indicated by
2
3
A special thanks is offered to Area Superintendent Margarita Byrum and Monica Uphoff, administrative assistant of data analysis for their help and support in providing the six years of TAAS data utilized in this report. http://www.tea.state.tx.us/student.assessment/reporting/formats/
5
W_#MAST, the number of objectives mastered in the area of Reading is indicated by R_#MAST, and the number of objectives mastered in the area of Mathematics is indicated by M_#MAST.
1997 ANOVA of Core TAAS Objectives
Descriptives
N W_#MAST
Control Treatment Total R_#_MAST Control Treatment Total M_#_MAST Control Treatment Total
110 114 224 225 243 468 225 243 468
Mean 2.09 1.85 1.97 3.71 3.70 3.70 7.924 7.597 7.754
Std. Deviation Std. Error 1.246 .119 1.459 .137 1.361 .091 2.383 .159 2.772 .178 2.590 .120 4.0925 .2728 5.2075 .3341 4.7024 .2174
95% Confidence Interval for Mean Lower Bound Upper Bound Minimum 1.86 2.33 0 1.58 2.12 0 1.79 2.15 0 3.40 4.02 0 3.35 4.05 0 3.47 3.94 0 7.387 8.462 .0 6.939 8.255 .0 7.327 8.181 .0
Maximum 3 3 3 6 6 6 12 12 12
ANOVA
W_#MAST
R_#_MAST
M_#_MAST
Between Groups Within Groups Total Between Groups Within Groups Total Between Groups Within Groups Total
Sum of Squares 3.225 409.556 412.781 .029 3131.687 3131.716 12.549 10314.193 10326.741
df 1 222 223 1 466 467 1 466 467
Mean Square 3.225 1.845
F 1.748
Sig. .187
.029 6.720
.004
.948
12.549 22.133
.567
.452
It should be noted that although none of these observed differences in student mastery of objectives are statistically significant, in each case the control school is scoring slightly better. This advantage at the time of the initial data would certainly lend credence to the argument that any future growth in student performance could be attributed to the continuing implementation and use of the Curry/Samara Model. At a 6
bare minimum it illustrates the beginning comparability of the two schools chosen for use in this study. The following means plots illustrate these observed differences.
1997 Means Plots 2.2
2.1
2.0
1.9
1.8 Control
CAMPUS#
Treatment Mean of W_#MAST
4.4
4.2
4.0
3.8
3.6
3.4
3.2
3.0 Control
CAMPUS#
Treatment Mean of R_#_MAST
7
8.0
7.9
7.8
7.7
7.6
7.5 Control
CAMPUS#
Treatment Mean of M_#_MAST
These graphs clearly show that at the time the baseline for this research was drawn there was not an unfair advantage given to the treatment group by selection of the schools. Indeed, the treatment school is at a noticeable disadvantage in each area measured. As earlier indicated this is also reflected in the number of students classified as at-risk and Limited English Proficient (LEP). When one considers the initial disadvantages demonstrated by the student performance at the treatment school it makes the later growth illustrated in the following ANOVA's all the more impressive.
8
1998 ANOVA of Core TAAS Objectives
Descriptives 95% Confidence Interval for Mean N W_#MAST
R_#_MAST
M_#_MAST
Control
113
Mean 2.10
Std. Deviation 1.172
Std. Error .110
Lower Bound 1.88
Upper Bound 2.32
Minimum 0
Maximum 3
Treatment
146
2.18
1.258
.104
1.97
2.38
0
3
Total
259
2.14
1.220
.076
1.99
2.29
0
3
Control
228
4.20
1.974
.131
3.94
4.46
0
6
Treatment
271
4.60
2.309
.140
4.33
4.88
0
6
Total
499
4.42
2.169
.097
4.23
4.61
0
6
Control
228
8.689
3.4302
.2272
8.241
9.136
.0
12
Treatment
271
9.192
4.2470
.2580
8.684
9.700
.0
12
Total
499
8.962
3.8994
.1746
8.619
9.305
.0
12
ANOVA
W_#MAST
R_#_MAST
M_#_MAST
Between Groups Within Groups Total Between Groups Within Groups Total Between Groups Within Groups Total
Sum of Squares .415 383.299 383.714 19.784 2323.679 2343.463 31.364 7540.912 7572.277
df 1 257 258 1 497 498 1 497 498
Mean Square .415 1.491
F .278
Sig. .598
19.784 4.675
4.232
.040
31.364 15.173
2.067
.151
By 1998 the baseline situation presented in the previous ANOVA has been significantly changed to the benefit of the treatment school. Although these results are only statistically significant in the case of reading (p ≤ .04), it shows a marked turnaround from the baseline. This difference is made very clear when examining the means plots which follow.
9
1998 Means Plots 2.20
2.18
2.16
2.14
2.12
2.10
2.08
Control
Treatment
Mean of W_#MAST CAMPUS#
4.7
4.6
4.5
4.4
4.3
4.2
4.1
Control
CAMPUS#
Treatment
Mean of R_#_MAST
9.3
9.2
9.1
9.0
8.9
8.8
8.7 8.6
Control
CAMPUS#
Treatment
Mean of M_#_MAST
10
A visual inspection of the slope of these plots when compared with the previous year provides for a striking illustration of the improvement measured. Although statistical significance has yet to be achieved in each content area, the impact of the Curry/Samara Model can already be seen on student improvement as measured by the TAAS. As we will show in the subsequent analyses not only will significant statistical significance be achieved, but at a level rarely seen in educational research.
11
1999 ANOVA of Core TAAS Objectives
Descriptives
N W_#MAST
Control Treatment
R_#_MAST
Total Control Treatment
M_#_MAST
Total Control Treatment
Total
116 141 257 245 269 514 245 269 514
Mean 2.00 2.11 2.06 3.96 4.50 4.24 7.690 9.405 8.588
Std. Deviation 1.292 1.302 1.296 2.453 2.304 2.390 4.3014 3.7620 4.1145
Std. Error .120 .110 .081 .157 .141 .105 .2748 .2294 .1815
95% Confidence Interval for Mean Lower Bound Upper Bound 1.76 2.24 1.89 2.32 1.90 2.22 3.65 4.26 4.23 4.78 4.03 4.45 7.149 8.231 8.954 9.857 8.231 8.944
Minimum 0 0 0 0 0 0 .0 .0 .0
Maximum 3 3 3 6 6 6 12 12 12
ANOVA
W_#MAST
R_#_MAST
M_#_MAST
Between Groups Within Groups Total Between Groups Within Groups Total Between Groups Within Groups Total
Sum of Squares .720 429.404 430.125 38.330 2891.755 2930.086 377.303 8307.257 8684.560
df 1 255 256 1 512 513 1 512 513
Mean Square .720 1.684
F .428
Sig. .514
38.330 5.648
6.787
.009
377.303 16.225
23.254
.000
By the third year student performance at the treatment school has reached statistical significance in the core content areas of reading and mathematics. Furthermore, these results go far beyond the levels typically associated with educational significance (p ≤ .05). For the case of reading we see that the probability of this difference being due to chance is (p ≤ .009) and for mathematics (p ≤ .0005). To find significance at this level is more typical of medical research than educational or social research.
12
The following mean plots illustrate this continuing, and now in two cases statistically significant, growth in student achievement on the part of the treatment school.
1999 Means Plots 2.12
2.10
2.08
2.06
2.04
2.02
2.00
1.98 Control
CAMPUS#
Treatment
Mean of W_#MAST
4.6
4.5
4.4
4.3
4.2
4.1
4.0
3.9 Control
CAMPUS#
Treatment
Mean of R_#_MAST
13
10.0
9.5
9.0
8.5
8.0
7.5 Control
CAMPUS#
Treatment
Mean of M_#_MAST
14
2000 ANOVA of Core TAAS Objectives Descriptives
N W_#MAST
Control Treatment Total R_#_MAST Control Treatment Total M_#_MAST Control Treatment Total
137 134 271 277 269 546 277 269 546
Mean 1.90 2.41 2.15 4.16 5.00 4.58 7.957 9.870 8.899
Std. Deviation Std. Error 1.190 .102 1.028 .089 1.140 .069 2.347 .141 1.833 .112 2.148 .092 4.0973 .2462 3.1412 .1915 3.7776 .1617
95% Confidence Interval for Mean Lower Bound Upper Bound Minimum 1.70 2.10 0 2.23 2.59 0 2.01 2.29 0 3.88 4.44 0 4.78 5.22 0 4.39 4.76 0 7.472 8.441 .0 9.493 10.247 .0 8.582 9.217 .0
Maximum 3 3 3 6 6 6 12 12 12
ANOVA
W_#MAST
R_#_MAST
M_#_MAST
Between Groups Within Groups Total Between Groups Within Groups Total Between Groups Within Groups Total
Sum of Squares 17.802 332.995 350.797 95.732 2419.690 2515.421 499.533 7277.926 7777.460
df 1 269 270 1 544 545 1 544 545
Mean Square 17.802 1.238
F 14.381
Sig. .000
95.732 4.448
21.523
.000
499.533 13.379
37.338
.000
By the fourth year the superior student performance at the treatment school has resulted in the achievement of statistical significance in all three areas of writing, reading and mathematics. Furthermore, these results again go far beyond the levels typically associated with educational significance (in this case p ≤ .0005). Given the initial inequities between the two schools in favor of the control group, to have achieved such amazingly significant results is truly impressive. The mean plots illustrate this now three-way statistically significant growth in student achievement.
15
2000 Means Plots 2.5
2.4
2.3
2.2
2.1
2.0
1.9
1.8 Control
Treatment
Mean of W_#MAST
CAMPUS# 5.2
5.0
4.8
4.6
4.4
4.2
4.0 Control
Treatment
Mean of R_#_MAST
CAMPUS# 10.0
9.5
9.0
8.5
8.0
7.5 Control
CAMPUS#
Treatment
Mean of M_#_MAST
16
2001 ANOVA of Core TAAS Objectives Descriptives
N W_#MAST
Control Treatment Total R_#_MAST Control Treatment Total M_#_MAST Control Treatment Total
133 151 284 280 266 546 280 266 546
Mean 2.21 2.15 2.18 4.40 5.04 4.71 7.211 9.729 8.438
Std. Deviation Std. Error .922 .080 .969 .079 .946 .056 1.775 .106 1.633 .100 1.736 .074 2.9625 .1770 1.8208 .1116 2.7735 .1187
95% Confidence Interval for Mean Lower Bound Upper Bound Minimum 2.05 2.37 0 1.99 2.30 0 2.07 2.29 0 4.19 4.61 0 4.84 5.24 0 4.57 4.86 0 6.862 7.559 .0 9.510 9.949 .0 8.205 8.671 .0
Maximum 3 3 3 6 6 6 12 12 12
ANOVA
W_#MAST
R_#_MAST
M_#_MAST
Between Groups Within Groups Total Between Groups Within Groups Total Between Groups Within Groups Total
Sum of Squares .297 252.900 253.197 56.110 1585.745 1641.855 865.304 3327.079 4192.383
df 1 282 283 1 544 545 1 544 545
Mean Square .297 .897
F .331
Sig. .565
56.110 2.915
19.249
.000
865.304 6.116
141.483
.000
This year marks an anomaly from the marked improvement trend for the treatment school’s writing performance. Although still strong, the treatment school no longer enjoys a statistically higher level than that of the control. The associated probability that this event could be due to chance however, is quite high (p ≤ .565). And, as will be shown in the final 2002 analysis, there is much evidence to indicate that this is a one time random fluctuation. There is very little probability however, that the differences observed in the areas of reading and mathematics can be ascribed to chance (p