Saxon Elementary Math Program Effectiveness Study

Web Version

Final Report

Saxon Elementary Math Program Effectiveness Study October 2006

Dr. Kimberly Good Edvantia, Inc. Dr. Robert Bickel Marshall University Dr. Caitlin Howley Edvantia, Inc.

Edvantia is a nonprofit education research and development corporation, founded in 1966, that partners with practitioners, education agencies, publishers, and service providers to improve learning and advance student success. We provide clients with a range of services, including research, evaluation, professional development, and consulting.

For information about Edvantia projects, programs, and services, contact

P.O. Box 1348 Charleston, WV 25325-1348 304.347.0400 800.624.9120 304.347.0487 (fax) [email protected] www.edvantia.org

© 2006 by Edvantia, Inc. All rights reserved.

This publication was prepared for Harcourt Achieve. Its contents do not necessarily reflect the positions or policies of Edvantia.

Edvantia is an Equal Employment Opportunity/Affirmative Action Employer

2

Final Report

Contents EXECUTIVE SUMMARY..................................................................................................................................6 PROJECT BACKGROUND...............................................................................................................................8 STUDY PURPOSE AND RESEARCH QUESTIONS.......................................................................................8 DESIGN AND METHODOLOGY......................................................................................................................9 Sample .................................................................................................................................................9 Measures . ..........................................................................................................................................11 Analyses . ...........................................................................................................................................14 FINDINGS........................................................................................................................................................15 RESULTS..........................................................................................................................................................16 Question 1: To what extent does the implementation of the Saxon K–3 Math program improve student achievement in math?...................................................................................16 Average Gains for All Saxon Math Students....................................................................................17 Achievement Gains Among Saxon Students by Grade Level..........................................................18 Question 2: To what degree does the implementation of the Saxon K–3 Math program improve the math achievement of student subgroups, such as low socioeconomic status (e.g., students eligible for free or reduced-price lunch), special education, racial minority?................20 Saxon Students Eligible for Free/Reduced-Price Lunch..................................................................20 Saxon Students Enrolled in Special Education................................................................................21 Saxon Students Who Are English Language Learners....................................................................22 Saxon Students Who Are Members of Racial/Ethnic Minority Groups..........................................23 Question 3. To what extent does the fidelity of the Saxon Math program implementation affect math achievement?.........................................................................................................................25 Fidelity of Saxon Implementation and Overall Math Achievement Growth of Saxon Students...............................................................................................................................26 Fidelity of Saxon Implementation and Math Problem Solving Achievement Growth of Saxon Students.................................................................................................................26 Fidelity of Saxon Implementation and Math Procedures Achievement Growth of Saxon Students.................................................................................................................26 Saxon’s Assessment Measures...........................................................................................................27 Question 4. To what extent are there significant differences in the math achievement of students participating in the Saxon Math program compared to students not participating in the program? .......................................................................................27 Reconfigured Fidelity of Saxon Implementation and Math Achievement Growth . .....................28 Reconfigured Fidelity of Saxon Implementation and Math Achievement Growth with Student-Level Controls.............................................................................................................29 Reconfigured Fidelity of Saxon Implementation and Overall Math Achievement Growth with Student-Level Controls and School-Level Controls............................................................30 Multilevel Regression Analysis of Saxon Schools’ Performance.....................................................31

Saxon Elementary Math Program Effectiveness Study

Question 5. To what degree of fidelity did the teachers in this study implement the Saxon Math Program?..............................................................................................................................32 Videotaped Classroom Lessons ........................................................................................................32 LoU Telephone Interviews ................................................................................................................33 Teacher Surveys ................................................................................................................................33 SUMMARY.......................................................................................................................................................34 Limitations ................................................................................................................................................34 Concluding Comments .............................................................................................................................34 REFERENCES.................................................................................................................................................35

LIST OF TABLES Table 1. Number of Teacher Participants Per Grade Level ..................................................................10 Table 2. Demographic Characteristics of Participating Students ........................................................11 Table 3. Percentage of Participating Students by Grade Level ............................................................11 LIST OF FIGURES Figure 1. Participating study schools ......................................................................................................10 Figure 2. A three-level nesting model .....................................................................................................15 Figure 3. Average SAT 9 math achievement growth for Saxon students, Grades K–3 .......................17 Figure 4. Average SAT 9 math achievement growth, kindergarten......................................................18 Figure 5. Average SAT 9 math achievement growth, first grade .........................................................18 Figure 6. Average SAT 9 math achievement growth, second grade ......................................................19 Figure 7. Average SAT 9 math achievement growth, third grade.........................................................20 Figure 8. Average SAT 9 math achievement growth, students eligible for free/reduced-price lunch ..21 Figure 9. Average SAT 9 math achievement growth, special education students ...............................22 Figure 10. Average SAT 9 math achievement growth, English language learner students................23 Figure 11. Average SAT 9 math achievement growth, racial/ethnic minority students ......................24 Figure 12. Teacher fidelity of Saxon Math implementation, as received by students .........................25 Figure 13. Teacher fidelity of Saxon Math implementation, as received by all students (comparison and experimental) .............................................................................................................. 28

4

Final Report

Appendixes

The following APPENDIXES are available at www.saxonpublishers.com/2919 Appendix A. Statistical Figures and Tables – Research Questions 3 and 4 Question 3. To what extent does fidelity of Saxon Math program implementation affect math Appendix A. Statistical Figures and Tables – Research Questions 3 and 4 achievement? Question 3. To what extent does fidelity of Saxon Math program implementation affect math Question 4. To what extent are there significant differences in the math achievement of students achievement? participating in the Saxon Math program compared to students not participating in the program? Question 4. To what extent are there significant differences in the math achievement of students Appendix participating B. Detailed in the Saxon Results Math of program Research compared Question to students 5 – Implementation not participating Fidelity in the program? Videotaped Classroom Lessons Appendix B. Detailed Results of Research Question 5 – Implementation Fidelity LoU Telephone Interviews Videotaped Classroom Lessons Experimental Teacher Survey LoU Telephone Interviews Comparison Teacher Survey Experimental Teacher Survey Appendix Comparison C. Saxon Teacher Math Survey Teacher Implementation Specifics Videotaped Classroom Observations—Additional Materials and Resources Appendix C. Saxon Math Teacher Implementation Specifics Experimental Teacher Open-Ended Survey Responses Videotaped Classroom Observations—Additional Materials and Resources Appendix Experimental D. Instrumentation Teacher Open-Ended and Study SurveyCorrespondence Responses Sample Letter of Invitation (Experimental Sites) Appendix D. Instrumentation and Study Correspondence Sample Letter of Invitation (Comparison Sites) Sample Letter of Invitation (Experimental Sites) Memorandum of Understanding (Experimental Sites) Sample Letter of Invitation (Comparison Sites) Memorandum of Understanding (Comparison Sites) Memorandum of Understanding (Experimental Sites) Teacher Consent Forms (Experimental and Comparison Site) Memorandum of Understanding (Comparison Sites) Parent/Guardian Consent Form (Experimental and Comparison Sites) Teacher Consent Forms (Experimental and Comparison Site) Student Assent Script Parent/Guardian Consent Form (Experimental and Comparison Sites) Site Coordinator Pretest Administration Instructions Student Assent Script Teacher Pretest Administration Instructions Site Coordinator Pretest Administration Instructions Test Security Affidavit Teacher Pretest Administration Instructions Student Demographic Data-Collection Sheet Test Security Affidavit Instructions for Videotaped Classroom Lesson Student Demographic Data-Collection Sheet Kindergarten Innovation Configuration Matrix (ICM) Instructions for Videotaped Classroom Lesson First-Grade Innovation Configuration Matrix (ICM) Kindergarten Innovation Configuration Matrix (ICM) Second- and Third-Grade Innovation Configuration Matrix (ICM) First-Grade Innovation Configuration Matrix (ICM) Introductory Protocol for LoU Telephone Interviews Second- and Third-Grade Innovation Configuration Matrix (ICM) LoU Telephone Interview Protocol Introductory Protocol for LoU Telephone Interviews Experimental Site Coordinators’ Posttest Administration Instructions LoU Telephone Interview Protocol Experimental Teachers’ Posttest Administration Instructions Experimental Site Coordinators’ Posttest Administration Instructions Comparison-Site Coordinators’ Posttest Administration Instructions Experimental Teachers’ Posttest Administration Instructions Comparison-Site Teachers’ Posttest Administration Instructions Comparison-Site Coordinators’ Posttest Administration Instructions Instructions for Recording Saxon Math Student-Assessment Data Comparison-Site Teachers’ Posttest Administration Instructions Experimental Teacher Surveys (Grade-Level Specific) Instructions for Recording Saxon Math Student-Assessment Data Comparison Teacher Surveys (Grade-Level Specific) Experimental Teacher Surveys (Grade-Level Specific) Appendix Comparison E. Lessons Teacher Learned Surveys (Grade-Level Specific) Appendix E. Lessons Learned


EXECUTIVE SUMMARY Harcourt Achieve requested the assistance of Edvantia, an independent research organization, to conduct a randomized effectiveness evaluation of the Saxon Math program in Grades K–3 to determine its effects on children’s mathematics performance. The purpose of the research study was to examine the effectiveness of the Saxon Math program over 1 academic year (2005–2006). A secondary purpose was to assess the fidelity with which participating teachers implement the Saxon Math program. A quasiexperimental design was used in this study. A group of 33 schools were randomly selected from a list of schools provided by Harcourt Achieve that were already using the Saxon Math program. These schools (Saxon Math users) are designated as experimental sites. Twenty-four schools with demographic characteristics similar to those of the experimental schools were selected and are referred to in this report as comparison sites. These 24 schools use a variety of other math programs. With a true experimental design, participating sites would have been randomly assigned to one of, in this case, two conditions—use of the Saxon Math program or nonuse of the program. The Stanford Achievement Test, Ninth Edition (SAT 9) was used as a pre- and posttest measure of student achievement. Pretesting occurred in October or November 2005, and posttesting took place within the last month of the school year (e.g., May through August 2006). Participating students completed only the math subtest of the SAT 9. Resultant data include overall math scores for all students as well as Grades 2 and 3 scores from two math subtests: (a) Math Problem Solving and (b) Math Procedures. Teachers at experimental sites were also asked to record Saxon Math student assessment data for participating students at three points throughout the school year (beginning, middle, and end). To measure the fidelity with which teachers at experimental sites implemented the Saxon Math program, three data collection methods were employed: levels of use (LoU) telephone interviews, videotaped classroom lessons, and an implementation survey.

6

Descriptive statistics were calculated to examine implementation trends. Thematic analysis techniques were employed to analyze qualitative data. Bivariate and multilevel regression analyses were conducted to analyze pre- and posttest assessment results and growth over time for the Saxon Math student assessments. Data were collected from randomly selected students within randomly selected classrooms within randomly selected schools, resulting in a three-level nested model. Randomization was conducted at the school level and to some extent at the classroom level. However, due to extreme variability in the amount of Saxon program use across students within classrooms, multilevel regression analysis was conducted at the student level. The five research questions and associated major findings are reported as follows: 1. To what extent does the implementation of the Saxon K–3 Math program improve student achievement in math? • Overall and for each grade level, Saxon students made significant gains on all three SAT 9 math achievement measures over the course of the study year: overall math achievement, math problem solving, and math procedures. • Moreover, students in Saxon sites made significant achievement gains despite evidence from all but one bivariate analysis indicating that they began each grade at a higher-thanaverage achievement level. • Specifically, kindergarten through third-grade students had an average scaled-score gain of 36.69 points from pre- to posttest. Second- and third-grade gains were 33.69 points on the Math Problem Solving subtest and 53.28 points on the Math Procedures subtest. 2. To what degree does the implementation of the Saxon K–3 Math program improve the math 1

In some classrooms, eight or fewer parents granted permission for their children to participate in the study. In those situations, all students with parental permission were selected for inclusion in the study.

2

In some schools, only one teacher at a particular grade level volunteered for study participation.

Final Report

achievement of student subgroups, such as low socioeconomic status, special education, racial minority, and English language learner students? • The groups of students commonly regarded as academically, economically, or culturally disadvantaged who were in Saxon schools made significant annual gains on the three SAT 9 math achievement measures. • The mean overall SAT 9 math achievement gain for Saxon students eligible for free or reducedprice school lunch was 37.20 points. • Saxon special education students’ overall SAT 9 math achievement scores increased, on average, by 34.06 scaled score points. • Saxon English language learners gained an average of 40.69 scaled-score points on the overall SAT 9 math test. • Saxon minority group members gained an average of 36.01 scaled-score points over the course of a school year on the overall SAT 9 math test. 3. To what extent does the fidelity of the Saxon Math program implementation affect math achievement? • The efficacy of the Saxon intervention depends, in large measure, on the fidelity of the Saxon implementation. For this study, the fidelity of Saxon implementation was measured through ratings assigned following the LoU telephone interviews with Saxon Math program teachers. • For each one-level increase in the measure of the fidelity of Saxon implementation, there was, on average, a 26.09-point annual increase in overall math achievement. • Each one-level increase in the measure of the fidelity of Saxon implementation corresponds, on average, to a 16.77-point increase in math problem-solving growth. • Each one-level increase in the fidelity of Saxon implementation results in math procedures growth from fall to spring of, on average, 15.17 scaled-score points. • The Saxon Math program provides achievement tests aligned with the curriculum that can be

used in conjunction with the SAT 9 to assess growth. Saxon assessments were administered at the beginning, middle, and end of the school year. Test results should be interpreted with caution, however, because they are characterized by marked ceiling effects, with average scores of 87.26% correct, 89.27% correct, and 86.26% correct. With percentage scores this high, it is unrealistic to expect scores to increase from one testing occasion to another. Nevertheless, analyses suggest that average Saxon test scores increase slightly as fidelity of Saxon implementation improves. 4. To what extent are there significant differences in the math achievement of students participating in the Saxon Math program compared to students not participating in the program? • When comparing Saxon students’ performance to the performance of students from comparison schools, it becomes clear that the fidelity of the Saxon implementation is crucial. • Saxon students outperform students from comparison schools when implementation exceeds a minimum level. As the level of implementation improves, Saxon students’ comparative performance improves. • Differences in Saxon students’ performance and comparison school students’ performance are especially evident when the relationship between the Saxon implementation level and math achievement is treated as a quadratic relationship rather than a linear relationship. At the bivariate level, the result is an upward sweeping curve, indicating dramatic improvement in performance with higher levels of implementation of the Saxon intervention. • With a properly specified multilevel regression model, students in schools implementing Saxon, on average, perform better on all three Saxon measures of math achievement than do students in comparison schools. The differences are not large, but they are statistically significant. When interpreting these Saxon advantages, it is important to remember that comparison schools implement other types of math programs.


• Multilevel regression analysis indicates that, as the fidelity with which the Saxon Math program is implemented increases, in the experimental schools there is an average increase of 2.51 scaled-score points on the overall SAT 9 math test for experimental students compared to those attending comparison schools. Such analyses also indicate that Math Problem Solving subtest scores increased on average by 2.98 scaled points, and Math Procedures subtest scores increased on average by 7.59 scaled points.

Harcourt Achieve requested the assistance of Edvantia, an independent research organization, to conduct a randomized effectiveness evaluation of the Saxon Math program in Grades K through 3 to determine its effects on children’s mathematics performance over the course of 1 academic year. In addition, the study was prompted by the No Child Left Behind Act of 2001, which mandates that educational materials purchased with public funds must be proven by scientific research to improve student achievement in the classroom.

5. To what degree of fidelity did the teachers in this study implement the Saxon Math program?

The Saxon Math program is based on an incremental pedagogical approach that emphasizes practice, review, and frequent cumulative assessment. It is the Saxon philosophy that mathematics learning should build on prior learning. Mathematical strands are integrated throughout the year rather than taught in isolated units (Larson, 2004). Saxon Math is the only major math program on the market today that systematically distributes instruction, practice, and assessment across the academic year as opposed to concentrating, or massing, the instruction, practice, and assessment of related concepts into a short period of time—usually within a unit or chapter (Larson). Saxon Math’s approach to math instruction aims to ensure that students both gain and retain essential math skills.

• In general, the videotaped lessons indicate that teachers implemented the Saxon Math program as intended. • The large majority of teachers in this study routinely use the Saxon Math program. According to data from LoU interviews, the program is generally used as intended, with little deviation from the curriculum. • Survey data indicate that teachers believe themselves to be implementing the Saxon Math program as it was designed to be implemented. Several teachers reported that they use additional materials to supplement the Saxon Math program. • Teachers reported being pleased with the Saxon Math program. The components of the program with which they are most satisfied include repetition of content, the spiraling approach, and overall design of the lessons. Teachers also noted that these program components were those that originally led to their schools’ adoption of Saxon.

PROJECT BACKGROUND Harcourt Achieve provides customer-driven educational materials that are designed to fundamentally and positively change the lives of young, adolescent, and adult learners and to empower those who teach them. Harcourt Achieve has shown its commitment to rigorous research; the What Works Clearinghouse has accepted a research report about the effectiveness of the Saxon Math program in Grades 6 through 8.

8

STUDY PURPOSE AND RESEARCH QUESTIONS The purpose of the research study described herein was to determine the effectiveness of the Saxon Math program over the course of 1 academic year. Specifically, the study was designed to answer the following five research questions: 1. To what extent does the implementation of the Saxon K–3 Math program improve student achievement in math? 2. To what degree does the implementation of the Saxon K–3 Math program improve the math achievement of student subgroups, such as low socioeconomic status, special education, racial minority, and English language learner students?

Final Report

3. To what extent does the fidelity of the Saxon Math program implementation affect math achievement? 4. To what extent are there significant differences in the math achievement of students participating in the Saxon Math program compared to students not participating in the program? 5. To what degree of fidelity did the teachers in this study implement the Saxon Math program?

DESIGN AND METHODOLOGY This section of the report describes the study design and data-collection methods. Sampling techniques are detailed, as are the measures employed to assess the effectiveness of the Saxon Math program. Data-analysis methods are also explained.

Sample The national study described here employed a quasiexperimental research design. With a true experimental design, participating sites would have been randomly assigned to one of, in this case, two conditions—use of the Saxon Math program or nonuse of the program. However, it was not possible to randomly assign schools to use the Saxon Math program or to use other math programs. Therefore, schools using the Saxon Math program were randomly selected to participate in the study (e.g., experimental schools), and schools not using the Saxon Math program were randomly selected to participate in the study (e.g., comparison schools). A total of 33 experimental and 24 comparison schools across 16 states participated in the study. Random assignment and selection are generally effective safeguards against confounding treatment and control independent variables with extraneous factors (Shadish, Cook, & Campbell, 2002). In this case, however, such safeguards apply only at the school level. Random assignment or selection at the school level is not effective to address confounding at the level of the classroom or the individual student. Nevertheless, because the treatment is

applied or withheld at the school level, the rationale for random selection at the school level is evident. In the spring of 2005, Harcourt Achieve sent Edvantia researchers a spreadsheet containing the names of U.S. schools implementing the Saxon Math program. Edvantia staff randomly selected schools to participate in the study. Letters of invitation were mailed or faxed to the randomly selected schools, and follow-up phone calls were made. Researchers sought 40 schools implementing Saxon to participate in the study. Ultimately, a total of 33 schools agreed to become involved in the study. These schools (Saxon Math users) are referred to as “experimental” sites. Comparison schools were selected based on their similarities to the experimental schools on several measures, including school size; grade-level configuration; percentage of students eligible for free and reduced-price school lunch (the conventional education research proxy measure for poverty); percentage of racial/ethnic minority students; migrant percentages; charter school designation; Title I school designation; locale (e.g., urban, rural, large town, small town); and geographic location. Experimental and comparison schools were matched as closely as possible based on these characteristics. Data with which to identify matches were obtained from the U.S. Department of Education’s (ED) National Center on Educational Statistics (NCES) Common Core of Data (CCD) for public schools from the 2003–2004 school year. Schools not implementing the Saxon Math program that had characteristics similar to those of schools in the experimental group were mailed or faxed letters of invitation, followed by phone calls. A total of 24 comparison schools agreed to participate in the study. As shown in Figure 1, experimental and comparison schools were located across 16 states, including Alabama (1), Arizona (5), California (6), Georgia (3), Indiana (1), North Carolina (9), Nebraska (5), Nevada (2), New York (2), Oklahoma (9), Oregon (2), Tennessee (2), Texas (2), Utah (1), Virginia (6), and Washington (1).


Figure 1. Participating study schools.

Note. Circles • represent experimental schools, and diamonds ♦ represent comparison schools.

Four teachers at each experimental and comparison school—one for each grade, K through 3—were either randomly selected or volunteered to participate in the study. Table 1 illustrates the number of teachers participating at each grade level. The Saxon Math program was not implemented in kindergarten at one of the experimental schools and not used at the thirdgrade level at two of the experimental schools. One second-grade and three kindergarten teachers in the experimental group withdrew prior to study completion. Although teacher survey data and videotaped lessons were not collected from these teachers, student pretest and posttest data were obtained. In addition, telephone interview data were collected from these teachers. Table 1. Number of Teacher Participants Per Grade Level Grade

Experimental

Comparison

K

29

24

1

33

24

2

32

24

3

31

24

Total

125

96

Note. Numbers reflect teachers who completed all study components. Three kindergarten and one second-grade teacher in the experimental group withdrew from the study. One school does not use Saxon Math in kindergarten, and two schools do not use the program in third grade.

10

Permission was secured from parents of children in the randomly selected classrooms to participate in the pre- and posttesting; permission was also granted for researchers to collect student demographic information and Saxon Math studentassessment data for the participating classrooms at the experimental sites. Of the total number of children who received parental permission to participate in the study, 8 children from each classroom were selected. The selection of 8 children in each classroom was the target number designated in the research study design proposal. In a few instances, there were fewer than 8 students in the classroom, or signed parental consent forms were not received from 8 children’s parents. In these cases, all students for whom parental consent was obtained participated in the study. To facilitate communication with study participants and systematize data collection, each participating site nominated a site coordinator, who received a small stipend for his or her efforts. Site coordinators were responsible for responding to researchers’ requests for information, distributing datacollection instruments, collecting completed datacollection instruments, facilitating the videotaping of Saxon lessons, scheduling telephone interviews, and returning data to researchers. In addition, to encourage study participation and discourage attrition, all participating teachers were provided a small stipend, half to be paid in the middle of the study year and the remainder at the conclusion of the year. Teachers and site coordinators read and signed a Memorandum of Understanding, indicating the roles and responsibilities upon which payment was contingent. Students were not offered any compensation for their participation in the study. Ultimately, 1,541 students, nested within 221 classrooms, nested within 57 schools, participated in this study. The 57 schools were nested within 16 states: Alabama (1), Arizona (5), California (5), Georgia (3), Indiana (1), North Carolina (9), Nebraska (5), Nevada (3), New York (2), Oklahoma (9), Oregon (2), Tennessee (2), Texas (2), Utah (1), Virginia (6), and Washington (1). A total of 865 children from experimental schools and 676 children from comparison schools participated in the study. However, due to attrition, complete data (i.e., both SAT 9 pre- and posttest data) were

Final Report

obtained from 829 experimental-school children and 647 comparison-school children. At the time the SAT 9 pretest was administered, demographic data were collected about each participating student. Teachers reported the gender, birth date, free and reduced-price school meal eligibility, special education (SPED) status, English language learner (ELL) status, and ethnicity for participating students. As shown in Table 2, the percentages of experimental- and comparison-school students across demographic categories are comparable.

Table 2. Demographic Characteristics of Participating Students Demographics

ExperimentalSchool Students

ComparisonSchool Students

Gender (% male)

51%

50%

Free/reduced-price lunch eligible (% yes)

40%

47%

5%

6%

SPED (% yes) ELL (% yes)

11%

7%

Ethnicity (% White)

65%

64%

Note. Demographic data were not available for some students. Therefore, the percentages of students reported in the various categories may be underestimated.

The percentages of students participating in the study at each grade level are roughly equivalent in experimental and comparison schools (see Table 3). Roughly a quarter of students at experimental and comparison sites were in kindergarten, first, second, and third grades. Table 3. Percentage of Participating Students by Grade Level Grade

ExperimentalSchool Students

Comparison-School Students

K

24%

25%

1

26%

26%

2

25%

25%

3

25%

25%


Measures Stanford Achievement Test, Ninth Edition (SAT 9). The SAT 9 was administered as the pre- and posttest measure of student math achievement. Participating students completed only the math portion of the SAT 9. The SAT 9 mathematics subtests assess the entire breadth of mathematical content recommended by the National Council of Teachers of Mathematics (Harcourt Educational Measurement, 2000). The nationally normed SAT 9 test was selected over other achievement measures because it is available at the kindergarten level, whereas other assessments did not provide kindergarten assessments. Students were pretested and posttested. The tests were administered by either the classroom teacher or the site coordinator. Pretesting took place in November 2005 and posttesting took place during each school’s last month in session during the spring or summer of 2006. In the fall, students took the appropriate grade-level versions of the SAT 9, which were the SESAT 1, SESAT 2, abbreviated Primary 1, and abbreviated Primary 2 tests, respectively, for kindergarten through third grades. The spring tests that were administered to K–3 students included SESAT 2, abbreviated Primary 1, abbreviated Primary 2, and abbreviated Primary 3. The levels administered to the students were those recommended by the SAT 9 publishers, based on grade level and timing of the test (fall and spring). The SESAT 1 and 2 contain 40 multiple-choice items. The Primary 1, 2, and 3 each have two sections: Procedures and Problem Solving, which include 20 items and 30 items, respectively. Test items are clustered in the following seven categories: Number Sense and Numeration, Geometry and spatial Sense, Measurement, Patterns and Relationships, Statistics and Probability, Computation in Context, and Number Facts. The data were analyzed and reported using scaled scores. Overall math test scores from pre- and posttest SAT 9 administrations were calculated for all students. Additionally, pretest and posttest scores were calculated for Math Problem Solving and Math Procedures subtests at the second- and third-grade levels.3 3

The abbreviated SAT 9 test administered in this study does not provide cluster scores. Results are analyzed by overall math score and subtest only.

11

Saxon Math program student assessments. Teachers employed at experimental sites were asked to record Saxon Math student-assessment data for participating students at three points over the course of the school year (beginning, middle, and end). Teachers were requested to provide scores from a designated beginning-of-the-year assessment. To accommodate differences in the speed with which teachers progressed through the Saxon Math curriculum throughout the year, they were asked to report the specific assessments they used at the middle and end-of-year assessment periods. In addition, kindergarten teachers indicated the number of total points possible for middle and end-of-year assessments. At the firstthrough third-grade levels, a total of 100 points were possible for all assessments used. Ultimately, all scores were converted to a percentage, with a maximum possible of 100%, to ease interpretation. Videotaped classroom lesson. Saxon Math teachers each submitted one videotaped Saxon Math classroom lesson (n = 125). The purpose of the videotaped math lessons, and the subsequent ratings of each, was to assess the fidelity with which participating teachers implemented the Saxon Math program. Each teacher chose the lesson taught for the videotape. A memo detailing instructions for videotaping lessons, an information sheet to be completed by each teacher, and blank videotapes were mailed to each site coordinator (see Appendix D). Schools were divided into three groups; mailings of such materials to the three groups were then staggered over 3 months (January, February, or March 2006). Teachers had 1 month in which to videotape a complete Saxon Math lesson of their choice. Each videotape was to include the entire span of time the teacher devoted to teaching math over the course of one day (e.g., The Meeting, Lesson, Guided or Lesson Practice, Fact Practice, Handwriting Practice, and Counting Practice, if applicable). Teachers also were asked to submit a brief information sheet describing the lesson number and topic, time of day The Meeting and Lesson were conducted, and the perceived and actual amount of time to conduct The Meeting and Lesson.

12

Prior to viewing the videotaped lessons, Edvantia staff collaborated with Harcourt Achieve research and professional development staff to develop an Innovation Configuration Matrix (ICM) detailing the key components that should be observed in fully implemented Saxon lessons. The ICM, a component of the Concerns Based Adoption Model (CBAM), outlines critical program elements and displays variations in implementation ranging from no implementation to full implementation (Hord, Rutherford, Huling-Austin, & Hall, 1987). The ICMs were customized for each grade level and included the main components of the Saxon Math program (i.e., The Meeting, Lesson, Guided or Lesson Practice, Number Fact Practice, Handwriting Practice, and Counting Practice) as well as constituent characteristics of each. The resultant tool documents program fidelity in teaching the various components of Saxon Math. For each critical component of Saxon displayed on the ICM, four variations of implementation fidelity were specified. Each critical component and its constituents are summarized and explained in this report. A copy of each grade-level ICM has been included in Appendix D and may be referenced for more detail. The Meeting. Five constituents of The Meeting were identified: practice of skills, use of manipulatives, frequency of questioning, use of wait time, and classroom management. Lesson. Eight constituents of the Lesson were identified: stating the objective, modeling the concepts, use of manipulatives, modeling the use of the manipulatives, frequency of questioning, use of wait time, students sharing learnings, and classroom management. Guided or Lesson Practice. Lesson Practice is conducted at the kindergarten level and Guided Practice is used in the first through third grades. The kindergarten ICM included two constituents: completing Lesson or Guided Practice sheet and teacher interaction during Lesson or Guided Practice sheet. In addition to these two constituents, the firstthrough third-grade ICMs included time allowed for completing the Guided Practice sheet.

Final Report

Number Fact Practice. Number Fact Practice is conducted in the first through third grades. Four constituents of Number Fact Practice were identified: frequency of questioning, use of manipulatives, completing class Fact Practice sheet, and teacher interaction during class Fact Practice sheet. Handwriting Practice. Handwriting Practice is conducted at the kindergarten level. Two constituents of Handwriting Practice were identified: completing Handwriting Practice sheet and teacher interaction during Handwriting Practice sheet completion. Counting Practice. Counting Practice is conducted at the kindergarten level. Two constituents of Counting Practice were identified: completing Counting Practice sheet and teacher interaction during Counting Practice sheet completion. For each constituent of critical Saxon components, four implementation variants were identified. The first variation represents the fullest implementation of Saxon, whereas the fourth variation represents the Saxon implementation with the least fidelity to the program. Edvantia staff, trained in the use of ICMs, viewed the videotapes and assigned one variation rating to each component in the ICM. If a component or its constituent was not observed, a not observed code was assigned. Not observed ratings were omitted from all analyses. Telephone interviews. Telephone interviews took place with the experimental group teachers (n = 129) during the month following their videotape submissions (e.g., February through April). After teachers submitted their videotapes, they were contacted to participate in a telephone interview conducted by Edvantia staff. Edvantia researchers, trained and certificated in the Levels of Use methodology, used the validated LoU protocol to conduct the telephone interviews (see Appendix C). The LoU interview is a method developed and tested by Loucks, Newlove, and Hall (1975) and is a component of CBAM. LoU interviews assist the determination of the degree to which respondents are implementing a given innovation (in this instance, the innovation is the Saxon Math program). Eight levels of use may be


delineated using such interviews, ranging from lack of knowledge of the innovation to “an active, sophisticated, and highly effective use” of the innovation (Hall, Loucks, Rutherford, & Newlove, 1975). Specifically, levels are nonuse, orientation, preparation, mechanical use, routine, refinement, integration, and renewal. Teachers were rated on seven categories: knowledge, acquiring information, sharing, assessing, planning, status reporting, and performing. Knowledge is what the teacher knows about the characteristics of the Saxon Math program, how to use it, and the consequences of its use. Acquiring information concerns the extent to which the teacher solicits information about the Saxon Math program (e.g., seeking Internet-based resources, attending professional development). Sharing assesses the extent to which teachers discuss the Saxon Math program with others. It includes sharing plans, ideas, resources, outcomes, and problems related to the use of the program. Assessing involves examining the potential or actual use of the Saxon Math program or some aspect of it. Planning is the designing and outlining of short- and/or long-range steps to be taken as the program is implemented. For status reporting, each teacher describes her or his personal stance toward the use of the innovation at the time of the interview. Performing refers to the actions and activities that are carried out in operationalizing the Saxon Math program. A specific series of questions aligned with each of the seven categories was posed to each interviewee. Responses to questions in each category determined LoU ratings (nonuse, orientation, preparation, mechanical use, routine, refinement, integration, and renewal) for each of the seven categories. In the final rating step, an overall LoU rating was assigned to each interviewee based on ratings assigned to each of the seven categories. During the course of each interview, Edvantia staff wrote notes. Staff then completed a rating sheet using the LoU guide to determine appropriate ratings. A rating was given for each of seven categories of behavior, in addition to an overall rating. Additionally, 13 interviews (10% of all interviews conducted) were tape recorded and

13

submitted to a second review to ensure sufficient interrater reliability. Analysis of interrater reliability indicated that the first and second raters provided identical overall LoU ratings in 92% of instances, representing an adequate level of interrater reliability. Teacher survey. Teachers from both experimental (n = 125) and comparison (n = 96) groups completed grade-level surveys (see Appendix C for a copy of the surveys). The teacher surveys were intended to gather information on mathematics classroom curriculum and instruction. Surveys were tailored for experimental and comparison groups and grade levels. Experimental teacher surveys asked questions about the implementation of Saxon Math program components. The majority of the questions were multiple choice. Open-ended questions pertained to supplemental materials used, reason the program was adopted, identification of strengths of the program, and suggested changes. Comparison teacher surveys asked more general questions regarding scope and sequence of math curriculum, instructional practices, and mathematics resources used. Nearly all questions were multiple choice, with a few short-answer items. Self-reported classroom and teacher demographic data also were gathered via the survey.

Analyses Quantitative data were entered, cleaned, and analyzed using the Statistical Package for the Social Sciences (SPSS) software program. Descriptive statistics (frequencies, percentages, means, medians, variances, standard deviations) were examined for all quantitative data collected. Replies to open-ended survey items were analyzed by theme and by interview question, as appropriate. Data were segmented into passages through coding. Emerging themes were identified and data reviewed for replicating categories. These categories were given broad codes. Once significant data were categorized, finer coding was employed. Finer coding was completed using patterns emerging within each coded set. Themes were then

14

summarized to provide examples of the most salient and prevalent issues. Bivariate and multilevel regression were used to analyze pre- and posttest assessment results and growth over time for the Saxon Math student assessments. Multilevel regression enables the researcher to address the nested nature of educational data. By formally representing each level of the data structure (e.g., repeated observations of persons or of persons within communities), this modeling technique can account for the variance associated with each level, thereby generating a more accurate general model (Francis, Schatschneider, & Carlson, 2000). Multilevel regression models, properly specified, can help determine if changes in students’ math achievement are statistically significant or simply artifacts of measurement error or chance, and if there are meaningful differences between Saxon Math in experimental sites and the implementation of other programs in comparison schools. This sort of analysis is effective and sensitive enough to locate moderate effect sizes. Multilevel regression also allows researchers to examine the independent and combined effects of important variables on the dependent variable— math achievement, in this instance. Age, gender, special education and English language learner status, racial minority categorization, free and reduced-price lunch eligibility (employed as a proxy measure of poverty), and pretest scores were used as student-level covariates. The three-level multilevel regression model employed in this study consists of submodels at levels 1 (child), 2 (classroom), and 3 (school). In this case, the research problem includes data from diverse children (English language learners, low SES, or special education) nested within schools. Figure 2 illustrates this nesting schematically. This study consists of a simple two-level analysis (child and school); classrooms were not included as a separate level because too much implementation variability was located at the classroom level.

Final Report

Figure 2. A Three-level nesting model.

2. To what degree does the implementation of the Saxon K–3 Math program improve the math achievement of student subgroups, such as low socioeconomic status, special education, racial minority, and English language learner students? • The groups of students commonly regarded as academically, economically, or culturally disadvantaged who were in Saxon schools made significant annual gains on the three SAT 9 math achievement measures.

School Level 3

• The mean overall SAT 9 math achievement gain for Saxon students eligible for free or reducedprice school lunch was 37.20 points.

Classroom Level 2

Child Level 1

• Saxon special education students’ overall SAT 9 math achievement scores increased, on average, by 34.06 scaled score points. • Saxon English language learners gained an average of 40.69 scaled-score points on the overall SAT 9 math test.

FINDINGS Major findings from this study of the effectiveness of the Saxon Math program for Grades K through 3 include the following: 1. To what extent does the implementation of the Saxon K–3 Math program improve student achievement in math? • Overall and for each grade level, Saxon students made significant gains on all three SAT 9 math achievement measures over the course of the study year: overall math achievement, Math Problem Solving, and Math Procedures. • Moreover, students in Saxon sites made significant achievement gains despite evidence from all but one bivariate analysis indicating that they began each grade at a higher-thanaverage achievement level. • Specifically, kindergarten through third-grade students had an average scaled score gain of 36.69 points from pre- to posttest. Second- and third-grade gains were 33.69 points on the Math Problem Solving subtest and 53.28 points on the Math Procedures subtest.


• Saxon minority group members gained an average of 36.01 scaled-score points over the course of a school year on the overall SAT 9 math test. 3. To what extent does the fidelity of the Saxon Math program implementation affect math achievement? • The efficacy of the Saxon intervention depends, in large measure, on the fidelity of the Saxon implementation. For this study, the fidelity of Saxon implementation was measured through ratings assigned following the LoU telephone interviews with Saxon Math program teachers. • For each one-level increase in the measure of the fidelity of Saxon implementation there was, on average, a 26.09-point annual increase in overall math achievement. • Each one-level increase in the measure of the fidelity of Saxon implementation corresponds, on average, to a 16.77-point increase in Math Problem Solving growth. • Each one-level increase in fidelity of Saxon implementation results in Math Procedures growth from fall to spring of, on average, 15.17 scaled-score points.

15

• The Saxon Math program provides achievement tests aligned with the curriculum that can be used in conjunction with the SAT 9 to assess growth. Saxon assessments were administered at the beginning, middle, and end of the school year. Test results should be interpreted with caution, however, because they are characterized by marked ceiling effects, with average scores of 87.26% correct, 89.27% correct, and 86.26% correct. With percentage scores this high, it is unrealistic to expect scores to increase from one testing occasion to another. Nevertheless, analyses suggest that average Saxon test scores increase slightly as fidelity of Saxon implementation improves. 4. To what extent are there significant differences in the math achievement of students participating in the Saxon Math program compared to students not participating in the program? • When comparing Saxon students’ performance to the performance of students from comparison schools, it becomes clear that the fidelity of the Saxon implementation is crucial. • Saxon students outperform students from comparison schools when implementation exceeds a minimum level. As the level of implementation improves, Saxon students’ comparative performance improves. • Differences in Saxon students’ performance and comparison school students’ performance are especially evident when the relationship between the Saxon implementation level and math achievement is treated as a quadratic relationship rather than a linear relationship. At the bivariate level, the result is an upward sweeping curve, indicating dramatic improvement in performance with higher levels of implementation of the Saxon intervention. • With a properly specified multilevel regression model, students in schools implementing Saxon, on average, perform better on all three Saxon measures of math achievement than do students in comparison schools. The differences are not large, but they are statistically significant. When interpreting these Saxon advantages, it is important to remember that many comparison schools implement other types of math programs.

16

• Multilevel regression analysis indicates that, as the fidelity with which the Saxon Math program is implemented increases, in the experimental schools there is an average increase of 2.51 scaled-score points on the overall SAT 9 math test for experimental students compared to those attending comparison schools. Such analyses also indicate that Math Problem Solving subtest scores increased on average by 2.98 scaled points, and Math Procedures subtest scores increased on average by 7.59 scaled points. 5. To what degree of fidelity did the teachers in this study implement the Saxon Math program? • In general, the videotaped lessons indicate that teachers implemented the Saxon Math program as intended. • The large majority of teachers in this study routinely use the Saxon Math program. According to data from LoU interviews, the program is generally used as intended, with little deviation from the curriculum. • Survey data indicate that teachers believe themselves to be implementing the Saxon Math program as it was designed to be implemented. Several teachers reported that they use additional materials to supplement the Saxon Math program. • Teachers reported being pleased with the Saxon Math program. The components of the program with which they are most satisfied include repetition of content, the spiraling approach, and overall design of the lessons. Teachers also noted that these program components were those that originally led to their schools’ adoption of Saxon.

RESULTS This section presents a detailed summary of the study results organized by the five major research questions and the related findings.

Question 1: To what extent does the implementation of the Saxon K–3 Math program improve student achievement in math?

Final Report

Average gains on the SAT 9 overall math test and subtests are presented for all Saxon Math students here. Following these data are results by grade level for Saxon Math students. The Math Problem Solving and Math Procedures subtest results are available for only Grades 2 and 3. Paired t tests were used to determine if statistically significant differences occurred over time.

Average Gains for All Saxon Math Students Figure 3 provides a graphical representation of the average overall math achievement growth made by all participating students in Saxon schools. Participating kindergarteners through third graders in sites using Saxon Math achieved an average scaled-score gain of 36.69 points, equivalent to 0.37 of a standard deviation. Scaled scores are calculated to have a standard deviation of 100 and a mean of 500 (Nunnally & Bernstein, 1994). In this study, participating Saxon students began school in the fall of 2005, exceeding the mean score of 500, on

average, by 0.39 of a standard deviation. By the end of the school year, the same Saxon students exceeded the mean score, on average, by 0.76 of a standard deviation. This growth was statistically significant, t(828) = -32.98, p < .001. Similar gains were made by Saxon students on the Math Problem Solving and Math Procedures subtests of the SAT 9. The average Saxon student gain in Math Problem Solving was 33.69 points, equivalent to 0.34 of a standard deviation. Scores increased from 597.14 at the beginning of the school year to 630.83 by the end of the year (see Figure 3). This growth was statistically significant, t(410) = –20.29, p < .001. The average Saxon student gain on the Math Procedures subtest of the SAT 9 was 53.28 points, equal to 0.53 of a standard deviation. Saxon students’ scores increased from an average of 553.35 to 606.63 by the end of the academic year (see Figure 3). This growth was statistically significant, t(410) = –22.50, p < .001.

Figure 3. Average SAT 9 math achievement growth for Saxon students, Grades K–3. 640

630.83

620

606.63 597.14

Scaled Scores

600 576.00

580

553.35

560 539.31 540 520 500 480

Overall Math Achievement

Math Problem Solving

Pretest

Math Procedures

Posttest

Note. Math Problem Solving and Math Procedures, Grades 2 and 3 only.


17

Achievement Gains Among Saxon Students by Grade Level

Figure 5. Average SAT 9 math achievement growth, first grade.

Kindergarten

560

554.75 550

Scaled Scores

Kindergarten students participating in the Saxon Math program made substantial gains in overall math achievement. Kindergarten students’ scores increased by an average of 34.71 scaled-score points from the beginning to the end of the school year. It should be noted that Saxon kindergarten students in this study scored below the average of 500 at the beginning of the school year, but by spring 2006, their mean score was above 500. This gain corresponds to an increase of 0.35 of a standard deviation. Figure 4 illustrates this gain. This growth was statistically significant, t(202) = –16.88, p < .001.

540

530

524.87 520

510

Pretest

Posttest


Figure 4. Average SAT 9 math achievement growth, kindergarten. 520

509.43

510

Scaled Scores

500 490 480

474.72

470 460 450

Pretest

Posttest


First Grade Overall math achievement gains of participating Saxon first graders are comparable to those of Saxon kindergarten students. In the case of first graders, fall to spring average overall math achievement growth was 29.88 points, or 0.30 of a standard deviation, with mean scores increasing from 524.87 to 554.75. Figure 5 illustrates this gain. This growth was statistically significant, t(215) = –13.23, p < .001.

18

Second Grade A substantial average achievement gain also characterizes the experience of Saxon second graders participating in this study. Whereas the Saxon second-grade students earned a scaled-score mean of 556.66 at the opening of the school year, their average end-of-year score was 599.38. This represents a statistically significant average gain of 42.72 points, or 0.43 of a standard deviation, t(207) = -19.04, p < .001. Figure 6 illustrates this gain. In terms of the Math Problem Solving subtest, the average achievement gain over the second-grade year was 31.80 points and 0.32 of a standard deviation, t(207) = –13.93, p < .001. The mean Math Problem Solving score of students participating in the Saxon Math program in this study increased from 578.11 to 609.91. As with five of the six achievement gains documented earlier in this report, participating Saxon students scored above average at the beginning of the school year and still made substantial gains (see Figure 6). As with the Math Problem Solving subtest, comparatively high-achieving Saxon students also made substantial gains over the course of the school year on the Math Procedures subtest. In this instance, Saxon second graders’ mean score at the beginning of the school year was 526.54, and it

Final Report

Figure 6. Average SAT 9 math achievement growth, second grade. 620

609.91 599.38

600

586.97 578.11

Scaled Scores

580

560

556.66

540 526.54 520

500

480



Pretest

increased to 586.97 points by the end of the year. This represents a statistically significant average gain of 60.43 points and 0.60 of a standard deviation, t(207) = –17.33, p < .001 (see Figure 6).

Third Grade Third graders at participating Saxon schools, on average, scored just over 100 points above the mean of 500 scaled-score points at the beginning of the school year, and they still managed to gain an additional 39.72 points, or 0.40 of a standard deviation, by spring 2006. The average third-grade math score grew from 601.81 to 641.53. Recognizing the importance of the statistical artifact known as regression toward the mean (Campbell & Kenny, 1995), these findings represent especially impressive Saxon third-grade performance. Figure 7 illustrates this gain. This growth was statistically significant, t(201) = –17.82, p < .001. Saxon third graders participating in this analysis of the Saxon Math Program also made gains on the Math Problem Solving subtest, despite posting mean scores well above the average of 500 scaled-


Math Procedures

Posttest

score points at the beginning of the academic year. Whereas participating Saxon students earned 616.64 points at the beginning of the third grade, they nonetheless gained, on average, 35.62 scaledscore points. This gain corresponds to a substantially significant increase of 0.36 of a standard deviation, t(202) = –14.76, p < .001 (see Figure 7). Third-grade students participating in this investigation of the Saxon Math program started the year with a mean Math Procedures subtest score that was higher than the mean scaled score of 500. Nevertheless, these students’ mean gain for the school year was even larger than the mean gain made by second-grade Saxon students on this subtest. Third graders began the fall with a mean Math Procedures score of 580.83 and still managed to post an average gain of 45.94 points. Thus, a gain of one-half of a standard deviation (0.50) was made by students who were already 0.80 of a standard deviation above the mean scaled score, t(202) = –14.72, p < .001 (see Figure 7).

19

Figure 7. Average SAT 9 math achievement growth, third grade. 660

652.26 641.53

640 626.77 616.64

Scaled Scores

620 601.81 600

580.83 580

560

540



Pretest

Question 2: To what degree does the implementation of the Saxon K–3 Math program improve the math achievement of student subgroups, such as low socioeconomic status (e.g., students eligible for free or reduced-price lunch), special education, racial minority, and English language learner students? Findings from the overall SAT 9 math test and the two corresponding subtests are presented for each of the four types of student subgroups investigated here. Subgroups include students eligible for subsidized school meals, special education students, English language learners, and students belonging to racial/ethnic minority groups (e.g., African American, American Indian, and Hispanic). Paired t tests were used to determine if statistically significant differences occurred over time.

Saxon Students Eligible for Free/ Reduced-Price Lunch An oft-cited challenge to the utility of accountability systems based largely on standardized testing is that they do not acknowledge or seek to mediate

20

Math Procedures

Posttest

confounding factors (Bowles, Gintis, & Groves, 2005). Proponents of No Child Left Behind have argued that controlling for confounding factors such as socioeconomic status and race/ethnicity is a manifestation of the “soft bigotry of low expectations” (Bickel & Howley, 2003). Consequently, it is especially useful to measure the longitudinal performance of low-income Saxon students to explore the ability of the program to support the achievement of students who face academic disadvantages. The overall math achievement of Saxon students whose families are sufficiently poor to render them eligible for subsidized school meals seems unaffected by economic disadvantage. Specifically, the mean overall math achievement gain of such students from fall to spring was 37.20 points and 0.37 of a standard deviation, comparable to the average gains for all Saxon students in this study. As was shown previously in Figure 3, the average overall math achievement gain for all Saxon students was 36.69 points and 0.37 of a standard deviation. Economically disadvantaged Saxon students, on average, did quite well in comparison, as illustrated by Figure 8. This growth was statistically significant, t(321) = –21.25, p < .001.

Final Report

Figure 8. Average SAT 9 math achievement growth, students eligible for free/reduced-price lunch. 620

613.50 593.68

600 582.57 580 Scaled Scores

564.70 560 543.89 540

527.50

520

500

480



Pretest

These findings are especially promising given the significant effect socioeconomic class is shown to have on student achievement (Coleman et al., 1966; Riordan, 2004). Much research makes clear that social class differences almost invariably yield achievement test score differences that work to the disadvantage of students from low-income families (Bickel, Howley, & Maynard, 2003). In this case, however, students qualifying for subsidized school meals made math achievement gains nearly as large as those of their wealthier peers. Results from the Math Problem Solving subtest again indicate that economically disadvantaged students attending schools implementing the Saxon Math program made large and statistically significant gains over the course of 1 academic year, t(165) = –13.13, p < .001 (Figure 8). Not only did these Saxon students score 82.57 points above average at the beginning of the school year, but by the end of the school year their average performance had increased by 30.93 points and 0.31 of a standard deviation to 613.50. Economically disadvantaged Saxon students also performed impressively on the SAT 9 Math


Math Procedures

Posttest

Procedures subtest, with average scaled scores growing from 543.89 to 593.68 over the course of the year (see Figure 8). This growth was statistically significant, t(164) = –13.05, p < .001. The Math Procedures mean gain for all Saxon students was 53.28, little different from the mean gain of 49.79 for Saxon students eligible for subsidized school meals, representing a gain of 0.50 of a standard deviation. Moreover, the small difference between the average gain for all Saxon students and for economically disadvantaged Saxon students was not statistically significant.

Saxon Students Enrolled in Special Education A total of 37 students participating in this research effort at Saxon sites were enrolled in special education programs. As with other Saxon students, the average overall math achievement gain for special education students from fall to spring is substantial, as is illustrated in Figure 9. In addition to posting above-average scaled scores at the beginning and end of the school year, Saxon special education students’ overall math achievement

21

scores increased, on average, by 34.06 scaled score points and 0.34 of a standard deviation. The overall math achievement of special education students in Saxon schools increased from 515.97 to 550.03. This growth was statistically significant, t(38) = -7.86, p < .001. The average Math Problem Solving gain for special education students at Saxon schools in this study was 23.08 scaled-score points and 0.23 of a standard deviation, as is illustrated in Figure 9. This growth was statistically significant, t(23) = -3.77, p < .001. This was a substantial gain but only about two thirds as large as gains reported for other groups. Nevertheless, in spite of the special education designation shared by these students, their fall Math Problem Solving score of 570.50 was well above average, and the same was true of their average score in the spring. Special education students attending schools implementing the Saxon Math program in this study made substantial achievement gains on the SAT 9 Math Procedures subtest. The mean achievement gain for Saxon special education students was 51.74 scaled score points and 0.52 of

a standard deviation, increasing from 521.78 to 573.52. Figure 9 illustrates this gain. This growth was statistically significant, t(22) = –3.81, p < .001. On balance, test scores and scaled-score gains for Saxon special education students were comparable to those for the entire sample and for other specific groups. Such results should not be taken to mean that students’ characteristics are of no consequence when measuring achievement and achievement gains. Nevertheless, the finding that special education students in Saxon schools made substantial gains over the course of the study year is consistent with the claim that all students can make real achievement gains.

Saxon Students Who Are English Language Learners Schools in the United States are serving increasing numbers of students whose native languages are not English, the academic performance of whom is often decried (Riordan, 2004). However, the overall math achievement of Saxon English language learners in this study was comparable to that of other groups. At both the beginning and end of the

Figure 9. Average SAT 9 math achievement growth, special education students. 600

593.58

580

560 Scaled Scores

573.52

570.50

550.03

540

520

521.78

515.97

500

480

460



Pretest

22

Math Procedures

Posttest

Final Report

school year, Saxon English language learners earned above-average overall math achievement scaled scores, as is illustrated in Figure 10. Moreover, English language learners attending Saxon sites in this study gained, on average, 40.69 scaled-score points and 0.41 of a standard deviation, with mean scores increasing from 506.61 to 547.30. This growth was statistically significant, t(92) = –12.65, p < .001. Saxon students who were English language learners earned, on average, scaled scores above average on the SAT 9 Math Problem Solving subtest. In addition, their average achievement gain from fall to spring was 32.28 test-score points and 0.32 of a standard deviation, with scores increasing significantly from 568.83 to 601.11, t(35) = –5.59, p < .001 (see Figure 10). As with overall math test scores, Saxon English language learners performed better than average on the pretest and then continued to make substantial gains over the course of the academic year. Among English language learners at Saxon sites in this analysis, the largest gains, on average, were made on the Math Procedures subtest of the SAT 9. Saxon English language learners began the school year with an above-average Math Procedures pretest

score, and they gained, on average, 65.17 points and 0.65 of a standard deviation. Scores improved from 527.89 to 593.06, as is illustrated in Figure 10. This growth was statistically significant, t(35) = -8.72, p < .001. This is an achievement gain of nearly two thirds of a standard deviation over the course of 1 academic year. The achievement growth of English language learners attending Saxon sites was much the same for the entire sample and for other specific groups, with high levels of achievement accompanied by substantial achievement growth.

Saxon Students Who Are Members of Racial/Ethnic Minority Groups Members of racial/ethnic minority groups—African Americans, American Indians, and Hispanics— typically do less well on standardized achievement tests such as the SAT 9 than do other students (Hallinan, 2001). However, such a typification does not apply to the Saxon students in this sample. Saxon students who belong to racial/ethnic minority groups achieved at levels comparable to those of

Figure 10. Average SAT 9 math achievement growth, English language learners. 620 601.11

600

593.06

580 568.83

Scaled Scores

560 547.30

540

527.89 520 500

506.61

480 460 440 Overall Math Achievement


Pretest


Math Procedures

Posttest

23

other students, and their fall-to-spring gains were substantial. Saxon minority group members earned higher-than-average pre- and posttest overall math achievement scores, and they experienced statistically significant gains of, on average, 36.01 scaled score points and 0.36 of a standard deviation over the course of the school year, t(542) = –25.44, p < .001, as is illustrated in Figure 11. The overall math achievement of racial/ethnic minority students attending Saxon schools in this study increased from 545.20 to 581.21. African American, American Indian, and Hispanic students at Saxon schools in this research made gains on the Math Problem Solving subtest comparable to their gains in overall math achievement. This holds even though their average Math Problem Solving pretest score was already just over one standard deviation above the mean. By the end of the school year, moreover, the average score for African American, American Indian, and Hispanic students at Saxon schools in this study was 640.32 scaledscore points, or 1.40 standard deviations, above average, as is illustrated in Figure 11. The posttest score of 640.32 points includes a fall-to-spring gain of 35.04 points, comparable to that of other groups of

Saxon students studied here. This growth was statistically significant, t(257) = -16.34, p < .001. The largest gain made by African American, American Indian, and Hispanic students at Saxon schools in this analysis was on the Math Procedures subtest of the SAT 9. Already scoring above average on the Math Procedures pretest, members of racial/ethnic minority groups attending Saxon schools in this study gained, on average, 55.36 scaled-score points in Math Procedures and 0.55 of a standard deviation, as is illustrated in Figure 11. The mean Math Procedures subtest scaled scores of African American, American Indian, and Hispanic students at Saxon sites in this analysis increased from 557.34 to 612.70, representing statistically significant growth, t(258) = –18.41, p < .001. Although students belonging to racial/ethnic minority groups often confront challenges to their academic growth, the findings from this study suggest that the math achievement of African American, American Indian, and Hispanic students may benefit from the Saxon Math program. This applies to measured achievement levels as well as to achievement growth over the course of the school year.

Figure 11. Average SAT 9 math achievement growth, racial/ethnic minority students. 660 640.32

640 620

612.70 605.28

Scaled Scores

600 581.21

580 560 540

557.34 545.20

520 500 480 Overall Math Achievement


Pretest

24

Math Procedures

Posttest

Final Report

Question 3. To what extent does the fidelity of the Saxon Math program implementation affect math achievement? This research question examined the relationship between the fidelity of teacher implementation of the Saxon Math program and Saxon students’ math achievement. Measurement of fidelity of implementation is discussed in the subsequent paragraph. Measures of student math achievement included overall SAT 9 math achievement growth, Math Problem Solving achievement growth, Math Procedures achievement growth, and Saxon Math assessment measures. As with any reform or innovation, the effectiveness of the Saxon Math program depends in part on the fidelity of its implementation. The independent variable used to measure fidelity of implementation for this portion of the study came from LoU telephone interview ratings. The quality of implementation of the Saxon Math program could vary across five levels, as is shown in Figure 12. Ratings given to the ordinal variable Fidelity of Saxon Implementation were 3 through 7. The corresponding labels accompanying these values are as follows: 3 = mechanical use, 4 = routine use, 5 = refinement, 6 = integration, and 7 = renewal.

Mechanical use refers to the state in which the teacher focuses most effort on the short-term, dayto-day use of the Saxon Math program and has little time for reflection. Teachers at this stage are primarily engaged in a stepwise attempt to master the tasks required to use the Saxon Math program, often resulting in disjointed and superficial implementation. During routine use, the teacher makes few if any changes to the ongoing use of the Saxon Math program. At the refinement stage, the teacher varies use of the Saxon Math program to increase its impact on students. Variations are based on knowledge of both short- and long-term consequences for students. At the integration level, the teacher combines his or her efforts to use the Saxon Math program with related activities of colleagues to achieve a collective impact on students. Finally, at the renewal stage, the teacher reevaluates the quality of the use of the program, seeking major modifications of the program or alternatives to the program to achieve increased impact on students. Most of the 829 Saxon students in this study (573, or 69.1%) attended a school at which teachers were classified as Level 4(routine) users, as is illustrated

Figure 12. Teacher fidelity of Saxon Math implementation, as received by students. 700 600

573

Scaled Scores

500

400

300 195

200

100 33

21 0 Mechanical

Routine

Refinement

Integration

7 Renewal

LoU Ratings


25

in Figure 12. The median level is 4, the mean level is 4.31, and the standard deviation is 0.63. The concentration of cases at Level 4 (routine use) constrains the variability in “Fidelity of Saxon Implementation” measure, which limits its value for analytical purposes. Nonetheless, the measure does permit some analysis of whether or not level of implementation is associated with the math achievement of students at participating Saxon schools. The detailed summary of findings for the relationship of fidelity of Saxon Math Program implementation and the four measures of student math achievement is presented below. Statistical figures are shown in Appendix A.

Fidelity of Saxon Implementation and Overall Math Achievement Growth of Saxon Students The quadratic relationship provides a better fit than the linear relationship of the relationship between the Fidelity of Saxon Implementation and overall math achievement growth as measured by the SAT 9. The R2 values for the two functional forms are nearly the same: 6.9% for the linear relationship and 7.6% for the quadratic relationship (see Figure A-1 in Appendix A). Given this, the most parsimonious and otherwise useful rendering of the relationship between Fidelity of Saxon Implementation and overall math achievement growth is as follows: Each one-level increase in Fidelity of Saxon Implementation corresponds, on average, to a 26.09point annual increase in overall math achievement. This interpretation makes clear that Saxon Math program implementation should not be construed as a dichotomous phenomenon. Instead, as the Fidelity of Saxon Implementation (as measured using its five levels) increases, so does overall math achievement, as measured by the SAT 9.

Fidelity of Saxon Implementation and Math Problem Solving Achievement Growth of Saxon Students

curvilinear, though to a lesser degree than overall math achievement. R2 values for the two functional forms are nearly identical, so once again the more parsimonious relationship is selected. For Math Problem Solving, each one-level increase in the Fidelity of Saxon Implementation corresponds, on average, to a 16.77-point increase in Math Problem Solving growth (see Figure A-2 in Appendix A). As with overall math achievement, the Saxon Math program variable is better treated as a five-level ordinal measure corresponding to levels of implementation rather than as a simple dichotomy.

Fidelity of Saxon Implementation and Math Procedures Achievement Growth of Saxon Students As with the SAT 9 measures of overall math achievement and Math Problem Solving, the relationship between SAT 9 Math Procedures subtest scores and the Fidelity of Saxon Implementation variable is slightly curvilinear in this study. While a quadratic function provides a better fit, the difference between the explanatory power of a quadratic functional form and a linear functional form is sufficiently small to warrant opting for the parsimony of a linear relationship. This enables the relationship between Math Procedures achievement growth and level of implementation of the Saxon intervention to be summarized as follows: For each one-level increase in Fidelity of Saxon Implementation, Math Procedures growth from fall to spring increases, on average, by 15.17 scaled-score points (see Figure A3 in Appendix A). Again, these data suggest that various levels of Saxon implementation are not equal in their consequences. The efficacy of the Saxon Math program depends, in large measure, on the fidelity with which it is implemented. At the bivariate level, this has been observed with regard to overall math achievement, Math Problem Solving, and Math Procedures. The upward sweeping character of the best-fitting functional form serves to emphasize the importance of implementation.

The relationship between Fidelity of Saxon Implementation and Math Problem Solving is also

26

Final Report

Saxon’s Assessment Measures In addition to the SAT 9 tests, the Saxon Math program provides participating teachers with its own math assessment measures to gauge student performance. Saxon assessment scores from the beginning, middle, and end of the school year were collected from teachers at Saxon sites participating in the study. Scores were reported as the percentage of correct answers. For the three testing occasions, Saxon assessment mean scores were 87.26%, 89.27%, and 86.26%. With average scores so close to the maximum possible, it is unrealistic to expect Saxon students to consistently show gains from one testing occasion to another. However, the relationship between Saxon percentage scores and Fidelity of Saxon Implementation can be examined. Using a linear functional form, the relationship between Saxon achievement test score and level of implementation is positive but not statistically significant (see Figure A-4 in Appendix A). A quadratic functional form provides a better fit, but the association remains weak. Saxon students’ average scores are so high that there is little room for scores to improve but a great deal of room within which scores might decrease due to random measurement error. The most important finding from the aforementioned analysis of Saxon Math implementation and student performance is that, as the fidelity of Saxon Math implementation improves beyond routine levels of use, student performance improves as measured by the SAT 9. In other words, there are differences in student performance depending on the fidelity with which teachers implement the Saxon Math program.

Question 4. To what extent are there significant differences in the math achievement of students participating in the Saxon Math program compared to students not participating in the program? This research question examined whether there were difference in math achievement of students who were receiving Saxon Math program instruction compared to students who received other types of math curriculum instruction. Again,


the measures of math achievement were overall SAT 9 math achievement growth, Math Problem Solving achievement growth, Math Procedures achievement growth, and Saxon Math assessment measures. Also introduced into the analysis was the fidelity of implementation. In other words, the analyses examined whether differences in math achievement for students receiving Saxon Math program instruction were different depending on the teacher fidelity of implementation as compared to the math achievement for students receiving instruction under another math curriculum. The focus of Research Question 3 was on the relationships between math achievement and the Saxon Math program and therefore was limited to students in Saxon schools. At the bivariate level, however, it is possible to include students attending comparison schools simply by redefining the Fidelity of Saxon Implementation independent variable. Because the Fidelity of Saxon Implementation variable is ordinal, it does not possess a welldefined unit of measure. As a result, observations can be ranked in terms of Fidelity of Saxon Implementation, but the distance of one ranked observation from another is not known. Because the original Fidelity of Saxon Implementation variable ranks observations in terms of five levels, 3 through 7, another level has been added to enable the analysis here, indicating the absence of Saxon implementation, for comparison schools. The ordinal level Fidelity of Saxon Implementation variable thus now includes six levels, 2 through 7, with level 2 representing comparison schools not implementing the Saxon Math program at all. See Figure 13 for an illustration. The detailed summary of findings for the reconfigured relationship of fidelity of Saxon Math program implementation and the three measures of student math achievement is presented next. Again the reconfigured relationship of fidelity of Saxon Math program implementation includes all students. The purpose of including students who are receiving Saxon Math program instruction and students receiving instruction from other curricula

27

is to determine (a) whether differences in student achievement exist between the two groups and (b) to examine whether a relationship in student achievement depends on the fidelity of Saxon Math implementation. Bivariate analysis results are displayed without any controls, with student-level controls, and then with both student-level and school-level controls. The section concludes with the results of the multivariate analyses. Statistical figures are contained in Appendix A.

Reconfigured Fidelity of Saxon Implementation and Math Achievement Growth Overall Math Achievement Growth. The relationship between the level of implementation and scaled scores is much more curvilinear than data only from students and teachers at schools using the Saxon Math program. This judgment is consistent with the R2 values reported for the two functional forms. While both values are small, there is nonetheless a substantial difference between them, which warrants the use of a quadratic relationship.

The slope of the linear relationship remains statistically significant, however, and can be interpreted as follows: For each one-level increase in the Reconfigured Level of Saxon Implementation variable, annual overall math achievement growth increases by 3.31 points (see Figure A-5 in Appendix A). In addition, the curve representing the quadratic relationship between overall math achievement growth and the Reconfigured Fidelity of Saxon Implementation variable turns upward as it reaches implementation Level 4, or routine use (Wooldridge, 2006, p. 201). It is at this point of implementation that the Saxon Math program begins to influence test scores. Applying the quadratic equation, the estimated overall math achievement mean score at that level is 590.83. The most important finding from such analyses is that the average performance of students in Saxon schools surpasses the average performance of students in comparison schools as the level of Saxon implementation increases. At the lowest level of implementation, a Saxon school is a Saxon school in name only.

Figure 13. Teacher Fidelity of Implementation, as received by all students (comparison and experimental). 700

680

600

573

Scaled Scores

500

400

300 195

200

100 33

21 0 Comparison Group

Mechanical

Routine

Refinement

Integration

7 Renewal

Comparison School and Saxon School LoU Ratings

28

Final Report

Math Problem Solving Achievement Growth. The relationship between Math Problem Solving and the Reconfigured Fidelity of Saxon Implementation variable is very similar to the relationship presented previously. Again, there is a curvilinear relationship best described using a quadratic functional form. Moreover, the Saxon Math program also begins to pay off in terms of scaled score points at implementation Level 4, or routine use, on the Math Problem Solving subtest of the SAT 9 (see Figure A-6 in Appendix A). Using the quadratic equation, the estimated Math Problem Solving score at that level is 604.16. Erring on the side of parsimony, the linear functional form indicates that each one-level increase in the Reconfigured Fidelity of Saxon Implementation variable corresponds to a 2.36point increase in the SAT 9 Math Problem Solving score. This, however, ignores the obviously curvilinear character of the relationship. As with the analysis of overall math achievement scores, the average performance of students in schools implementing the Saxon Math program surpasses the average performance of students in comparison schools as the level of Saxon implementation increases. Math Procedures Achievement Growth. The relationship between the Reconfigured Fidelity of Saxon Implementation variable and Math Procedures achievement growth is similar to that between implementation and overall math achievement and Math Procedures achievement. It is again shown that a curvilinear relationship provides a better fit than a linear relationship and that the Saxon Math program begins to pay off at implementation Level 4, or routine use (see Figure A-7 in Appendix A). The best estimate of the math procedures score at that level is 575.57. To review, as the level of fidelity with which Saxon is implemented increases, student achievement in schools using the program is slightly better that that of students in comparison schools.

Reconfigured Fidelity of Saxon Implementation and Math Achievement Growth with Student-Level Controls Overall Math Achievement Growth. Sometimes it is essential to work with curvilinear relationships. That, after all, is why polynomial regression was invented (Halcoussis, 2005). Not infrequently, however, curvilinearity turns out to be a manifestation of specification error in the form of excluded independent variables (Kennedy, 2003). In this study, when student-level controls for free/ reduced-price school meal eligibility, race/ethnicity, special education status, English language learner status, grade level, and pretest score are introduced, the curvilinear character of the relationship between overall math achievement growth and the Reconfigured Fidelity of Saxon Implementation variable disappears. Specifying a quadratic functional form once the relationship between achievement growth and Saxon implementation level has been purged of the influence of student-level confounding factors produces little useful information. In this instance, the most useful interpretation of the relationship between overall math achievement and the Reconfigured Fidelity of Saxon Implementation measure is as follows: For each one-level increment in the Saxon implementation variable, the posttest score increases, on average, by 1.29 scaled-score points (see Figure A-8 in Appendix A). The relationship remains statistically significant but is weaker than previous estimations lacking studentlevel controls. Math Problem Solving Achievement Growth. The relationship between the Reconfigured Fidelity of Saxon Implementation variable and Math Problem Solving growth is similar to that of overall math achievement. The inclusion of student-level controls linearizes the relationship between achievement test score and level of implementation (see Figure A-9 in Appendix A). In this instance, however, the relationship is not statistically significant. Math Procedures Achievement Growth. As with overall math achievement and Math Problem Solving, when the same complement of studentlevel controls is applied for Math Procedures, the


29

achievement-by-level of Saxon implementation relationship is linearized (see Figure A-10 in Appendix A). In this case, the linear relationship is statistically significant: For each one-level increase in the Reconfigured Saxon Fidelity of Implementation variable, the Math Procedures score increases, on average, by 6.51 points. In summary, controlling for student-level variables, as the fidelity of Saxon Math program implementation increases, there are statistically significant differences in overall math achievement and Math Procedures between students receiving Saxon Math instruction and students receiving other types of math instruction

Reconfigured Fidelity of Saxon Implementation and Overall Math Achievement Growth with Student-Level Controls and School-Level Controls Overall Math Achievement Growth. In all analyses presented previously in this report, the Fidelity of Saxon Implementation and Reconfigured Fidelity of Saxon Implementation variables were treated as if they were characteristics of individual students. In other words, each student attends either a Saxon school or a comparison school, and this distinction was construed as an individual-level trait. However, the Saxon Math program, or another curriculum, is adopted at the level of the school. As a result, both Fidelity of Saxon Implementation and Reconfigured Fidelity of Saxon Implementation variables are best understood as characteristics of schools. Each should be aggregated to the school level and used as a contextual variable, to be used in conjunction with school-level controls as well as student-level controls. In the subsequent analyses, contextual control variables include school size, percentage of each school’s students who are American Indian, percentage of each school’s students who are African American, percentage of each school’s students who are Hispanic, percentage of each school’s students who are migrants, percentage of each school’s students who are eligible for subsidized school meals, and pretest scores aggregated to the school level.

30

This more adequate specification of the relationship between overall math achievement and level of Saxon implementation is linear: For each one-level increase in the Reconfigured Level of Saxon Implementation contextual variable, overall math achievement increases, on average, by 2.24 scaledscore points (see Figure A-11 in Appendix A). While the relationship is not substantively strong, it is statistically significant. The most important finding in this analysis is that when Reconfigured Fidelity of Saxon Implementation is construed as an individual-level variable and only individual controls are employed, the estimate of the payoff for each level of Saxon implementation, 1.29 scaled-score points, is biased downward. The better estimate is 2.24 scaled-score points. A more adequately specified statistical model yields more accurate estimates. Math Problem Solving Achievement Growth. A quadratic function provides a better fit than a linear function when the respecified statistical model—with the Reconfigured Fidelity of Saxon Implementation variable treated as a characteristic of schools and using both individual-level and school-level contextual variables as controls—is applied to the Math Problem Solving subtest. However, the difference between the two is small. The linear functional form is more parsimonious and easier to understand. Thus, each one-level increase in the Reconfigured Fidelity of Saxon Implementation variable corresponds, on average, to 2.52 scaled-score points on the SAT 9 test of problem solving (see Figure A-12 in Appendix A). In other words, controlling for individual- and schoollevel factors, students in schools implementing the Saxon Math program earn a mean of 2.52 scaledscore points on the Math Problem Solving subtest of the SAT 9 per each LoU increase in fidelity of implementation. Construing the Saxon intervention as a school-level variable and introducing a reasonable complement of controls at both the individual level and the school level results in a more adequate specification and a better estimate of the payoff for improved implementation of the Saxon Math program.

Final Report

Math Procedures Achievement Growth. The relationship between fidelity of Saxon Math program implementation and mean student scores on the Math Procedures subtest of the SAT 9 is better represented as a quadratic relationship than as a linear relationship. Moreover, the difference between the two functional forms is small; employing the more parsimonious and readily interpreted linear relationship, it is concluded that each one-level increment in the Reconfigured Fidelity of Saxon Implementation variable corresponds, on average, to a 9.28-point scale score point increase on the SAT 9 Math Procedures test (see Figure A-13 in Appendix A). In other words, controlling for school- and individual-level factors, each level increase in the fidelity of Saxon Math program implementation results in a mean increase of 9.28 points on the Math Procedures subtest. As with the respecified analyses of overall math achievement and Math Problem Solving, treatment of the Saxon Math program as a school-level variable complemented by both individual-level and school-level controls eliminates negative bias in estimating the effects of different levels of implementation. There was an estimated Saxon effect of 6.93 scaled-score points on the SAT 9 9dures test for each additional level of effective Saxon implementation in an earlier analysis, where implementation was treated as a student-level phenomenon. The analysis in which implementation is treated as a school-level phenomenon results in an estimate of the Saxon effect as 9.28 scaled-score points on the Math Procedures test. In summary, when the fidelity of Saxon Math program implementation is considered as a schoollevel variable—and individual-level and school-level controls are added as a part of the analysis— analysis suggests that, as the fidelity of Saxon Math program implementation increases, students receiving Saxon Math instruction outperform students receiving other types of math instruction on overall math achievement, Math Problem Solving and Math Procedures. In other words, as teacher implementation of the Saxon Math program improves, Saxon Math students perform better than students receiving other types of math instruction.


Multilevel Regression Analysis of Saxon Schools’ Performance Bivariate analyses suggest that Saxon students experience substantial gains in overall math achievement, Math Problem Solving, and Math Procedures, as measured by the SAT 9 (see Figures 3–11). A disadvantage of such analyses is that variables occurring at different levels are treated identically, resulting in misspecification of the effects of interventions under study. However, multilevel regression analyses provide superior specification capabilities, with explicit formulation of Level 1 and Level 2 models, permitting selected Level 1 coefficients to vary across Level 2 units. In the process, contextual variables help account for variability in random components of Level 1 coefficients. Ultimately, multilevel regression analyses provide more accurate estimates in nested research designs than do ordinary least squares multiple regression techniques. Multilevel regression analysis is needed to formulate a properly specified statistical model to gauge the relationship between the Reconfigured Fidelity of Saxon Implementation variable and the three SAT 9 measures of math achievement. The use of a two-level regression model (at the individual child and school levels) enables researchers to examine the independent and combined effects of important variables (e.g., special education, English language learner status, racial minority categorization, and free and reduced-price lunch eligibility) on the dependent variable of math achievement. The previous analysis was only of the independent effects of these variables on the dependent variable without taking into statistical account the nested nature of schools. Multilevel regression is warranted in nested contexts because the technique takes into account the potentially different variances at each level. This results in far more accurate analyses because one-level regression models may overestimate the effects of the program being studied. For each one-level increase in the Reconfigured Fidelity of Saxon Implementation variable, overall math achievement increases, on average, by 2.509 scaled-score points (see Table A-7 in Appendix A). Moreover, the effect of implementing Saxon is

31

statistically significant. Thus, it may be concluded that, controlling for meaningful school- and individual-level factors, the overall math achievement of students in schools implementing the Saxon Math program with increasing fidelity is better than that of students in schools not implementing Saxon or implementing Saxon with less fidelity. For each one-level increase in the Reconfigured Fidelity of Saxon Implementation variable, Math Problem Solving increases, on average, by 2.982 scaled-score points (see Table A-8 in Appendix A). This relationship is statistically significant. Controlling for meaningful school- and individuallevel factors, the achievement of students on the Math Problem Solving subtest of the SAT 9 in schools implementing the Saxon Math program with increasing fidelity is better than that of students in schools not implementing Saxon or implementing Saxon with less fidelity. For each one-level increase in the Reconfigured Fidelity of Saxon Implementation variable, Math Procedures increases, on average, by 7.587 scaledscore points (see Table A-9 in Appendix A). Moreover, the positive role Saxon implementation plays on student Math Procedures subtest performance is statistically significant. Thus, controlling for meaningful school- and individuallevel factors, the achievement of students on the Math Procedures subtest of the SAT 9 in schools implementing the Saxon Math program with increasing fidelity is better than that of students in schools not implementing Saxon or implementing Saxon with less fidelity. In summary, the math achievement of students in schools implementing the Saxon Math program with increasing fidelity is better than that of students in schools not implementing Saxon or implementing Saxon with less fidelity. Moreover, the differences in achievement across all three measures of math achievement (overall SAT 9 math achievement, SAT 9 Math Problem Solving subtest, and SAT 9 Math Procedures subtest) are statistically significant. In other words, students whose teachers implement the Saxon Math program at routine or higher levels perform better than students whose teachers used other math programs.

32

Question 5. To what degree of fidelity did the teachers in this study implement the Saxon Math Program? Three measures were used to assess the fidelity with which teachers participating in this study implemented the Saxon Math program. Summary results are presented by data-collection method: videotaped classroom lessons, LoU telephone interviews, and teacher surveys. A more detailed summary of the results is shown in Appendix B.

Videotaped Classroom Lessons Each participating kindergarten through Grade 3 teacher at schools using the Saxon Math program in this study submitted a videotape of a complete Saxon Math lesson of their choice. Videotapes were to include the entire span of time the teacher devoted to teaching math over the course of 1 day. Researchers rated each videotaped classroom lesson using an ICM. In general, the videotaped lessons revealed that most participating teachers implemented the Saxon Math program as intended. Across all grade levels, teachers were least likely to request that students articulate their learnings at the end of lessons. Kindergarten and first-grade teachers were the most likely not to attend to all of the skills to be covered in The Meeting. Second- and third-grade teachers were less likely to use manipulatives. Teachers tended to estimate that The Meeting and the Lesson required more time than they were actually able to devote to them. The average amount teachers spent conducting The Meeting was fairly consistent across grade levels (i.e., 13–18 minutes), although observed times increased slightly by grade level. On average, teachers at all four grade levels made similar estimates of the amount of time required to complete The Meeting (i.e., 17–20 minutes). The average time devoted to the Lesson increased from kindergarten through second grade (i.e., 26–43 minutes), but decreased slightly at the third grade (i.e., 40 minutes). First- through third-grade teachers tended to make similar estimates of the time needed to conduct the Lesson (i.e., 46–48 minutes), whereas kindergarten teachers estimated that the Lesson required approximately 20 fewer minutes (i.e., 27 minutes).

Final Report

Analyses of classroom arrangements employed during videotaped lessons indicated, for the most part, students sat in a semicircle or circle in front of the board for The Meeting. This is the arrangement recommended by the Saxon Math program. This classroom arrangement was less often observed in third-grade classrooms, where students were more likely to be seated at desks in clusters or in an arrangement other than rows. The opposite was true for the Lesson, however. Students were frequently seated at desks in clusters or an arrangement other than rows during the Lesson. Kindergarten students were more likely to remain in a semicircle or circle. Analyses of the Saxon Math program resources and materials used in videotaped lessons revealed that the Meeting Board was employed in nearly all cases. On the other hand, manipulatives were not used as often as might be expected. This was especially true at the second and third grades, in which manipulatives were used in only about half of the lessons observed.

LoU Telephone Interviews LoU interviews were conducted with participating teachers implementing the Saxon Math program. Data from such interviews provided an additional measure of the fidelity with which teachers implemented the program. The majority of teachers (90 out of 128, or 70%) at experimental sites in this study routinely use the Saxon Math Program. According to data from LoU interviews, the program is generally used as intended, with little deviation from the curriculum. A minority of the teachers (29 out of 128, or 23%) were rated at the refinement level. At this level, teachers vary their use of the Saxon Math program somewhat in order to improve student outcomes. The types of changes include using supplemental materials to address standards either not covered or not covered in the depth necessary for students to perform well on state assessments. Teachers also supplied extra materials to reinforce concepts or to extend learning for higher performing students.


Teacher Surveys The third and final measure of Saxon Math implementation was a teacher survey. Participating teachers at comparison schools were also administered a survey about their math instructional practices. Results of both surveys are presented next. Experimental Teacher Survey Results. Survey data indicate that teachers appear to be implementing the Saxon Math program as it was designed to be implemented. However, teachers were not as likely to ask students to articulate their learnings at the end of a lesson, as would be ideal. There was also some variation in how teachers asked students to complete the Guided Practice sheets in first through third grade. Although most teachers use the Saxon Math assessments, fewer teachers use the recording forms or the scoring guides that accompany the assessments. Few teachers also use the Teacher’s Resource CD-ROM and the Lesson Planner CDROM. A number of teachers noted that they use additional materials to supplement the Saxon Math program. Their primary reasons were to reinforce skills covered by Saxon or to target skills required by state standards not attended to through the Saxon Math program. Overall, teachers appeared pleased with the Saxon Math program. The aspects that they like most are the repetition of concepts, the spiraling approach, and the overall design of lessons. Teachers indicated that these, in part, also accounted for their schools’ adoption of the Saxon Math program. Comparison Teacher Survey Results. The majority of participating teachers at comparison sites (86 out of 96, or 90%) teach many of the concepts covered by the Saxon Math program curriculum. Kindergarten teachers at comparison schools reported teaching the least number of the skills that are a part of the Saxon Math program. In general, comparison teachers use written assessment methods. Several also use oral assessment techniques.

33

Comparison teachers use a variety of instructional approaches on a fairly regular basis in their math classrooms. Teachers also reported that students regularly appear engaged and involved in math lessons. Classroom management challenges appear to be infrequent. About one fifth of comparison-site teachers indicated that they expect to complete their math curriculum by the conclusion of the school year. A variety of math curricula are used in the comparison schools. The publishers of the programs tended to be Harcourt Brace, Houghton Mifflin, Silver Burdett Ginn, McGraw-Hill, and Scott Foresman. As with the Saxon Math teachers, the comparison group teachers reported that they use supplemental materials in addition to their adopted math curriculum.

SUMMARY Limitations As with any study, the limitations of the research presented here must be taken into account when interpreting findings. Limitations include the following: • A quasiexperimental design was used in this study. Experimental sites (Saxon Math users) were randomly selected from the population of users. The comparison sites were randomly selected and matched to demographic characteristics of the experimental sites. With a true experimental design, participating sites would have been randomly assigned to one of, in this case, two conditions—use of the Saxon Math program, or nonuse of the program. Experimental designs allow great control over internal validity factors. • Because schools make their own curriculum choices, it was not possible to employ this type of design. • Participating schools and teachers used the Saxon Math program for varying amounts of time. Thus, comparisons based on implementation may be influenced by duration of usage. • Comparison-site teachers use a variety of other types of mathematics curricula and materials.

34

As a result, comparing the aggregated results of this group to the Saxon Math users introduces variation that can not be accounted for. A discussion of the lessons learned through the conduct of the study is contained in Appendix E.

Concluding Comments The results of this study support the claim that the Saxon Math program improves the math achievement of students receiving the curriculum. Participating kindergarten through third-grade students at schools implementing the Saxon Math program made statistically significant gains on the SAT 9 math test and two subtests (Math Problem Solving and Math Procedures). Moreover, the math achievement of students belonging to groups that are commonly regarded as academically, economically, or culturally disadvantaged also increased over the course of the study year. The math achievement of students in schools implementing the Saxon Math program improved despite their initially high achievement on the SAT 9 pretest. Analyses of Saxon students’ performance on Saxon Math program assessments indicate a high percentage of correct answers. In fact, ceiling effects limited the possibilities for achievement gains. This study also suggests that the math achievement of students attending schools implementing the Saxon Math program with a high degree of fidelity is stronger than the achievement of students attending schools not implementing the program at all or implementing it with a lesser degree of fidelity. Comparing Saxon students’ performance with the performance of students from comparison schools, it is clear that implementation fidelity of the Saxon Math program is crucial. Students at schools using Saxon outperform those from comparison schools when implementation exceeds a minimum level. As level of implementation improves, Saxon students’ comparative performance improves as well. Properly specified multilevel regression models indicate that students attending schools implementing the Saxon Math program have higher math achievement than students in comparison schools. The differences are not large, but they are statistically significant.

Final Report

REFERENCES Bickel, R., & Howley, C. W. (2003). Mathematics achievement for rural development: Effects of contextual factors intrinsic to the modern world. Educational Foundations, 17(4), 83–105. Bickel, R., Howley, C., & Maynard, S. (2003). ‘No child left behind’ in poor, Appalachian districts: Confronting contextual factors in the modern world. Journal of Appalachian Studies, 9, 321–340.

Harcourt Educational Measurement. (2000). Stanford Achievement Test (9th ed.): Reviewer’s edition. San Antonio, TX: Author. Hord, S. M., Rutherford, W. L., Huling-Austin, L., & Hall, G. E. (1987). Taking charge of change. Austin, TX: Southwest Educational Development Laboratory. Kennedy, P. (2003). A guide to econometrics. Cambridge, MA: MIT.

Bowles, S., Gintis, H., & Groves, M. (2005). Unequal chances. Princeton, NJ: Princeton University Press.

Larson, N. (2004). Saxon Math teacher’s manual. Orlando, FL: Saxon.

Campbell, D., & Kenny, D. (1995). A primer on regression artifacts. New York: Guilford Press.

Loucks, S. F., Newlove, B. W., & Hall, G. E. (1975). Measuring levels of use of the innovation: A manual for trainers, interviewers, and raters. Austin, TX: Southwest Educational Development Laboratory.

Coleman, J., Campbell, E., Hobson, C., Mcpartland, J., Mood, A., Weinfeld, F., et al. (1966). Equality of educational opportunity. Washington, DC: U.S. Government Printing Office.

No Child Left Behind Act of 2001, 20 U.S.C. 70 § 6301 et seq. (2002)

Francis, D., Schatschneider, C., & Carlson, C. (2000). Introduction to individual growth curve analysis. In D. Drotar (Ed.), Handbook of research in pediatric and clinical child psychology (pp. 51–73). New York: Kluwer Academic/Plenum.

Nunnally, J., & Bernstein, I. (1994). Psychometric theory. New York: McGraw-Hill.

Hall, G. E., Loucks, S. F., Rutherford, W. L., & Newlove, B. W. (1975). Levels of use of the innovation: A framework for analyzing innovation adoption. Journal of Teacher Education, 26(1), 52– 56.

Shadish, W., Cook, T., & Campbell, D. (2002). Experimental and quasi-experimental designs for generalized causal inference. New York: Houghton Mifflin.

Hallinan, M. (2001). Sociological perspectives on black-white inequalities in American schooling (Extra Issue). Sociology of Education, 74, 50–70.

Riordan, C. (2004). Equality and achievement. Upper Saddle River, NJ: Prentice Hall.

Wooldridge, J. (2006). Introductory econometrics: A modern approach. Mason, OH: ThompsonSouthwestern.

Halcoussis, D. (2005.) Understanding econometrics. Knoxville, TN: South-Western.


35


Appendix A Statistical Figures and Tables–Research Questions 3 and 4

Appendix A

Question 3. To what extent does fidelity of Saxon Math program implementation affect math achievement? Fidelity of Saxon Implementation and Overall Math Achievement Growth of Saxon Students Figure A-1 displays the bivariate relationship between Fidelity of Saxon Implementation and overall math achievement growth as measured by the SAT 9. Figure A-1. Average overall math achievement scaled score by Fidelity of Saxon Implementation.

Distribution of Scores at Student Level

3

4

5

6

7

Coefficients for Linear Relationship: b = 26.09 R2 = 6.9% Coefficients for Quadratic Relationship b1 = – 65.32 b2 = 9.60 R2 = 7.6%

S a x o n E l e m e n t a r y M a t h P r o gr a m E f f e c t i v e n e s s S t u d y

2

Appendix A

Fidelity of Saxon Implementation and Math Problem Solving Achievement Growth of Saxon Students Figure A-2 shows that the relationship between Fidelity of Saxon Implementation and Math Problem Solving is also curvilinear, though to a lesser degree than overall math achievement.

Figure A-2. Average Math Problem Solving scaled score by Fidelity of Saxon Implementation.


3

4

5

6

7



3

Appendix A

Fidelity of Saxon Implementation and Math Procedures Achievement Growth of Saxon Students As with the SAT 9 measures of overall math achievement and Math Problem Solving, the relationship between SAT 9 Math Procedures subtest scores and the Fidelity of Saxon Implementation variable is slightly curvilinear in this study, as is illustrated in Figure A-3. Figure A-3. Average Math Procedures scaled score by Fidelity of Saxon Implementation.


3

4

5

6

7



4

Appendix A

Saxon’s Assessment Measures Saxon assessment scores from the beginning, middle, and end of the school year were collected from teachers at Saxon sites participating in the study. Scores were reported as the percentage of correct answers. The relationship between Saxon percentage scores and Fidelity of Saxon Implementation is shown in Figure A-4. Figure A-4. Saxon Math assessments growth by Fidelity of Saxon Implementation.


3

4

5

6

7

Coefficients for Linear Relationship b = 0.49 R2 = 0.0% Coefficient for Quadratic Relationship b1 = – 15.58 b2 = 1.69 R2 = 0.4%


5

Appendix A

Question 4. To what extent are there significant differences in the math achievement of students participating in the Saxon Math program compared to students not participating in the program? Reconfigured Fidelity of Saxon Implementation and Overall Math Achievement Growth for All Students In Figure A-5, the relationship between the SAT 9 measure of overall math achievement and the Reconfigured Fidelity of Saxon Implementation variable, which assigns comparison schools to ordinal category number 2, is shown. Figure A-5. Average overall math achievement scaled score by reconfigured Fidelity of Saxon Implementation (comparison group assigned a value of 2).


2

3

4

5

6

7

Coefficients for Linear Relationship a = 585.50 b = 3.31 R2 = 0.7% Coefficients for Quadratic Relationship a = 642.55 b1 = – 34.17 b2 = 5.31 R2 = 3.0%


6

Appendix A

Reconfigured Fidelity of Saxon Implementation and Math Problem Solving Achievement Growth for All Students The relationship between Math Problem Solving and the Reconfigured Fidelity of Saxon Implementation variable as illustrated in Figure A-6 is very similar to the relationship presented in Figure A-5. Figure A-6. Average Math Problem Solving scaled score by Fidelity of Implementation (comparison group assigned a value of 2).


2

3

4

5

6

7



7

Appendix A

Reconfigured Fidelity of Saxon Implementation and Math Procedures Achievement Growth for All Students The relationship between the Reconfigured Fidelity of Saxon Implementation variable and Math Procedures achievement growth (see Figure A-7) is similar to that between implementation and overall math achievement and Math Procedures achievement shown in Figure A-6. Figure A-7. Average Math Procedures scaled score by Reconfigured Fidelity of Saxon Implementation (comparison group assigned a value of 2).


2

3

4

5

6

7

Coefficients for Linear Relationship: a = 560.86 b = 4.65 R2 = 1.0% Coefficients for Quadratic Relationship a = 610.81 b1 = – 28.16 b2 = 4.65 R2 = 2.4%


8

Appendix A

Reconfigured Fidelity of Saxon Implementation and Overall Math Achievement for All Students with Student-Level Controls As is shown in Figure A-8, specifying a quadratic functional form once the relationship between achievement growth and Saxon implementation level has been purged of the influence of student-level confounding factors produces little useful information. Figure A-8. Average overall math achievement scaled score by Reconfigured Fidelity of Saxon Implementation (comparison group assigned a value of 2 and full complement of controls at Level 1).


2

3

4

5

6

7

Linear Relationship: b = 1.29 R2 = 0.2% Quadratic Relationship b1 = 1.18 b2 = 0.36 R2 = 0.3%


9

Appendix A

Reconfigured Fidelity of Saxon Implementation and Math Problem Solving for All Students with Student-Level Controls The relationship between the Reconfigured Fidelity of Saxon Implementation variable and Math Problem Solving growth is shown in Figure A-9. The relationship displayed in Figure A-9 is similar to that shown in Figure A-8. Figure A-9. Average Math Problem Solving scaled score by Reconfigured Fidelity of Saxon Implementation (comparison group assigned a value of 2 and full complement of controls at Level 1).


2

3

4

5

6

7



10

Appendix A

Reconfigured Fidelity of Saxon Implementation and Math Procedures for All Students with Student-Level Controls The relationship between Math Procedures and the Reconfigured Fidelity of Saxon Implementation variable is displayed in Figure A-10.


2

3

4

5

6

7

Linear Relationship: b = 6.51 R2 = 3.8% Quadratic Relationship b1 = 6.93 b2 = – 1.02 R2 = 3.9%

Figure A-10. Average Math Procedures scaled score by Reconfigured Fidelity of Saxon Implementation (comparison group assigned a value of 2 and full complement of controls at Level 1).


11

Appendix A

Reconfigured Fidelity of Saxon Implementation and Overall Math Achievement for All Students with Student-Level Controls and School-Level Controls Figure A-11 provides a graphical representation of the relationship between overall math achievement at the student level and the Reconfigured Fidelity of Saxon Achievement variable at the school level. Figure A-11. Average overall math achievement scaled score by Reconfigured Fidelity of Saxon Implementation as a school-level variable (comparison group assigned a value of 2 and full complement of controls at Levels 1 and 2).


2

3

4

5

6

7



12

Appendix A

Reconfigured Fidelity of Saxon Implementation and Math Problem Solving for All Students with Student-Level Controls and School-Level Controls When the respecified statistical model—with the Reconfigured Fidelity of Saxon Implementation variable treated as a characteristic of schools and using both individual-level and school-level contextual variables as controls—is applied to the Math Problem Solving subtest, the relationship shown in Figure A-12 results. Figure A-12. Average Math Problem Solving scaled score by Reconfigured Fidelity of Saxon Implementation as a school-level variable (comparison group assigned a value of 2 and full complement of controls at Levels 1 and 2).


2

3

4

5

6

7



13

Appendix A

Reconfigured Fidelity of Saxon Implementation and Math Procedures for All Students with Student-Level Controls and School-Level Controls Figure A-13 similarly shows that the relationship between the fidelity of the Saxon Math program implementation and mean student scores on the Math Procedures subtest of the SAT 9 is better represented as a quadratic relationship than as a linear relationship. Figure A-13. Average Math Procedures growth by Reconfigured Fidelity of Saxon Implementation as a school-level variable (comparison group assigned a value of 2 and full complement of controls at Levels 1 and 2).


2

3

4

5

6

7


Multilevel Regression Analysis of Saxon Schools’ Performance As shown in Tables A-1, A-3, and A-5, multilevel regression analysis is needed to formulate a properly specified statistical model to gauge the relationship between the Reconfigured Fidelity of Saxon Implementation variable and the three SAT 9 measures of math achievement. In each instance, using a one-tailed test, the intraclass correlation coefficient is statistically significant. This suggests that conventional ordinary least squares multiple regression analysis would yield deflated standard errors, making tests of significance unreliable. In addition, selected student-level coefficients do, in fact, vary from school to school; this variability should be built into multilevel equations by assigning a random component to one or more coefficients. Finally, the statistically significant intraclass correlation coefficients also


14

Appendix A

indicate that contextual variables may be useful in explaining variability in student-level random components. Tables A-2, A-4, and A-6 enable the estimation of conditional intraclass correlation coefficients for each of the three independent variables. Conditional intraclass correlation coefficients are estimated with contextual variables in the multilevel regression equations. In each instance, the contextual variables render the conditional intraclass coefficients weaker than the unconditional coefficients. In the case of overall math achievement and Math Problem Solving, the conditional coefficient is reduced to statistical nonsignificance, confirming that the use of a multilevel regression model represents a better choice than a conventional ordinary least squares regression model. Table A-1. Intraclass Correlation for Overall Math Achievement Parameter Estimate Standard Wald Z Sig. Level Error Variance within 3590.99 134.78 26.64 .00 schools 178.35 61.85 2.88 .00 Variance between schools Note. Intraclass correlation = variance between schools/total variance = 178.35/(178.35 + 3,590.99) = .047 Table A-2. Conditional Intraclass Correlation for Overall Math Achievement Parameter

Estimate

Standard Error 164.16

Wald Z

Sig. Level

Variance within 3,721.82 22.67 .00 schools 83.33 60.67 1.37 .17 Variance between schools Note. Conditional intraclass correlation = variance between schools/total variance = 83.33/(83.33 + 3,721.82) = .021.


15

Appendix A

Table A-3. Intraclass Correlation for Math Problem Solving Parameter Estimate Standard Wald Z Sig. Level Error Variance within 2,723.37 183.60 14.83 .00 schools 179.39 97.22 1.85 .07 Variance between schools Note. Intraclass correlation = variance between schools/total variance = 179.39/(179.39 + 2,723.37) = .062. Table A-4. Conditional Intraclass Correlation for Math Problem Solving Parameter Estimate Standard Wald Z Sig. Level Error Variance within 2,729.92 184.54 14.79 .00 schools Variance 115.05 101.02 1.139 .26 between schools Note. Conditional intraclass correlation = variance between schools/total variance = 115.05/(115.05 + 2,729.92) = .040. Table A-5. Intraclass Correlation for Math Procedures Parameter Estimate Standard Wald Z Sig. Level Error Variance within 3,328.54 169.75 19.609 .00 schools 170.54 75.18 2.27 .03 Variance between schools Note. Intraclass correlation = variance between schools/total variance = 170.54/(170.54 + 3,328.54) = .049.


16

Appendix A

Table A-6. Conditional Intraclass Correlation for Math Problem Solving Parameter Estimate Standard Wald Z Error Variance 3328.66 169.79 19.60 Within Schools 152.04 88.54 1.72 Variance Between Schools

Sig. Level .000 .076

Note. Intraclass correlation = variance between schools/total variance = 152.04/(152.04 + 3,328.66) = .043. Having made the case for multilevel regression, the properly specified statistical models can be estimated that provide accurate estimates of payoffs for effective Saxon implementation. In Tables A-7, A-8, and A-9, it is shown that using one-tailed t tests, each of the Saxon implementation coefficients (Saxon Implement3) are statistically significant and positive.

Table A-7. Overall Math Achievement: Fixed Components Parameter Estimate Standard t Value Error Intercept 571.236 5.117 111.635 Minor1 – 9.391 3.362 – 2.793 Gradelvl 16.316 1.661 9.822 Poor1 – 4.329 2.922 – 1.482 Special1 – 6.488 4.833 – 1.343 Languag1 7.450 4.452 1.673 Pretest1 0.641 0.030 21.052 Size3 0.009 0.009 1.002 Native3 1.046 1.266 0.826 Black3 0.203 0.086 2.360 Asian3 1.735 1.912 1.456 Poor3 – 0.116 0.880 – 1.324 Migrant3 – 0.204 0.367 – 0.556 Saxon 2.509 1.369 1.833 Implement3

Sig. Level .000 .005 .000 .139 .180 .095 .000 .328 .418 .026 .159 .197 .585 .074

R12 = 76.1% N1 = 1,066 N2 = 57 Deviance Difference (from null model) = 7,260.38 df = 13 Note. Statistically significant results for one-tailed tests are bolded and italicized.


17

Appendix A

Table A-8. Math Problem Solving: Fixed Components Parameter Estimate Standard Error Intercept 613.614 5.584 Minor1 – 7.226 3.393 Gradelvl 17.139 2.740 Poor1 – 10.622 3.362 Special1 – 14.783 5.339 Languag1 – 3.824 5.380 Pretest1 0.631 0.036 Size3 0.009 0.009 Native3 1.046 1.266 Black3 0.203 0.086 Asian3 1.735 1.912 Poor3 – 0.116 0.880 Migrant3 – 0.204 0.367 Saxon 2.982 1.596 Implement3

t Value 109.888 – 2.130 6.255 – 3.159 –2.769 – 0.771 17.441 1.002 0.826 2.360 1.456 – 1.324 – 0.556 1.869

Sig. Level .000 .034 .000 .002 .006 ..477 .000 .328 .418 .026 .159 .197 .585 .070

R12 = 60.5% N1 = 816 N2 = 57 Deviance Difference (from null model) = 517.539 df = 13 Note. Statistically significant results for one-tailed tests are bolded and italicized.


18

Appendix A

Table A-9. Math Procedures: Fixed Components Parameter Estimate Standard Error Intercept 581.443 6.617 Minor1 – 8.774 4.162 Gradelvl 17.739 3.387 Poor1 – 12.272 4.132 Special1 – 14.783 5.339 Languag1 5.610 6.645 Pretest1 0.451 0.034 Size3 0.004 0.012 Native3 – 0.009 0.357 Black3 0.007 0.108 Asian3 0.406 1.187 Poor3 0.045 0.103 Migrant3 0.218 0.650 Saxon 7.587 2.260 Implement3

t Value 76.331 – 2.108 5.237 – 3.159 – 3.369 0.844 13.194 0.303 – 0.027 0.070 0.342 0.435 0.356 3.356

Sig. Level .000 .035 .000 .002 .001 .399 .000 .763 .979 .945 .735 .666 .724 .002

R12 = 62.1% N1 = 816 N2 = 57 Deviance Difference (from null model) = 5,508.861 df = 13 Note. Statistically significant results for one-tailed tests are bolded and italicized.


19


Appendix B Detailed Results of Research Question 5–Implementation Fidelity

Appendix B

Videotaped Classroom Lessons Each participating K–3 teacher at schools using the Saxon Math program in this study submitted a videotape of a complete Saxon Math lesson of their choice during January, February, or March 2006. Videotapes were to include the entire span of time the teacher devoted to teaching math over the course of 1 day—including The Meeting, Lesson, Guided or Lesson Practice, Fact Practice, Handwriting Practice, and Counting Practice, if applicable. Teachers were also asked to submit a brief information sheet describing the lesson number and topic, time of day The Meeting and Lesson were conducted, and the perceived and actual amount of time to conduct The Meeting and Lesson. Researchers rated each videotaped classroom lesson using an ICM. An overall summary of ICM results across the sample is presented in Table B-1, followed by individual grade-level summaries. The majority of lessons were given a “4” rating on each of the components in the areas assessed. A rating of “4” indicates that the Saxon Math program curriculum was implemented as it is designed to be implemented. This was not the case for students sharing learnings (component 12), however. In nearly 80% of the videotaped lessons, students were not given an opportunity to share what they had learned (variation 1). In addition, teachers asked few questions of the students during Number Fact Practice (variation 1) in half of the videotaped lessons. Table B-1. ICM Ratings for Kindergarten Through Third-Grade Saxon Math Lessons Component 4 3 2 The Meeting 1. Practice of skills (n = 116) 2. Use of manipulatives (n = 116) 3. Frequency of questioning (n = 115) 4. Use of wait time (n = 115) 5. Classroom management (n = 116) Lesson 6. Stating objective (n = 120) 7. Modeling concepts (n = 120) 8. Use of manipulatives (n = 120) 9. Modeling use of manipulatives (n = 83) 10. Frequency of questioning (n = 121) 11. Use of wait time (n = 121) 12. Students sharing learnings (n = 115) 13. Classroom management (n = 120) Guided or Lesson Practice 14. Completing Guided or Lesson Practice sheet (n = 90)

1

43% 86% 84% 90% 78%

33% 10% 14% 10% 21%

19% 1% 2% — 2%

5% 3% — — —

75% 77% 62% 55%

10% 16% 10% 31%

3% 3% — 11%

12% 4% 28% 2%

78% 87% 18% 64%

17% 12% — 31%

3% 1% 3% 2%

2% 1% 79% 3%

10%

1%

61%


28%

2

Appendix B

15. Time allowed for completing Guided Practice Sheet (n = 70) (Gr 1–3 only) 16. Teacher interaction during Guided or Lesson Practice sheet completion (n = 86) Number Fact Practice (Gr 1–3) 17. Frequency of questioning (n = 62) 18. Use of manipulatives (n = 63) 19. Completing class Fact Practice sheet (n = 63) 20. Teacher interaction during class Fact Practice sheet (n = 53) Handwriting Practice (K) 21. Completing Handwriting Practice sheet (n = 13) 22. Teacher interaction during Handwriting Practice sheet completion (n = 15) Counting Practice (K) 23. Completing Counting Practice sheet (n = 5) 24. Teacher interaction during Counting Practice sheet completion (n = 4)

86%

4%

9%

1%

56%

17%

26%

1%

24% 68% 83%

8% 3% 5%

18% — 5%

50% 29% 8%

70%

19%

—

11%

85%

—

15%

—

73%

13%

13%

—

80%

—

20%

—

25%

—

75%

—

Note. Due to rounding, percentages may not equal 100%. Not all components observed in each lesson; consequently, n values are less than 125 (the number of tapes submitted). Two videotapes were nonfunctional and are not included in this analysis. Six (6) of the videotapes submitted did not include a taping of The Meeting. Two (2) videotapes did not include a taping of the Lesson. Thirty (30) videotaped lessons did not contain Fact Practice. Not all videotaped lessons for Grades 1-3 were to have Number Fact Practice as a part of that particular lesson. However, 21 of the 30 lessons should have included Number Fact Practice but were conducted without them. Twenty-nine (29) videotaped lessons did not contain completion of the Guided Practice sheet (Grades 1–3) or Lesson Practice sheet (kindergarten). It is not known whether Fact Practice, Guided Practice, or Lesson Practice was omitted from these lessons or the teacher elected not to videotape that part of the Lesson. Not all kindergarten videotaped lessons were intended to include Handwriting or Counting Practice. Consequently, few videotaped kindergarten lessons show either of these two elements (n = 15 Handwriting Practice and n = 5 Counting Practice). One videotape included only the administration of the student written and fact assessments (i.e., no Meeting or Lesson was shown). One videotape included a math lesson, but there was no evidence that it was a Saxon lesson, although students completed the Guided Practice sheet.

Teachers were asked to indicate how long it should take to conduct The Meeting and also the Lesson. Tables B-2 and B-3 illustrate these findings. Researchers recorded the time actually spent conducting The Meeting and the Lesson on each videotape. More than a third of teachers (36%) reported that The Meeting would require 15 to19 minutes. Nearly a third (31%) thought The Meeting would last 20 to 24 minutes.


3

Appendix B

Observations revealed, however, that more than a third (35%) spent 10 to14 minutes conducting The Meeting. The mean amount of time participating teachers used to conduct The Meeting was 15.3 minutes, with a standard deviation of 6.3 minutes, whereas the mean amount of time teachers perceived necessary to conduct The Meeting was 18.7 minutes, with a standard deviation of 8.7 minutes. A paired samples t test indicates that the differences in perception and reality are statistically significant at the .01 level (p = .00). Table B-2. Perceived Versus Observed Length of Time for The Meeting Minutes Perceived (n = 119) Observed (n = 115) 0–4 — — 5–9 2% 17% 10–14 15% 35% 15–19 36% 25% 20–24 31% 14% 25–29 2% 8% 30–34 8% 1% 35–39 2% 1% 40–44 — — 45–49 2% — 50–54 1% — 55–59 — — 60–64 1% — Note: Due to rounding, percentages may not equal 100%.

There was a fair amount of variation in the amount of time teachers thought it should take to conduct the Lesson. The majority (23%) reported that the lesson should take 60 to 64 minutes to complete. One fifth (20%) indicated that the Lesson should take 30 to 34 minutes, and another 19% thought it should take 45 to 49 minutes. Not only did teachers have different perceptions of the amount of time needed to conduct the Lesson, videotaped lessons revealed variations in the amount of time teachers actually used to do so. Many teachers conducted their lessons in 30 to 34 minutes (16%) or 40 to 44 minutes (16%). Observations revealed that the average amount of time in which participating teachers conducted the Lesson was 36.2 minutes, with a standard deviation of 13.6 minutes. The average amount of time teachers perceived lessons to require was 42.7 minutes, with a standard deviation of 16.8. A paired samples t test indicated that the differences between teachers’ perceptions of the time required and observed amount of time used were statistically significant (p = .000).


4

Appendix B

Table B-3. Perceived Versus Observed Length of Time for the Lesson Minutes Perceived (n = 119) Observed (n = 121) 0–4 — — 5–9 — 1% 10–14 1% 3% 15–19 2% 4% 20–24 8% 13% 25–29 7% 13% 30–34 20% 16% 35–39 3% 12% 40–44 8% 16% 45–49 19% 9% 50–54 2% 5% 55–59 2% 4% 60–64 23% 1% 65–69 1% — 70–74 2% 3% 75 or more 3% — Note. Due to rounding, percentages may not equal 100%.

In more than half of the videotapes (58%), students were observed to be seated on the classroom floor in a semicircle or circle in front of the board for The Meeting. A majority of the lessons (44%) were conducted with students seated in desks, clusters, or arrangements other than rows. Table B-4. The Meeting and Lesson Classroom Arrangement Classroom Arrangement Students seated on floor in semicircle or circle in front of the board Students seated in chairs in semicircle or circle in front of the board Students seated in desks in rows Students seated in desks in clusters or other arrangement other than rows Other:

The Meeting (n = 125) 58% 6% 10% 15% 2%

Lesson (n = 125) 19% 2% 16% 44% 10%

The Meeting: • Students in walled cubicle desks in clusters • Students seated around a table (n = 2) Lesson: • First part on floor in circle and then at tables in groups of four • On floor in circle and then at desks in clusters • Students seated at long tables in a semicircle (n = 2) • Seated in chairs around a round table facing the board • Students seated in chairs around table (n = 3)


5

Appendix B

• •

Seated on floor for half of lesson and then seated in rows Both seated at desks for latter half of lesson and seated on floor around teacher’s rocking chair for first half • Students in desks but arrangement not observable • Students in walled cubicle desks in clusters Did not observe

9%

9%

Table B-5 presents the Saxon Math program resources and materials observed in videotaped lessons. It should be noted that not all resources and materials would be used in every lesson. In at least half of the lessons observed, The Meeting Board (85%), kindergarten Lesson Practice sheet (75%), manipulatives for the Lesson (62%), manipulatives for The Meeting (56%), firstthrough third-grade Guided Practice sheet (54%) or kindergarten Handwriting Practice sheet (54%) were employed. Table B-5. Use of Resources and Materials Resources & Materials (Grade)

Percentage Used in Lesson (N = 125 K–3; n = 28 K; n = 33 1st; n = 32 2nd; n = 32 3rd)

The Meeting Board (K–3) Student Lesson Practice Sheet (K) Manipulatives (the Lesson) (K–3) Manipulatives (The Meeting) (K–3) Student Guided Practice sheet (1–3) Student Handwriting Practice sheet (K) Student Class Fact Practice sheet (1–3) Poster-Size Lesson Charts (1–3) Student Math Folders (2–3) Lesson Masters (for students) (1–3) Learning Wrap-UpsTM (1–3) Meeting Masters (for students) (2–3) Teacher Fact Cards (1–3) Student Counting Practice Sheet (K) Student Number Lines (K) Student Written Assessments (1–3) Student Number Cards (K) Math Center Materials and Activities (K) Student Fact Cards (1–2) Student Math Offices (1st)

85% 75% 62% 56% 54% 54% 42% 40% 37% 28% 27% 24% 24% 18% 7% 6% 4% 4% 3% 2%

Researchers also recorded the videotaped use of resources and materials other than those associated with the Saxon Math program. These data are presented in the summaries of gradelevel ICM results, which follow.


6

Appendix B

Kindergarten ICM Results Summary A majority of lessons were given a “4” rating on each of the components in the areas assessed, as illustrated in Table B-6. A rating of “4” indicates that the Saxon Math program curriculum was implemented as designed. This was not the case for the practice of The Meeting skills (component 1), however. In one third of the videotaped lessons (33%), The Meeting focus was entirely on the practice of skills intended to be taught during The Meeting. However, in another one third of the videotaped lessons (33%), The Meeting focus was on either half of the skills (variation 2) or fewer (variation 3). Rarely did teachers ask students to share their learnings at the conclusion of a lesson (component 12). In only 7% of the videotaped lessons did this occur. In general, when students completed the Counting Practice sheet, teachers sat at their desks or stood in the front of the room and responded to questions (component 19, variation 2). This was the case in 75% of the observed classroom lessons. This component was observed only in four lessons, however, so findings might not be generalizable. Table B-6. ICM Ratings for Kindergarten Saxon Math Lesson Component 4 3 The Meeting 1. Practice of skills (n = 27) 2. Use of manipulatives (n = 27) 3. Frequency of questioning (n = 26) 4. Use of wait time (n = 26) 5. Classroom management (n = 26) Lesson 6. Stating objective (n = 28) 7. Modeling concepts (n = 28) 8. Use of manipulatives (n = 28) 9. Modeling use of manipulatives (n = 25) 10. Frequency of questioning (n = 28) 11. Use of wait time (n = 28) 12. Students sharing learnings (n = 28) 13. Classroom management (n = 28) Lesson Practice 14. Completing lesson practice sheet (n = 20) 15. Teacher interaction during lesson practice sheet completion (n = 21) Handwriting Practice 16. Completing Handwriting Practice sheet (n = 13) 17. Teacher interaction during Handwriting Practice sheet completion (n = 15)

2

1

33% 82% 89% 89% 69%

33% 11% 12% 12% 27%

22% 4% — — 4%

11% 4% — — —

86% 68% 89% 72% 86% 93% 7% 54%

14% 18% 4% 20% 7% 7% — 39%

— 4% — 8% 4% — 7% 4%

— 11% 7% — 4% — 86% 4%

60%

25%

15%

—

71%

14%

14%

—

85%

—

15%

—

73%

13%

13%

—


7

Appendix B

Counting Practice 18. Completing Counting Practice sheet (n = 5) 19. Teacher interaction during Counting Practice sheet completion (n = 4)

80%

—

20%

—

25%

—

75%

—

Note. Due to rounding, percentages may not equal 100%. Not all components were observed in each lesson; consequently, n values are less than 28. One (1) of the videotapes submitted did not include a taping of The Meeting. Seven (7) videotapes did not include a taping of Lesson Practice. It is not known whether Lesson Practice was omitted from these lessons or the teacher elected not to have that part of the lesson videotaped. Not all videotaped lessons were to have Handwriting or Counting Practice as a part of that particular lesson. Consequently, few submitted videotaped lessons contain either of these two elements (n = 15 Handwriting Practice and n = 5 Counting Practice).

Kindergarten teachers were asked to estimate the amount of time The Meeting and the Lesson required. Information is displayed in Tables B-7 and B-8. Researchers also recorded the time teachers spent conducting The Meeting and the Lesson on each videotape. The majority of teachers (41%) reported their perception that The Meeting required between 15 and 19 minutes. Observations, however, revealed that the majority of teachers (44%) devoted 10 to 14 minutes to The Meeting. On average, The Meeting was conducted in 13.4 minutes, with a standard deviation of 5.4. The average amount of time teachers perceived necessary to conduct The Meeting was 17.3 minutes, with a standard deviation of 7.5. The differences between perceived and observed amounts of time required were statistically significant at the .01 level (p = .01). Table B-7. Kindergarten: Perceived Versus Observed Length of Time for The Meeting Minutes Perceived (n = 27) Observed (n = 27) 0–4 — — 5–9 4% 15% 10–14 22% 44% 15–19 41% 22% 20–24 15% 15% 25–29 — 4% 30–34 15% — 35–39 4% — Note. Due to rounding, percentages may not equal 100%.

The majority of kindergarten teachers (43%) thought the Lesson should take 30 to 34 minutes to complete (see Table B-8). Observations indicated that many teachers required 25 to 29 minutes to conduct the Lesson (25%) or 20 to 24 minutes (21%). The average time for conducting the Lesson was 25.8 minutes, with a standard deviation of 10.2. The average amount of time teachers perceived it should have taken was 27.3 minutes, with a standard deviation of 8.2 minutes. However, the differences between teachers’ estimates and observations were not statistically significant at the .01 level (p = .04).


8

Appendix B

Table B-8. Kindergarten: Perceived Versus Observed Length of Time for the Lesson Minutes Perceived (n = 28) Observed (n = 28) 0–4 — — 5–9 — 4% 10–14 4% 11% 15–19 7% 7% 20–24 18% 21% 25–29 14% 25% 30–34 43% 18% 35–39 4% 4% 40–44 4% 7% 45–49 7% — 50–54 — 4% Note: Due to rounding, percentages may not total to 100%.

Videotaped lessons were analyzed in terms of classroom arrangement. The majority of kindergarten students were seated on the floor in a semicircle or circle in front of the chalkboard for both The Meeting (93%) and the Lesson (61%), as illustrated in Table B-9. Table B-9. The Meeting and Lesson Classroom Arrangement Classroom Arrangement

Students seated on floor in semicircle or circle in front of the board Students seated in chairs in semicircle or circle in front of the board Students seated in desks in rows Students seated in desks in clusters or other arrangement other than rows Other: Lesson: • First part on floor in circle and then at tables in groups of 4 • On floor in circle then at desks in clusters Did not observe

The Meeting (n = 28) 93% 4%

Lesson (n = 28)

— —

— 25%

—

7%

4%

—

61% 7%

The percentage of kindergarten lessons in which various Saxon Math program resources and materials were observed in use are presented in Table B-10. It should be noted that not all resources and materials should have been present, as these items are specific to certain activities (e.g., Handwriting Practice, Counting Practice). Three fourths or more of the lessons used The Meeting Board (93%), manipulatives for the Lesson (93%), manipulatives for The Meeting (89%), and the Lesson Practice sheets (75%).


9

Appendix B

Table B-10. Use of Resources and Materials Resources & Materials

The Meeting Board Manipulatives (Lesson) Manipulatives (The Meeting) Student Lesson Practice sheet Student Handwriting Practice sheet Student Counting Practice sheet Student Number Lines Student Number Cards Math Center Materials and Activities

Percentage Used in Lesson (n = 28) 93% 93% 89% 75% 54% 18% 7% 4% 4%

First-Grade ICM Results Summary A total of 20 components were observed in videotaped first-grade math lessons. As shown in Table B-11, researchers used the four implementation variations. More than half of the lessons were given a rating of “4” on 17 of the 20 components. A rating of “4” indicates that teachers implemented the Saxon Math program curriculum as designed. One of the three exceptions to this rating was the component assessing practice of The Meeting skills (component 1). The Meeting focus was on approximately only half of the skills to be taught in The Meeting (variation 2) in half of the videotaped lessons. Teachers did not request that students share their learnings in nearly three fourths of the lessons (74%) (component 12, variation 1). More than half (52%) of the teachers sat at their desks or stood in the front of the classroom responding to questions about the Guided Practice sheet (variation 2) instead of circulating around the room and checking students’ papers (variation 4). In 41% of observed lessons, the teacher did not make any reference to the lesson objective (component 6, variation 1). Table B-11. ICM Ratings for First-Grade Saxon Math Lessons Component 4 3 The Meeting 1. Practice of skills (n = 32) 2. Use of manipulatives (n = 32) 3. Frequency of questioning (n = 32) 4. Use of wait time (n = 32) 5. Classroom management (n = 32) Lesson 6. Stating objective (n = 32) 7. Modeling concepts (n = 32) 8. Use of manipulatives (n = 31) 9. Modeling use of manipulatives (n = 26) 10. Frequency of questioning (n = 32) 11. Use of wait time (n = 32)

2

1

34% 19% — — 9%

50% — — — —

6% — — — —

53% 84% 77% 69%

3% 9% 7% 23%

3% 3% — 4%

41% 3% 16% 4%

88% 97%

6% —

3% —

3% 3%

9% 81% 100% 100% 91%


10

Appendix B

12. Students sharing learnings (n = 31) 13. Classroom management (n = 31) Guided Practice 14. Completing Guided Practice sheet (n = 27) 15. Time allowed for completing Guided Practice sheet (n = 27) 16. Teacher interaction during Guided Practice sheet completion (n = 27) Number Fact Practice 17. Frequency of questioning (n = 12) 18. Use of manipulatives (n = 13) 19. Completing class Fact Practice sheet (n = 13) 20. Teacher interaction during class Fact Practice sheet (n = 12)

26% 81%

— 10%

— —

74% 10%

78%

19%

—

4%

93%

4%

—

4%

37%

7%

52%

4%

67% 77% 69%

— — —

8% — 15%

25% 23% 15%

75%

—

—

25%

Note. Due to rounding, percentages may not equal 100%. Not all components were observed in each lesson; consequently, n values are less than 33. One of the videotapes submitted did not include a taping of The Meeting or the Lesson. That videotape included administration of the student written and fact assessments. One videotape depicted a math lesson, but there was no evidence that a Saxon Math program lesson was used (although the teacher did ask students to complete a Guided Practice sheet). Twenty (20) videotaped lessons did not contain Fact Practice. Not all videotaped lessons were to have Number Fact Practice as a part of that particular lesson. However, in 14 of the 20 lessons that were to have Number Fact Practice included, the teachers did not do so. Five (5) videotaped lessons did not contain completion of the Guided Practice sheet. It is not known whether Fact Practice and Guided Practice were omitted from these lessons or the teacher elected not to have that part of the lesson videotaped. One videotaped lesson analyzed with the first-grade group was a kindergarten teacher. However, the school in which this kindergarten teacher works uses the next grade-level materials. Therefore, it was more appropriate to use the first-grade ICM to rate the videotaped lesson.

First-grade teachers were asked to estimate the length of time required to conduct The Meeting and the Lesson, and their responses are displayed in Tables B-12 and B-13. Researchers then recorded the time teachers spent conducting The Meeting and the Lesson as seen on videotapes of lessons. The majority of teachers (47%) reported that The Meeting required 20 to 24 minutes. Observers, however, recorded that the majority of teachers (41%) spent 10 to14 minutes conducting The Meeting. The mean observed amount of time to conduct The Meeting was 14.9 minutes, with a standard deviation of 6.4 minutes. The average amount of time perceived by teachers needed to conduct The Meeting was 20.3 minutes, with a standard deviation of 8.8. These differences were statistically significant at the .01 level (p = .00).


11

Appendix B

Table B-12. First Grade: Perceived Versus Observed Length of Time for the Meeting Minutes Perceived (n = 32) Observed (n = 32) 0–4 — — 5–9 3% 19% 10–14 9% 41% 15–19 22% 22% 20–24 47% 13% 25–29 3% 3% 30–34 6% — 35–39 3% 3% 40–44 — — 45–49 6% — Note: Due to rounding, percentages may not equal 100%.

The majority of first-grade teachers (28%) estimated that the Lesson should take 30 to 34 minutes to complete (see Table B-13). Another significant percentage of teachers (25%) reported that the Lesson required 60 to 64 minutes to complete. Observations of videotaped lessons revealed that many teachers used only 20 to 24 minutes to conduct the Lesson (22%); another 22% used 35 to 39 minutes. The average observed time devoted to conducting the Lesson was 35.8 minutes, with a standard deviation of 11.8. The average amount of time teachers perceived to be required for the Lesson was 45.8 minutes, with a standard deviation of 17.7. Such differences between teacher perceptions of time required and researcher observations were statistically significant at the .01 level (p = .00). Table B-13. First Grade: Perceived Versus Observed Length of Time for the Lesson Minutes Perceived (n = 32) Observed (n = 32) 0–4 — — 5–9 — — 10–14 — — 15–19 — — 20–24 3% 22% 25–29 3% 9% 30–34 28% 16% 35–39 3% 22% 40–44 13% 9% 45–49 16% 16% 50–54 — — 55–59 — 3% 60–64 25% — 65–69 — — 70–74 3% 3% 75 or more 6% 0 Note. Due to rounding, percentages may not total to 100%.


12

Appendix B

As Table B-14 shows, first-grade students were seated on the classroom floor in a semicircle or circle in front of the chalkboard for The Meeting in the majority of videotaped lessons (55%). Students were seated at desks in clusters or in some arrangement other than rows for the Lesson for most of the remaining videotaped lessons (42%). Table B-14. The Meeting and Lesson Classroom Arrangement Classroom Arrangement

Students seated on floor in semicircle or circle in front of the board Students seated in chairs in semicircle or circle in front of the board Students seated in desks in rows Students seated in desks in clusters or other arrangement other than rows Other Did not observe

The Meeting (n = 33) 55%

Lesson (n = 33)

21%

—

12% 6%

27% 42%

— 6%

— 12%

18%

The percentages of lessons in which various Saxon Math program resources and materials were observed are featured in Table B-15. It was not expected that all resources and materials would be observed, as some items align only with certain activities. Teachers in at least 70% of observed lessons used The Meeting Board (82%), poster-size lesson charts (76%), manipulatives for the Lesson (70%), and the Guided Practice sheet (70%). Table B-15. Use of Resources and Materials Resources & Materials

The Meeting Board Poster-Size Lesson Charts Manipulatives (the Lesson) Student Guided Practice sheet Manipulatives (The Meeting) Lesson Masters (for students) Student Class Fact Practice sheet Learning Wrap-Ups Student Fact Cards Student Written Assessments Teacher Fact Cards Student Math Offices

Percentage Used in Lesson (n = 33) 82% 76% 70% 70% 61% 49% 33% 15% 12% 9% 6% 6%


13

Appendix B

Second-Grade ICM Results Summary As with other grade levels, second-grade videotaped lessons were analyzed in terms of the extent to which they adhered to the Saxon Math program curriculum and structure. As shown in Table B-16, twenty components were rated in terms of four implementation variations. More than half of the lessons were rated a “4,” indicating that the Saxon Math program curriculum was implemented as designed in 16 of the 20 components assessed. The four exceptions to this trend included use of manipulatives (component 8), students sharing learnings (component 12), frequency of questioning (component 17), and completing class Fact Practice sheet (component 19). In half of the second-grade lessons observed (50%), manipulatives were not used (component 8, variation 1). In more than three fourths of the lessons (78%), teachers did not ask students to articulate what they had learned (component 12, variation 1). Little questioning occurred during Fact Practice (component 17, variation 1) in nearly half of the videotaped second-grade lessons (48%). Finally, teachers did not ask students to complete Fact Practice sheet (Side A) during class (component 19, variation 1) in 74% of observations. Table B-16. ICM Ratings for Second-Grade Saxon Math Lessons Component 4 3 The Meeting 1. Practice of skills (n = 28) 2. Use of manipulatives (n = 28) 3. Frequency of questioning (n = 27) 4. Use of wait time (n = 27) 5. Classroom management (n = 28) Lesson 6. Stating objective (n = 29) 7. Modeling concepts (n = 30) 8. Use of manipulatives (n = 30) 9. Modeling use of manipulatives (n = 15) 10. Frequency of questioning (n = 30) 11. Use of wait time (n = 30) 12. Students sharing learnings (n = 27) 13. Classroom management (n = 30) Guided Practice 14. Completing Guided Practice sheet (n = 23) 15. Time allowed for completing Guided Practice sheet (n = 23) 16. Teacher interaction during Guided Practice sheet completion (n = 21)

2

1

57% 89% 70% 78% 68%

32% 7% 22% 22% 29%

10% 4% 7% — 4%

— — — — —

83% 90% 33% 27%

14% 10% 17% 53%

— — — 20%

3% — 50% —

70% 70% 22% 50%

23% 27% — 50%

7% 3% — —

— — 78% —

65%

30%

4%

—

96%

—

4%

—

76%

14%

10%

—


14

Appendix B

Component Number Fact Practice 17. Frequency of questioning (n = 25) 18. Use of manipulatives (n = 24) 19. Completing class Fact Practice sheet (n = 23) 20. Teacher interaction during class Fact Practice sheet (n = 20)

4

3

2

1

16% 83% 9%

12% 4% 4%

24% — 13%

48% 13% 74%

60%

30%

—

10%

Note. Due to rounding, percentages may not equal 100%. Not all components were observed in each lesson; consequently, n values are less than 30. Two videos were nonfunctional and could not be rated. The above analysis is based on 30 videotaped lessons. Two (2) of the videotapes submitted did not include a taping of The Meeting. Five (5) videotapes did not contain Number Fact Practice. However, four of the five lessons should have included Number Fact Practice. Seven (7) videotapes did not contain Guided Practice. It is not known whether Fact Practice and Guided Practice were omitted from these lessons or the teacher elected not to videotape that part of the Lesson. One video was of a first-grade teacher. However, the school at which this teacher is employed uses the next gradelevel materials. The second-grade ICM was therefore used to rate the videotaped lesson.

Second-grade teachers were asked to estimate the time needed to conduct The Meeting and the Lesson. In addition, researchers watched each videotaped lesson and recorded the time teachers actually devoted to conducting The Meeting and the Lesson. These data are illustrated in Tables B-17 and B-18. The majority of teachers (41%) reported that The Meeting required 15 to 19 minutes. Observations of videotaped lessons revealed that the majority of teachers (29%) did, in fact, devote 15 to 19 minutes to The Meeting. However, 25% of teachers conducted The Meeting in 10 to 14 minutes, and another 25% in 5 to 9 minutes. The mean observed time second-grade teachers devoted to The Meeting was 15.0 minutes, with a standard deviation of 6.7. The average amount of time perceived necessary to conduct The Meeting was 18.2 minutes, with a standard deviation of 9.1 minutes. These differences were not statistically significant at the .01 level (p = .16). Table B-17. Second Grade: Perceived Versus Observed Length of Time for The Meeting Minutes Perceived (n = 29) Observed (n = 30) 0–4 — — 5–9 — 25% 10–14 21% 25% 15–19 41% 29% 20–24 21% 14% 25–29 3% 4% 30–34 7% 4% 35–39 3% — 40–44 — — 45–49 — — 50–54 3% — Note. Due to rounding, percentages may not total 100%.


15

Appendix B

The majority of second-grade teachers (38%) reported that the Lesson required 45 to 49 minutes or more to complete. Observations indicated that many teachers used 40 to 44 minutes for the Lesson (20%). This information is displayed in Table B-18. On average, second-grade teachers devoted 43.2 minutes to conducting the Lesson, with a standard deviation of 15.5 minutes. The average amount of time teachers perceived the Lesson to require was 48.3 minutes, with a standard deviation of 14.2. Differences between mean perceived and observed amount of time needed to conduct the Lesson were not statistically significant at the .01 level (p = .14). Table B-18. Second Grade: Perceived Versus Observed Length of Time for the Lesson Minutes Perceived (n = 29) Observed (n = 30) 0–4 — — 5–9 — — 10–14 — — 15–19 — 3% 20–24 7% 7% 25–29 3% 13% 30–34 — 10% 35–39 3% 7% 40–44 7% 20% 45–49 38% 7% 50–54 10% 13% 55–59 3% 7% 60–64 21% 3% 65–69 — — 70–74 3% 10% 75 or more 3% — Note. Due to rounding, percentages may not equal 100%.

As shown in Table B-19, the analysis of videotapes of second-grade Saxon Math Program instruction showed that students were seated on the classroom floor in a semicircle or circle in front of the chalkboard for The Meeting in 60% of cases. Slightly more than half (53%) of the videotaped second-grade Lessons were conducted with students seated at desks in clusters or arrangements other than rows. Table B-19. The Meeting and Lesson Classroom Arrangement Classroom Arrangement

Students seated on floor in semicircle or circle in front of the board Students seated in chairs in semicircle or circle in front of the board Students seated in desks in rows Students seated in desks in clusters or other arrangement other than rows



Lesson (n = 30)

—

3%

13% 17%

13% 53%

3%

16

Appendix B

Classroom Arrangement

Other:


Lesson (n = 30)

7%

7%

20%

The Meeting: • students seated around a table Lesson: • Students seated at long tables in a semicircle • Seated in chairs around a round table facing the board • Seated on floor for half of lesson and then seated in rows • At tables in semicircle around teachers • Students seated in chairs around table • Both seated at desks for latter half of lesson and seated on floor around teacher’s rocking chair for first half Did not observe

The various Saxon Math program resources and materials employed in videotaped second-grade lessons are presented in Table B-20. Because some resources are associated only with particular activities, it is not expected that all be present in every lesson. In at least two thirds of the videotaped lessons, teachers used The Meeting Board (87%), Guided Practice sheets (80%), and Learning Wrap-Ups (66%). Table B-20. Use of Resources and Materials Resources & Materials

The Meeting Board Student Guided Practice sheet Learning Wrap-UpsTM Student Class Fact Practice sheet Student Math Folders Poster-Size Lesson Charts Manipulatives (The Meeting) Manipulatives (Lesson) Meeting Masters (for students) Lesson Masters (for students) Teacher Fact Cards Student Written Assessments Student Fact Cards

Percentage Used in Lesson (n = 30) 87% 80% 66% 63% 63% 60% 47% 40% 33% 23% 17% 3% —


17

Appendix B

Third-Grade ICM Results Summary As with other grade levels, third-grade videotaped lessons were analyzed in terms of the extent to which they adhered to the Saxon Math program curriculum and structure. The results appear in Table B-21. Researchers rated 20 components on four implementation variations. Approximately half of the videotaped third-grade lessons were given a rating of “4” on 16 of the 20 components. A rating of “4” indicates that the Saxon Math program curriculum was observed to be implemented as designed. Four exceptions to this pattern included use of manipulatives (component 8), student sharing of learnings (component 12), frequency of questioning (component 17), and use of manipulatives (component 18). Approximately 40% of observed third-grade lessons (39%) did not include use of manipulatives (component 8, variation 1). In more than three fourths of observed lessons (79%), teachers did not ask students to articulate their learnings (component 12, variation 1). Little questioning occurred during Fact Practice (component 17, variation 1) in nearly two thirds of lessons (64%). In nearly half (46%) of observations of Number Fact Practice, manipulatives such as Fact Cards and Wrap-Ups were not used (component 18, variation 1). Table B-21. ICM Ratings for Third-Grade Saxon Math Lessons Component 4 3 The Meeting 1. Practice of skills (n = 29) 2. Use of manipulatives (n = 29) 3. Frequency of questioning (n = 30) 4. Use of wait time (n = 30) 5. Classroom management (n = 30) Lesson 6. Stating objective (n = 31) 7. Modeling concepts (n = 30) 8. Use of manipulatives (n = 31) 9. Modeling use of manipulatives (n = 17) 10. Frequency of questioning (n = 31) 11. Use of wait time (n = 31) 12. Students sharing learnings (n = 29) 13. Classroom management (n = 31) Guided Practice 14. Completing Guided Practice sheet (n = 20) 15. Time allowed for completing Guided Practice sheet (n = 20) 16. Teacher interaction during Guided Practice sheet completion (n = 17) Number Fact Practice 17. Frequency of questioning (n = 25) 18. Use of manipulatives (n = 26)

2

1

79% 93% 77% 90% 80%

14% 3% 23% 10% 20%

7% — — — —

— 3% — — —

81% 63% 48% 35%

10% 27% 13% 41%

10% 7% — 18%

— 3% 39% 6%

68% 87% 17% 71%

32% 13% — 26%

— — 3% 3%

— — 79% —

35%

40%

25%

—

65%

10%

25%

—

41%

41%

18%

—

12% 50%

8% 4%

16% —

64% 46%


18

Appendix B

19. Completing class Fact Practice sheet (n = 27) 20. Teacher interaction during class Fact Practice sheet (n = 21)

96%

—

—

4%

76%

19%

—

5%

Note. Due to rounding, percentages may not total to 100%. Not all components were observed in each lesson; consequently, n values are less than 32. Two (2) of the videotapes submitted did not include a taping of The Meeting. One (1) of the videotapes submitted did not include a taping of the Lesson. Five (5) videotapes did not contain Number Fact Practice. However, three of the five lessons should have included Number Fact Practice. Ten (10) videotapes did not contain Guided Practice. It is not known whether Fact Practice and Guided Practice were omitted from these lessons or the teacher elected not to have that part of the lesson videotaped. One video was of a second-grade teacher. However, the school at which this teacher is employed uses the next grade-level materials. The third-grade ICM was therefore used to rate the videotaped lesson. Likewise the third-grade teacher from this same school uses fourth-grade materials. That videotape was analyzed using the third-grade ICM, because components of the Saxon Math program are similar to those at the third grade.

Third-grade teachers were asked to estimate the length of time necessary to conduct The Meeting and the Lesson. Observers then recorded the time teachers spent conducting The Meeting and the Lesson on each videotape. The results are displayed in Tables B-22 and B-23. Analysis revealed that the majority of teachers (43%) perceived that The Meeting required 15 to 19 minutes. Observations indicated that the majority of teachers (28%) spent 10 to14 minutes conducting The Meeting. However, nearly a quarter (24%) of the third-grade teachers spent almost twice that much—20 to 24 minutes. The average observed time teachers devoted to The Meeting was 17.7 minutes, with a standard deviation of 6.1 minutes. The average amount of time teachers perceived to be necessary to conduct The Meeting was 18.6 minutes, with a standard deviation of 9.1. Differences between mean perceived and observed time for The Meeting were not statistically significant at the .01 level (p = .574). Table B-22. Third Grade: Perceived Versus Observed Length of Time for the Meeting Minutes Perceived (n = 30) Observed (n = 29) 0–4 — — 5–9 — 7% 10–14 10% 28% 15–19 43% 21% 20–24 37% 24% 25–29 3% 21% 30–34 3% — 35–39 — — 40–44 — — 45–49 — — 50–54 — — Minutes Perceived (n = 30) Observed (n = 29) 55-59 — — 60-64 3% — Note. Due to rounding, percentages may not total to 100%.


19

Appendix B

The majority of participating third-grade teachers (43%) thought the Lesson required 60 to 64 minutes to complete. These results are displayed in Table B-23. Observations, however, revealed that the majority (23%) conducted the Lesson in 40 to 44. The average observed time for conducting the Lesson was 39.8 minutes, with a standard deviation of 11.1. The average amount of time teachers estimated the Lesson to require was 48.2 minutes, with a standard deviation of 15.7 minutes. Differences between mean perceived and observed times for the Lesson were not statistically significant at the .01 level (p = .02). Table B-23. Third Grade: Perceived Versus Observed Length of Time for the Lesson Minutes Perceived (n = 30) Observed (n = 31) 0–4 — — 5–9 — — 10–14 — — 15–19 3% 6% 20–24 3% — 25–29 7% 6% 30–34 10% 19% 35–39 3% 13% 40–44 7% 23% 45–49 13% 19% 50–54 — 3% 55–59 3% 6% 60–64 43% — 65–69 3% 3% 70–74 — — 75 or more 3% — Note: Due to rounding, percentages may not equal 100%.

As shown in Table B-24, the analysis of classroom arrangements in videotaped thirdgrade classrooms showed students were seated at desks in clusters or arrangements other than rows for 38% of The Meetings observed. Students were seated on the classroom floor in a semicircle or circle in front of the chalk board in approximately one third of observations (34%). More than half of observed third-grade lessons (56%) were conducted with the students seated at desks in clusters or arrangements other than rows.


20

Appendix B

Table B-24. The Meeting and Lesson Classroom Arrangement Classroom Arrangement

Students seated on floor in semicircle or circle in front of the board Students seated in chairs in semicircle or circle in front of the board Students seated in desks in rows Students seated in desks in clusters or arrangement other than rows Other: The Meeting: • Students in walled cubicle desks in clusters • Students seated around a table Lesson: • They were in desks but couldn’t see from angle how they were arranged • Students in walled cubicle desks in clusters • Seated around a table • Students seated around a table Did not observe


Lesson (n = 32)

—

—

16% 38%

22% 56%

6%

13%

6%

9%

—

The various Saxon Math program resources and materials employed in videotaped second-grade lessons are presented in Table B-25. The Meeting Board (84%), student math folders (75%), and class Fact Practice sheets (72%) were observed in use most often. Table B-25. Resources and Materials Resources & Materials

The Meeting Board Student Math Folders Student Class Fact Practice sheet Meeting Masters (for students) Student Guided Practice sheet Manipulatives (the Lesson) Manipulatives (The Meeting) Lesson Masters (for students) Learning Wrap-Ups Poster-Size Lesson Charts Teacher Fact Cards Student Written Assessments Student Fact Cards

Percentage Used in Lesson (n = 32) 84% 75% 72% 63% 63% 53% 34% 34% 25% 22% 22% 13% —


21

Appendix B

LoU Telephone Interviews LoU interviews were conducted with participating teachers implementing the Saxon Math program. Data from such interviews provided an additional measure of the fidelity with which teachers implemented the program. As shown in Table B-26, the majority of teachers (70%) were classified as Routine users. Teachers at this level implement the Saxon Math program in a stable and routine manner. Such teachers indicated that they encountered no problems associated with the program’s use and felt knowledgeable enough about the program to teach it to their students. Teachers rated at Level IVA, Routine, reported making no changes in their pattern of use of the Saxon Math program components and intended to make few or no changes in their future use of the program. As one teacher put it, “I feel I’m getting the most out of the program by following it religiously.” Similarly, another teacher reported, “Saxon is wonderful. They are even working closely with us to modify their program to help us meet new [state] math standards. It doesn’t get much better than that.” Routine users of the Saxon Math program reported that they were very satisfied with the program, felt it was meeting the needs of their students, and did not see any reason to make changes to the program structure. Specifically, many teachers spoke positively about the spiraling nature of the curriculum and the repeated review of concepts. For example, one teacher said, “I think our children know their math facts better than they used to [because of the continual review].” Table B-26. Overall Levels of Use Ratings Level Number Name of Level 0 Nonuse I Orientation II Preparation III Mechanical IVA Routine IVB Refinement V Integration VI Renewal Totals

Number at Level 0 0 0 3 90 29 5 1 128

Percentage — — — 2% 70% 23% 4% 1% 100%

Note. N = 128 (30 kindergarten, 34 first-grade, 33 second-grade, and 31 third-grade teachers). The first three levels (0, I, and II) indicate nonuse of the innovation under analysis. In this study, the innovation is the Saxon Math program. The five upper levels (III, IVA, IVB, V, and VI) include only users of the innovation. Statistically significant differences in level of use were not found among grade levels.

Nearly one fourth of the teachers interviewed (23%) were rated Level IVB, Refinement. These teachers reported making purposeful changes in their use of Saxon Math program components in order to improve learning outcomes for their students. Several teachers remarked that changes they made in their use of the Saxon Math program were to address state standards not covered by Saxon. As one such teacher said, “The biggest issue is the alignment of the Saxon materials to the state standards and having to provide supplemental activities so students are prepared for the end-of-year state summative assessment.” Another teacher described her use of supplemental materials in the following way: “I don’t follow it [Saxon Math program] verbatim. I’ve taught for 20 years and can pull in other things that I know are good if the kids need it. The


22

Appendix B

kids often need more practice and activities in things like money and clocks.” Some teachers commented that the program tended to focus on the “average” child’s learning; such interviewees reported that they therefore sought alternative materials to accommodate both struggling and excelling students. Three teachers (2%) were classified as Level III, Mechanical use. Teachers at this level reported making changes in their use of the innovation to organize themselves better. These teachers indicated that they focused their efforts on time management, logistics, and management of materials. Such teachers devote most of their energy to gathering and making materials related to the Lesson and focus on the short-term, day-to-day use of the Saxon Math program with little time for reflection. Changes in use are more often made to meet teacher needs than student needs. Several teachers were rated Level V, Integration (4%). Teachers at this level are making deliberate efforts to coordinate the use of the Saxon Math program with colleagues. Teachers reported collaborating with other teachers to improve the use of the program in the classroom. For instance, one such teacher described her plans to work with other teachers to ensure that students meet the instructional goals and objectives. As she explained, we want to check how the kids are in meeting the goals and objectives, basically the California standards that we have to meet also. We have to take a look at what standards the kids are needing more help in; sometimes there are a few standards that aren’t as strongly covered in Saxon that we need to cover before April. The program just doesn’t stop in April after we test, so there are some things we have to push up and make sure get introduced so we plan that way. So we’ll say where are we at and what do we need to bring in. We do that in our collaboration with teachers. Another teacher described her collaboration in the following way: “It [the collaboration] really gives you a sounding board, someone to bounce your ideas off. It helps you with planning. We keep our lessons together. We’re teaching the same thing each day . . . so that has really helped.” One of the 128 teachers interviewed was rated Level VI, Renewal. This teacher reported reevaluating the quality of use of the Saxon Math program and seeking alternatives to it. According to the teacher, school staff recently voted to implement another math program because of the need for material for more advanced students. One of the teachers interviewed indicated that she no longer used the Saxon Math program. As a result, her interview is not included in this summary. A modified version of the interview revealed that this teacher had been a Routine user of the Saxon Math program when she was using the program. In addition to the overall LoU ratings, ratings were also given for each of the seven categories for which levels of use information was gathered in the interview. Table B-27 displays these results. As with the overall ratings, the majority of teachers were found to be at Level IVA, Routine use. For instance, 86% of teachers were at the Routine level in terms of Acquiring Information. In other words, the majority of teachers reported satisfaction with the amount of information they have about the Saxon Math program and were not deliberately seeking additional information about the program. S a x o n E l e m e n t a r y M a t h P r o gr a m E f f e c t i v e n e s s S t u d y

23

Appendix B

Table B-27. Categorical Levels of Use Ratings (n = 128) Level Category Acquiring Status Knowledge Information Sharing Assessing Planning Reporting Performing 0 — — — — — — — I — — — — — — — II — — — — — — — III 6% 3% 6% 4% 4% 2% 2% IVA 65% 86% 66% 56% 69% 59% 56% IVB 23% 9% 20% 35% 19% 34% 36% V 6% 2% 5% 2% 7% 4% 7% VI — — 1% 1% 1% 1% — Not 3% 2% 1% — — Doing Note. Percentages may not total to 100%, due to rounding. Statistically significant differences were not found among grade levels.

Approximately two thirds of participating Saxon teachers were classified at the Routine Level (IVA) for Planning (69%), Sharing (66%), and Knowledge (65%). In terms of planning, Routine users reported planning intermediate and long-range actions with little projected variation on how the Saxon Math Program would be used. Such teachers reported few or no plans to modify their use of Saxon. Routine users in the Sharing category described their use of the Saxon Math program with little or no reference to changing their use. For the category of Knowledge, Routine users reported knowing both the short- and long-term requirements for use and how to use the Saxon Math program with minimum effort or stress. More than half of the teachers were rated at the Routine level (IVA) for Status Reporting (59%), Assessing (56%), and Performing (56%). Approximately one third of participating Saxon teachers were rated at Level IVB, Refinement, for Status Reporting, Assessing, and Performing. Teachers at the Routine Level have established a routine level of use; small changes may be made routinely, but there are no recent or dramatic changes in use. However, at the Refinement Level, changes in the use of the innovation are made based on formal or informal evaluation. The changes are recent (within the past 3 months), and changes are made in order to improve student outcomes. For example, teachers at the Refinement level in the Performing category reported exploring and experimenting with alternative combinations of the Saxon Math program and their existing practices to optimize student performance. One teacher said she refined the use of the program by allowing students additional time to use manipulatives, both during the course of the lesson and during free time, so that they have more time to grasp the concept through hands-on methods. A third-grade teacher who teaches all sections of third-grade math reported discussions with the second-grade teacher at her school about next year’s students. Such conversations led to the discovery that the next year’s students had progressed further than expected. As a result, this third-grade teacher plans to introduce multiplication facts at the beginning of the new school year rather than later.


24

Appendix B

Experimental Teacher Survey Implementation of Saxon Math Program Components The Saxon Math program consists of several key components: The Meeting, Lesson, Guided Practice (Grades 1–3), Number Fact Practice (Grades 1–3), Lesson Practice (kindergarten), Counting Practice (kindergarten), and Handwriting Practice (kindergarten). Each component of the program is intended to be implemented in a particular manner. Teachers were asked to rate the frequency with which they implemented each element associated with each program component. A 5-point frequency scale was used where 1 = never, 2 = rarely, 3 = sometimes, 4 = often, and 5 = always. The Meeting. In general, teachers reported that they implement each element of The Meeting often. As shown in Table B-28, mean ratings ranged from a high of 4.9 (asking questions of students) to a low of 4.1 (focus of The Meeting on the practice of all skills intended to be taught in The Meeting). A one-way ANOVA revealed no statistically significant gradelevel differences. Table B-28. Elements of The Meeting Instructional Element

Mean

SD

N

The Meeting focus is entirely on the practice of skills intended to be taught in The Meeting.

4.1

.8

123

Saxon Math manipulatives and resources are used as a part of The Meeting (e.g., Meeting Board materials).

4.2

1.0

125

I ask questions of students.

4.9

.5

124

I use the appropriate amount of wait time for students to respond to questions.

4.5

.6

125

The Meeting is conducted in an orderly manner. (Only one student speaks at a time unless I ask for a choral response. All students are focused and on task.)

4.7

.7

124


25

Appendix B

Lesson. With one exception, all elements of the Lesson were rated by teachers as occurring often or always. Table B-29 shows mean ratings for eight of the nine elements, which ranged from 4.9 to 4.3. In general, teachers tended to report that they nearly always modeled the concepts taught in the Lesson (mean = 4.9). Teachers also indicated that they nearly always asked questions of students (mean = 4.9). Teachers were not as likely to allot time at the end of the Lesson for students to share their observations about what they learned during the Lesson, with a mean rating of 3.2 or sometimes on the 5-point scale. A one-way ANOVA detected no statistically significant differences among grade levels on each of the implementation elements associated with the Lesson. Table B-29. Elements of the Lesson Instructional Element

Mean

SD

N

At the beginning of the Lesson, I clearly state the objective of the Lesson to the students (e.g., what students will be learning in the Lesson).

4.5

.8

125

I model the concepts being taught in the Lesson (e.g., by writing numbers on the board or showing students how to do something).

4.9

.3

125

I use Saxon Math manipulatives as a part of the Lesson.

4.4

.7

124

I model the use of manipulatives prior to having students use the manipulatives.

4.7

.6

125

I clearly explain to students how they are to perform the task with the manipulatives.

4.8

.4

125


4.9

.4

124


4.6

.5

123

I allot 2 to 3 minutes at the end of the Lesson for students to share their observations about what they learned in the Lesson.

3.2

.9

124

The Lesson is conducted in an orderly manner. (Only one student speaks at a time unless I ask for a choral response. All students are focused and on task.)

4.3

.7

125

Guided Practice. Guided Practice is conducted in Grades 1 and 3 as part of the Saxon Math program. As Table B-30 shows, three of the six instructional elements occurring during Guided Practice were rated on average as occurring often or always, with means ranging from 4.2 to 4.5 on the 5-point scale. Mean ratings for the other three elements associated with Guided Practice suggest that teachers implement them often (3.8 to 3.9). Teachers reported that they read the directions separately for completion of each problem on Side A of the Guided Class Practice sheet most often (mean = 4.5). A one-way ANOVA found that first-grade teachers were statistically significantly more likely to use this practice (mean = 4.9) than third-grade teachers (mean = 4.2). S a x o n E l e m e n t a r y M a t h P r o gr a m E f f e c t i v e n e s s S t u d y

26

Appendix B

Teachers reported requesting that students explain how to find the answer to each problem on Side A of the Guided Class Practice sheet, one problem at a time, least often. The mean rating of the frequency with which teachers implemented this instructional element aspect was 3.8. It should be noted that this instructional practice is only used in second- and third-grade classrooms. Table B-30. Elements of Guided Practice Instructional Element

Mean

SD

N

I read the directions separately for completion of each problem on the Guided Class Practice sheet (Side A).

4.5

.8

95

I allow time for students to complete each problem on the Guided Class Practice sheet (Side A) and then write the answer to each problem on a chart. (Grade 1)

4.2

1.1

33

I circulate and checks students’ papers the entire time as they work on each problem of the Guided Class Practice sheet (Side A). (Grade 1)

4.2

.8

33

I do each problem on the Guided Class Practice sheet (Side A) with the students one problem at a time, walking the students through each problem. Students then do each problem on their own, one at a time. (Grades 2 and 3)

3.9

1.2

62

As a class, after the students have completed a problem, the students and I look at each problem one at a time to verify correctness. (Grades 2 and 3)

3.9

1.1

63

I have students explain how to find the answer to each problem on the Guided Class Practice sheet (Side A), one problem at a time. (Grades 2 and 3)

3.8

1.0

63

Lesson Practice. Lesson Practice is conducted at the kindergarten level, and Table B-31 displays findings related to this practice. Two of the three instructional elements associated with Lesson Practice were reported by the teachers as occurring between often and always, with means of 4.5 and 4.4. Teachers indicated that they almost always read the directions of the Lesson Practice sheet to their students and circulated and checked students’ papers continuously as they worked on the Lesson Practice sheet. Teachers were somewhat less likely on average to ask students to complete the Lesson Practice sheet independently (mean = 3.6). No statistically significant grade-level differences were located, according to a one-way ANOVA.


27

Appendix B

Table B-31. Elements of Lesson Practice Instructional Element

Mean

SD

N

I read the directions of the Lesson Practice sheet to the students.

4.5

.7

28

Students complete the Lesson Practice sheet independently.

3.6

.7

28

I circulate and check students’ papers continuously as they work on the Lesson Practice sheet.

4.4

1.0

28

Number Fact Practice. Number Fact Practice is conducted in first through third grades implementing the Saxon Math program. As Table B-32 shows, teachers indicated that they implemented the program elements associated with Number Fact Practice often. With a mean rating of 4.9, first-grade teachers reported that they almost always allowed time for students to complete Side A of the Class Fact Practice sheet independently. First-grade teachers read all the problems and answers for the Class Fact Practice sheet on Side A after the students completed the sheet often (mean = 3.9). A one-way ANOVA found no statistically significant differences in grade-level responses. Table B-32. Elements of Number Fact Practice Instructional Element

Mean

SD

N


4.5

.8

95

Saxon Math manipulatives are used as a part of Number Fact Practice (e.g., Fact Cards, Wrap-Ups).

4.0

.9

95

I allow time for students to independently complete the Class Fact Practice sheet (Side A). (Grade 1 only)

4.9

.4

33

I read all the problems and answers for the Class Fact Practice sheet (Side A) after students complete the sheet. (Grade 1 only)

3.9

1.3

33

I have students complete Class Fact Practice sheet (Side A) while timing them for a specified period of time. (Grades 2 and 3 only)

4.7

.7

63

I have the students read the answers to the Class Fact Practice sheet (Side A). (Grades 2 and 3 only)

4.1

1.1

62


28

Appendix B

Handwriting Practice. Handwriting Practice is conducted only at the kindergarten level. The three instructional aspects pertaining to Handwriting Practice were rated as occurring often, with mean frequency ratings ranging from 4.1 to 4.3. See Table B-33 for results. Table B-33. Elements of Handwriting Practice Instructional Element

Mean

SD

N

I read the directions for completion of the Handwriting Practice sheet to the students.

4.1

1.3

28

I give the class guidance on initial completion of the Handwriting Practice sheet.

4.2

1.3

28

I circulate and check students’ papers continuously as they work on the Handwriting Practice sheet.

4.3

1.3

28

Counting Practice. Counting Practice is also specific to kindergarten. Like Handwriting Practice, Counting Practice instructional elements were reported to be implemented often, with means ranging from 4.2 to 4.3 on the 5-point scale. See Table B-34 for complete information. Table B-34. Elements of Counting Practice Instructional Element

Mean

SD

N

I read the directions for completion of the Counting Practice sheet to the students.

4.3

1.1

26

I give the class guidance on initial completion of the Counting Practice sheet.

4.2

1.1

26

I circulate and check students’ papers continuously as they work on the Counting Practice sheet.

4.3

1.0

26

Implementation Logistics Teachers were asked to indicate how much time was spent on each of the different elements when taught as a part of the Saxon Math program, as Table B-35 shows. The majority of teachers (36%) indicated they devoted 11 to 15 minutes to The Meeting. More than one fourth (27.2%) reported conducting The Meeting within 16 to 20 minutes. One fourth of participating Saxon teachers (24.8%) indicated that they spent 16 to 20 minutes conducting the Lesson. An additional one fifth (20.8%) reported that 21 to 25 minutes were used for teaching the Lesson. A one-way ANOVA revealed statistically significant differences at second grade for the amount of time teaching a lesson: Second-grade teachers were more likely to spend 21 to 25 minutes on a lesson (31.3%) as compared to the other grade-level teachers.


29

Appendix B

More than a third each of kindergarten (34.5%) and third-grade teachers (38.7%) reported spending 16 to 20 minutes teaching the Lesson. Approximately one fourth of participating second-grade teachers (24.2%) indicated that the Lesson required 11 to 15 minutes; one fifth (21.2%) selected 16 to 20 minutes; another one fourth (24.2%) chose 21 to 25 minutes. Approximately one third of first-, second-, and third-grade teachers (32.3%) reported they devoted 16 to 20 minutes to Guided Practice. The component comparable to Guided Practice at the kindergarten level is Lesson Practice. Half of participating kindergarten teachers (50%) reported spending 11 to 15 minutes on Lesson Practice. Nearly half of firstthrough third-grade teachers (47.4%) indicated that they devote 10 minutes or less to Number Fact Practice. Approximately 70% of kindergarten teachers spend 10 minutes or fewer on Handwriting Practice (67.9%) and Counting Practice (70.4%). Table B-35. Time Spent Teaching Saxon Math Program Components None

10 min. or less

11–15 min.

16–20 min.

21–25 min.

26–30 min.

31–35 min.

36 min. or more

2.4%

17.6%

36.0%

27.2%

9.6%

4.8%

2.4%

—

Lesson (K–3) (n = 125)

—

2.4%

16.8%

24.8%

20.8%

17.6%

10.4%

7.2%

Guided Practice (1–3) (n = 96)

—

8.3%

29.2%

32.3%

19.8%

5.2%

4.2%

1.0%

Lesson Practice (K) (n = 28)

3.6%

28.6%

50%

7.1%

10.7%

—

—

—

Number Fact Practice (1–3) (n = 95)

1.1%

47.4%

37.9%

11.6%

2.1%

—

—

—

Handwriting Practice (K) (n = 28)

7.1%

67.9%

17.9%

3.6%

3.6%

—

—

—

Counting Practice (K) (n = 27)

11.1%

70.4%

11.1%

3.7%

—

—

3.7%

—

Saxon Math Program Components The Meeting (K–3) (n = 125)

Table B-36 displays information about time of day when teachers conduct Saxon Math components. Approximately two thirds of participating teachers (65%) said they convene The Meeting at the beginning of the school day. More than a third (35.8%) reported that they conduct the Lesson at the beginning of the afternoon. Approximately another third (35.8%) indicated that they conduct the Lesson at either the middle or end of the morning. Nearly 40% of firstthrough third-grade teachers (38.9%) lead Guided Practice at the beginning of the afternoon. Likewise, approximately a third of kindergarten teachers conduct Lesson Practice at the beginning of the afternoon. S a x o n E l e m e n t a r y M a t h P r o gr a m E f f e c t i v e n e s s S t u d y

30

Appendix B

Table B-36. Part of Day Saxon Math Components Conducted

Saxon Math Program Components

Don’t use

Beginning of the day

End of the Middle of morning (before the lunch) morning

Beginning of the afternoon (following lunch)

End of the day

Middle of the afternoon

The Meeting (K–3) (n = 123)

2.4%

65.0%

16.3%

3.3%

12.2%

0.8%

—

The Lesson (K– 3) (n = 123)

—

13.0%

18.7%

17.1%

35.8%

13.8%

1.6%

Guided Practice (1–3) (n = 95)

—

14.7%

13.7%

14.7%

38.9%

14.7%

3.2%

Lesson Practice (K) (n = 28)

3.6%

7.1%

14.3%

17.9%

32.1%

17.9%

7.1%

Number Fact Practice (1–3) (n = 96)

—

20.8%

9.4%

36.5%

11.5%

1.0%

—

Handwriting Practice (K) (n = 27)

11.1%

7.4%

11.1%

14.8%

29.6%

22.2%

3.7%

Counting Practice (K) (n = 28)

14.3%

46.4%

10.7%

14.3%

7.1%

7.1%

—

As shown in Table B-37, for the most part, participating teachers anticipated completing nearly all of the 135 Saxon Math lessons by the end of the school year. Kindergarten teachers expected to complete a mean of 121.4 lessons, somewhat fewer than the numbers teachers at the other three grade levels expected to conduct. Fewer than half (45.5%) of the kindergarten teachers anticipated completing lesson 135. The average number of lessons first-, second-, and third-grade teachers expected to complete ranged from 130.5 to 131.1. More than half of first(52%) and second-grade (57.7%) teachers anticipated completing the program by the end of the school year. Slightly fewer than half of third-grade teachers (48.2%) expected to finish at least 134 of the 135 lessons by the end of the year. Table B-37. Lesson Number Anticipated Completing Grade Level Mean SD N Range K 121.4 17.8 22 75–135 1 130.5 7.2 25 110–135 2 131.1 5.5 26 120–135 3 130.7 5.1 29 118–135


31

Appendix B

Use of Saxon Math Assessments and Resources As is shown in Table B-38, more than 90% of participating teachers reported using the Saxon Math program assessments. Kindergarten teachers were less likely to use the assessments (79%) than first- (93%), second- (97%) and third-grade (97%) teachers. Kindergarten teachers not using the Saxon Math program assessments cited reasons such as use of their own assessments correlated with the school’s report cards and requirements to use state-developed assessments to report student performance. Approximately half of teachers (53.3%) indicated that they used the assessment reporting forms. Teachers who did not use the Saxon assessment report forms reported using their own format instead. Three fourths of teachers (74.8%) indicated using the assessment scoring guides. Third-grade teachers were more likely to use the scoring guides (97%) than kindergarten (62%), first- (68%) and second-grade (78%) teachers. Teachers who elected not to use the Saxon assessment scoring guide used their own scoring methods. Table B-38. Using of Saxon Math Assessments Assessments

Yes

No

Saxon Math assessments used (n = 124)

92.0%

8.0%

Assessment reporting forms used (n = 121)

53.3%

46.7%

Assessment scoring guides used (n = 121)

74.8%

25.2%

As is shown in Table B-39, the majority of teachers did not use the Saxon Math Teacher’s Resource CD (81.5%) or the Lesson Planner CD (90.3%).. Table B-39. Use of Saxon Resources Resources Did you use the Teacher’s Resource CD? (n = 124) Did you use the Lesson Planner CD? (n = 124)

Yes

No

18.5%

81.5%

9.7%

90.3%

Additional Information Teachers were asked to respond to four open-ended questions on the teacher survey. Following is a summary of responses to each question categorized by grade level. A verbatim listing of all teachers’ responses to each question is shown in Appendix C. 1. Did you use other materials to supplement your mathematics curriculum? If yes, what materials did you use and for what purposes (e.g., what curriculum areas did the materials address)? If not, why not?


32

Appendix B

Kindergarten. Several kindergarten teachers indicated they use practice worksheets from other math curricula or develop their own to reinforce or extend skills. In some cases, teachers reported supplementing the Saxon curriculum to meet state requirements. For example, one kindergarten teacher indicated that her state requires that students learn to write numbers to 30 earlier in the year than did the Saxon curriculum; as a result, she developed her own worksheets. Another teacher said she used supplemental math worksheets for more focus on time, money, addition, and subtraction. Kindergarten teachers reported using parts of other math curricula in math instruction as well. These included Math Their Way (Addison Wesley) for computation skills, time, money, and geometric shapes; Scott Foresman for numeral writing; and Drop in the Bucket for sequencing and correspondence. Teachers have also used their own manipulatives to teach concepts. A kindergarten teacher reported using a globe, a Coke can, and an ice-cream cone to teach about spheres, cones, and cylinders. Teachers also indicated that they integrate literature when appropriate into math lessons. For instance, one teacher uses the book The Grouchy Ladybug (by Eric Carle) to teach telling time to the hour. First grade. Several first-grade teachers stated they use worksheets from other sources to supplement the Saxon Math program. Teachers also reported use of alternate manipulatives, such as coins when working with money and clocks when teaching time. One teacher said that Saxon manipulatives were not purchased by the school; instead, teachers ordered the manipulatives they needed from various education catalogs because they found this to be less expensive. In some situations first-grade teachers have used other math curricula to supplement and extend the Saxon Math program. Teachers mentioned the following resources: Heath Mathematics, Harcourt Math Advantage, Math Their Way, and Math Steps to teach fact families; Excel to challenge students; and Minute Math for math facts. Second grade. Second-grade teachers reported using worksheets they either developed or borrowed from other sources to supplement the Saxon Math program. One teacher explained, “I used supplemental worksheets and activities to reinforce the major concepts covered by Saxon.” Another teacher developed her own worksheets to include the types of items students would encounter on the state test. Other teachers used games (e.g., bingo, chalkboard challenges) for additional practice of concepts. Teachers cited specific math programs they used to supplemental the Saxon Math program. Those included Marcy Cook Math, Math Warm-Ups: Developing Fluency in Math (by Sheri Disbrow), Math Facts in a Flash software program, Accelerated Math, and Math Advantage.


33

Appendix B

Third grade. Third-grade teachers reported using worksheets from other math programs or ones they developed to reinforce or supplement math concepts. Teachers used other worksheets to address multiplication, division, fractions, geometry, and money. Teachers have also elected to use games for learning reinforcement (e.g., playing cards, multiplication games, clock games). Some teachers have pulled in materials from other math curricula including Math Facts in a Flash, Star Math, Accelerated Math, Measuring Up, Silver Burdett Ginn, Read It! Draw It! Solve it! (by Elizabeth Miller, for problem-solving activities), and MacMillan/McGraw-Hill. 2. What aspects of the Saxon Math program made you or your school decide to adopt the program? Kindergarten. Kindergarten teachers cited four principal reasons their schools adopted the program. Each reason was mentioned with approximately the same frequency. First, teachers liked that the program was scripted. As one teacher explained, “We enjoyed the Saxon Phonics Program so much that we were curious to see how our students would do with the Saxon Math Program. The scripted program (direct instruction) works really well with our student population of over 70% free or reduced-price lunch.” Second, and complementary to the scripted nature of the lessons, teachers appreciated the way the lessons were presented and were teacher-ready. In one teacher’s words, “Teacherfriendly lessons are laid out well.” Third, teachers indicated that the repetitive and spiraling nature of the program led their school to adopt it. Explained one teacher, “The repetition of the program really helps students grasp the skills.” Finally, teachers found the use of manipulatives to be a strength of the program. First grade. First-grade teachers indicated three main reasons their schools adopted the Saxon Math program. Listed most frequently was the spiraling approach and repetitive nature of the program. Said one teacher, “The structure and repetition (increasing in difficulty) is unique and very well designed.” Second, teachers appreciated the structure of the program and the scripted lessons. Finally, a few teachers noted that the Saxon Math program aligned well with their state standards. Second grade. Two of the major reasons their schools adopted the program, according to second-grade teachers were the spiraling approach and repetitive nature of the program. One teacher explained, “It spirals and gives lots of practice of many skills all year. Once you add you don’t leave that skill and go on to the next.” Another teacher added, “The continual review of skills and program format would help all our students improve in math.” Second-grade teachers also commented that the organization of the lessons was a factor that led to the school’s decision to adopt Saxon Math. In several instances, teachers indicated that the Saxon Math program correlated well with their state curriculum and assessments.


34

Appendix B

Third grade. The repetitive format of the Saxon Math program was the major reason third-grade teachers said their schools selected this curriculum. One teacher explained that we wanted a series that allowed our students the opportunity to learn more skills. In traditional math series, you learn by units and may not see another practice on that particular skill or concept until you get to a chapter review. We were really sold on the amount of skills that were taught with the distributed practice built into the program. Other teachers said the curriculum was adopted because of the structure of the program and its scripted format. As one teacher shared, “The fact that the program is scripted [led us to adopt it]. We wanted to make sure we used the same terms across the board and to assure that all students are being taught the same lessons.” 3. What do you like best about the Saxon Math program? Kindergarten. Overall, kindergarten teachers reported that they like the design of the Saxon Math program. They indicated that lesson preparation was easy and that the program design provided substitute teachers an easy-to-use template. According to one teacher, “I like how organized the program is. I like having the worksheets already made and ready to use. I like the easy format to follow for most of the lessons with the materials already made.” Teachers specified that they liked The Meeting, manipulatives, and worksheet practice sheets. One teacher summed up her sentiments in this way: “The fun ways of teaching math using a variety of great manipulatives and the worksheet practice really reinforces the skills.” First grade. First-grade teachers are pleased with the repetition and review built into the Saxon Math Program, use of manipulatives, and The Meeting. Said one teacher, “I love the Math Meeting, which reinforces skills needed in daily life and taught in the lessons. I think the manipulatives are awesome. It is so nice they are included in the program. All skill levels find satisfaction and a sense of accomplishment in the daily lessons and homework!” Other teachers appreciated the design and layout of the lessons, reporting that they are easy to implement. Other teachers focused on student outcomes, writing that their students seemed to enjoy math and were successful with the Saxon Math program. Second grade. Nearly every second-grade teacher reported the best things about the Saxon Math program were the repetition of skills and spiraling review. In the words of one teacher, “My favorite part about the Saxon Math program is the way the math skills are introduced and then reviewed throughout the year. The review helps the student retain the knowledge they have learned.” Several teachers commented on their appreciation of the program’s organization. They liked the preplanned, scripted lessons. Teachers also noted that the program design allowed parents to get involved in their child’s math education through the practice sheets.


35

Appendix B

Third grade. Third-grade teachers agreed that the spiral review, repetition, and review of concepts are the components they like best about the Saxon Math program. One teacher explained, “It reviews material constantly. Students cannot forget what they have learned because it is constantly brought back into problems for review or further development.” Another teacher said, “The Guided Practice pages provide distributed practice on the skills that have been previously taught.” A number of teachers also like the overall organization of the program, reporting that lessons are easy to follow and work especially well for substitute teachers. Several teachers reported that students remarked that they like The Meeting. Finally, a few teachers mentioned that students are successful with Saxon Math and that they enjoy it. Said one teacher, “Children seem to really understand the concepts much quicker.” 4. What changes, if any, should be made to the Saxon Math program? Kindergarten. Teachers requested the inclusion of, or additional focus on, several concepts, including addition and subtraction problems, handwriting practice of individual numbers, graphing, patterning, and greater-than and less-than concepts, numbers, counting practice, sorting, time, and money. In some situations, teachers remarked that the Saxon Math program does not align well with the state standards they are required to teach. As one teacher explained, “This program doesn’t match the [name of state standards] we are required to teach. The program jumps from topic to topic. Kindergarten students need more repetition of skills in the lessons; focusing on a skill for several days would benefit students.” Other teachers indicated that they wished the kindergarten curriculum included higher level activities to challenge advanced students. First grade. First-grade teachers offered varied suggestions. Three teachers reported that the beginning lessons of first grade were too easy and could be condensed. Said one such teacher, “The program starts out way too simple for first graders. I would like to see it start with lesson 30 and introduce more challenging skills like regrouping for the rest of the year.” Several teachers indicated that the lessons were too lengthy. Specific concepts to which teachers would like more time devoted included estimation, probability, groups of 10, and counting two kinds of coins. Several teachers also recommended introducing math facts and subtraction sooner in the year. Second grade. A number of second-grade teachers suggested changes to the Saxon Math program in terms of problem solving. Such teachers recommended more variety in story problems to encourage students to use more problem-solving strategies. Teachers also suggested more integration of problem solving. Several teachers indicated that too many lessons at the beginning of the year focused on review and that gifted students were not adequately challenged by the curriculum. In a few situations, teachers reported that the Saxon Math program did not meet all of the state standards, or that the assessments and practice sheets were not in a format that prepared students for state assessments.


36

Appendix B

Third grade. Many of the changes suggested by third-grade teachers pertained to program pacing. Some teachers thought the program started out too slowly at the beginning of the year and then introduced too many new concepts (e.g., multiplication) by a time in the school year too late for standardized testing. Other teachers reported that the time required for lessons varied too much (e.g., 10 to 45 minutes). Respondents also indicated that it was difficult for teachers to conduct a math lesson within their allotted math period on days assessments were administered. Third-grade teachers also requested more lessons or practice in addition and subtraction regrouping, long division, fractions, measurement, and multistep word problems. A few teachers also recommended that the Saxon Math program curriculum incorporate a standardized test format to prepare students for state tests. Classroom and Teacher Information In order to understand the context in which participating Saxon teachers work, the survey asked several demographic questions. Nearly all teachers (98%) indicated that they teach in a self-contained classroom. A small percentage of teachers teach in a multiage classroom (7%). On average, there were approximately 20 students per classroom (mean = 19.5, standard deviation = 4.1). Nearly three fifths of teachers (58%) reported that their classroom populations were heterogeneous, with a mixture of two or more ability levels in mathematics. Another 29.4% of teachers said that their class was fairly homogeneous and average in mathematics ability. More than one tenth (11.8%) indicated that their class was fairly homogeneous and of high ability. The remaining 0.8% of teachers reported that their class was fairly homogeneous and low in mathematical ability. On average, the number of English language learner or limited English proficient students in the class was 3.0 (standard deviation = 4.8). The average number of special education students was 1.4, with a standard deviation of 1.4. The average number of gifted and talented students was 1.1, with a standard deviation of 1.7. Nearly all responding teachers (98%) were female. Almost 30% of teachers were either between 41 and 50 years old (29.2%) or between 51 and 60 years old (27.5%). Approximately one fourth of teachers (24.2%) were between 31 and 40 years old. Nearly one fifth of the teachers (18.3%) were 20 to 30 years old, and the remaining 0.8% of teachers were 61 years of age or older. Participating teachers had taught in a regular teaching position for a mean of 12.7 years (standard deviation = 9.5) and had taught in their current district for an average of 11.6 years (standard deviation = 9.3). The average number of years responding teachers had taught at their current school was 10 (standard deviation = 8.2). The average number of years teachers taught at their current grade level was 7.2 (standard deviation = 7.1). Finally, on average, teachers had used the Saxon Math program for 4.9 years (standard deviation = 2.8).


37

Appendix B

Comparison Teacher Survey The comparison teacher survey consisted of six sections: math concepts taught, instructional practices and instructional time, assessment, student engagement and classroom management, math resources, and classroom and teacher information. Math Concepts Teachers were asked to indicate the mathematical concepts taught in their classrooms. Results appear in Table B-40. These mathematical concepts correspond to those covered by the Saxon Math program. Percentages are bolded for those concepts reported to be taught in 75% or fewer grade-level classrooms. More than half of the 17 concepts taught in the Saxon Math program at the kindergarten level are taught by 75% or fewer of comparison group teachers. In only 29.2% and 33.3% of the kindergarten classrooms, respectively, are probability and data analysis and statistics taught. At the first-grade level, 5 of the 23 concepts are taught by 75% or fewer of the comparison group first-grade teachers. Data analysis and statistics is taught by 45.8% of firstgrade teachers, and 50% of these teachers focus on algebra and functions. Five of the 23 concepts in the second-grade Saxon Math program are taught by 75% or fewer of the comparison group second-grade teachers. The two areas taught least frequently are data analysis and statistics (54.2%) and algebra and functions (58.3%). Two of the 21 third-grade Saxon Math program concepts are taught by 75% or fewer of the third-grade comparison group teachers. Both algebra and functions and writing about math were taught in 75% of these classrooms. Table B-40. Percentages of Teachers Teaching Math Concepts Concept Kindergarten First (n = 24) (n = 24) Number sense and numeration 100% 100% Concepts of whole-number 87.5% 95.8% computation Concepts of whole-number operations n/a 100% Number recognition 100% n/a Counting 100% 100% Fractions n/a 58.3% Fractions and decimals n/a 79.2% Money 95.8% 95.8% Geometry and spatial relationships 91.7% 75.0% Time 91.7% 100% Calendar 100% 95.8% Temperature 79.2% 54.2% Linear measurement 79.2% 54.2%


Second (n = 24) 100% 100%

Third (n = 24) 100% 100%

95.8% n/a 100% n/a 91.7% 100% 87.5% 100% 100% 75.0% 87.5%

100% n/a 100% n/a 91.7% 100% 95.8% 100% 95.8% 100% 91.7%

38

Appendix B

Estimation Weight (mass) Capacity (volume) Data analysis and statistics Probability Patterns, algebra, and functions Algebra and functions Patterning Weather Graphing Mental computation Place value Tallying Problem solving Writing of numbers Fact families Writing about math

75.0% 62.5% 50.0% 33.3% 29.2% 79.2% n/a n/a n/a n/a n/a n/a n/a n/a n/a n/a n/a

n/a 83.3% 70.8% 45.8% 70.8% n/a 50.0% 100% 95.8% 95.8% 75.0% 95.8% 100% 95.8% n/a n/a n/a

n/a 79.2% 62.5% 54.2% 70.8% n/a 58.3% 95.8% n/a 95.8% 95.8% 100% n/a 100% 91.7% 100% n/a

n/a 83.3% 83.3% 87.5% 91.7% n/a 75.0% 100% n/a 95.8% 91.7% n/a n/a 100% n/a n/a 75.0%

Note. n/a = not part of Saxon Math program grade-level curriculum. Boldface indicates concepts taught by 75% or fewer of comparison group teachers.

Instructional Time and Practices As Table B-41 shows, more than three fourths of comparison teachers (76.8%) indicated spending more than 35 minutes on math instruction each day. Kindergarten devoted the least amount of time to math instruction. Approximately one fifth of these teachers (20.8%) reported they spend 21 to 25 minutes and another 20.8% devote 11 to 20 minutes to math instruction. Table B-41. Daily Time Devoted to Math Instruction (n = 95) Time Allotment Percentage 10 minutes or less — 11–15 minutes 1.1% 16–20 minutes 2.1% 21–25 minutes 4.2% 26–30 minutes 7.4% 31–35 minutes 8.4% 36 minutes or more 76.8% Table B-42 shows the time of day devoted to Saxon Math. The majority of teachers reported that math instruction takes place in the morning. Slightly more than one fourth (26.4%) conduct math class at the end of the morning before lunch, and one fourth convene math class at the beginning of the day (25.3%). Third-grade teachers were more likely to conduct math class at the beginning of the day (41.7%) than teachers at other grade levels.


39

Appendix B

Table B-42. Time of Day of Math Instruction (n = 91) Time of Day Percentage Beginning of the day 25.3% Middle of the morning 19.8% End of the morning (before lunch) 14.3% End of the morning (before lunch) 26.4% Beginning of the afternoon (following lunch) 12.1% Middle of the afternoon 2.2% End of the day — See Table B-43 for information on how teachers rated the frequency with which they used a variety of instructional practices in their math classes. A 5-point frequency scale was used, where 1 = never, 2 = rarely, 3 = sometimes, 4 = often, and 5 = always. Comparison teachers indicated that they used 11 of the 30 practices at least often, with mean ratings of 4.0 or greater. For example, kindergarten teachers at comparison sites reported that their students practice handwriting during math class almost always, with an average of 4.7 on the 5-point scale. Kindergarten through third-grade teachers reported that they communicate the importance of what is being learned to students and allow wait time for students to think after they ask a question, with means for both practices of 4.6. Statistically significant differences in the amount of time devoted to four practices were located between grade levels. Second- and third-grade teachers were statistically significantly less likely than kindergarten teachers to introduce new concepts with hands-on activities (p = .00). Kindergarten and first-grade teachers were significantly more likely to use math manipulatives than second- and third-grade teachers (p = .00). Kindergarten teachers were also significantly more likely to direct students to participate in math center activities than teachers at other grade levels (p = .00). Table B-43. Math Instructional Practices Instructional Practices

Mean

SD

N

Students practice counting in my math class. (K)

4.7

.5

24

I communicate the importance of what we are learning to my students (e.g., state the objective).

4.6

.7

95

I allow wait time for students to think after I ask a question.

4.6

.6

95

I assess students on correctness of answers.

4.3

.8

95

My lessons include a variety of teaching methods.

4.3

.6

96

During the course of the Lesson, I am able to provide extra assistance to students who need help.

4.3

.7

95

I introduce new concepts with hands-on activities.

4.2

.8

96


40

Appendix B

Instructional Practices

Mean

SD

N

Students use math manipulatives.

4.2

.7

96

My students do their homework.

4.1

.7

87

My lessons provide learners with multiple opportunities to engage in activities.

4.1

.7

96

I provide enrichment activities for those students who have mastered the skills.

4.0

.8

96

I lecture or demonstrate to students.

3.9

.9

95

I plan different activities for students who are not mastering skills.

3.9

.8

95

Students are given real-world problems to solve.

3.8

.7

96

Students work individually.

3.8

.7

96

Students participate in math-center activities.

3.6

1.0

96

I use peer tutoring in my classroom.

3.6

.8

94

Students work out of their textbooks or workbooks in my math class.

3.5

1.5

95

Students work in cooperative groups.

3.5

.7

96

My lessons are teacher directed.

3.5

.9

95

I pose questions that require single, short answers.

3.4

.8

95

I pose questions that ask for explanations that may have multiple answers.

3.4

.8

94

I use “compare and contrast” diagrams in teaching.

3.3

.8

96

Students practice handwriting in my math class. (K)

3.2

1.3

23

I use graphic organizers in my lessons.

3.2

.9

96

I use reciprocal teaching in the classroom.

3.2

.9

87

My students use computers in the classroom.

3.2

1.0

93

I use technology in my lessons.

3.1

1.0

95

I use Think Pair Share in the classroom.

2.7

1.1

90

I use semantic word maps to introduce new vocabulary.

2.6

1.0

95


41

Appendix B

Assessments As Table B-44 shows, nearly all of the teachers reported using written assessments (91.7%). Most (87.5%) also use oral assessment methods. Other assessment methods teachers reported using included performance-based assessment, projects, and observation. Table B-44. Assessment Methods Used (n = 96) Assessment Method Percentage Written 91.7% Oral 87.5% Rubrics 33.3% Other 17.7% More than two thirds of teachers (68.1%) reported they assess students at least once a week. Only 6.6% indicated that they conducted assessment once a semester or less often. See Table B-45 for details. Table B-45. Frequency of Assessment (n = 91) Assessment Method Percentage Once a week or more 68.1% Bimonthly 14.3% Quarterly 11.0% Once a semester or 6.6% less Student Engagement and Classroom Management Teachers reported the frequency with which they employed a variety of student engagement and classroom management practices in their math classes, using a 5-point frequency scale, where 1 = never, 2 = rarely, 3 = sometimes, 4 = often, and 5 = always. See Table B-46 for details. On average, comparison-site teachers indicated that 7 of the 12 practices were used often or nearly always. For instance, teachers indicated that two practices were nearly always implemented, with mean ratings of 4.7 on the 5-point scale: I move around the room when I teach a lesson and I am able to monitor the behavior of my class. For the most part, comparison teachers reported that they possessed good classroom management skills. Teachers reported that students were rarely off task during lessons (mean = 2.2), that students rarely failed to respond to the teacher during lessons (mean = 2.0), and that students rarely held private conversations during instruction (mean = 1.9). Grade-level differences existed for two of the engagement and classroom management practices. Kindergarten teachers were more likely to report their students are engaged and interacting during the math lessons than second- and third-grade teachers (p = .02). Additionally, second- and third-grade teachers were more likely to be frustrated when their students did not understand a concept they taught (p = .02)


42

Appendix B

Table B-46. Student Engagement and Classroom Management Practices Student Engagement and Classroom Management Practices

Mean

SD

N

I move around the room when I teach a lesson.

4.7

.5

93

I am able to monitor the behavior of my class.

4.7

.5

93

Most students are engaged and interacting during my lessons.

4.3

.6

93

I am successful in redirecting students who are off task when I am teaching a lesson.

4.3

.6

93

I change the level of my questioning in response to learners’ interactions.

4.2

.6

92

I review the previous lesson before I begin teaching the new lesson.

4.2

.8

92

Student behavior meets my expectations during my instruction.

4.2

.6

93

Most students listen passively when I teach.

3.1

.9

91

I am frustrated when my students do not understand a concept I just taught.

2.6

.8

93

My students are off task when I am teaching a lesson.

2.2

.6

92

My students do not respond when I am teaching a lesson.

2.0

.7

91

My students hold private conversations when I am teaching a lesson.

1.9

.7

92

Mathematics Resources As Table B-47 shows, one fifth of participating comparison site teachers (20%) indicated that they anticipate completing the entire math textbook by the end of the school year. More than half (57.6%) reported that they expected to complete at least three fourths (76%–99%) of the curriculum. Table B-47. Completion of Math Textbook (n = 85) Percentage of Text to be Completed Percentage Less than 50% 8.2% 51%–75% 14.1% 76%–99% 57.6% 100% 20.0% Teachers at comparison schools reported using a variety of math curricula. Mentioned in order of frequency were Harcourt Math (n = 18); Houghton Mifflin Math (n = 13); Everyday Mathematics, published by McGraw Hill (n = 12); Mathematics: The Path To Math Success, published by Silver Burdette Ginn (n = 8); Bridges in Mathematics, published by The Math Learning Center (n = 6); Investigations, published by Scott Foresman (n = 6); TERC, published by Dale Seymour Publications (n = 3); and Excel, published by Ans Mar Publishers (n = 3).


43

Appendix B

Comparison teachers also indicated that they supplement their math instruction with resources beyond those provided by their primary curriculum. They use worksheets from other curricula to supplement or reinforce concepts as well as materials they developed themselves. These teachers also drew upon other manipulatives and hands-on activities for their students. Some teachers reported that they used supplemental materials to address state standards. Classroom and Teacher Information To explore the contexts in which comparison teachers worked and to better understand the teachers themselves, the survey included several demographic questions. Nearly all of participating comparison-site teachers (97%) reported teaching in a self-contained classroom. A small percentage (4%) taught in multiage classrooms. The average number of students per participating comparison-site teacher classroom was approximately 20 (mean = 19.5, standard deviation = 4.6). Nearly three fourths of comparison teachers (72%) reported that their classes were heterogeneous, with a mixture of two or more ability levels in mathematics. Another 22.6% indicated that their classes were fairly homogeneous and average in mathematics ability. A small percentage of teachers (3.2%) reported that their classes were fairly homogeneous and of high math ability. Very few (2.2%) indicated that their classes were fairly homogeneous and weak in math ability. The average number of English language learners or limited English proficient students in the classrooms of comparison teachers was 4.6 (standard deviation = 7.5). The average number of special education students was 1.8, with a standard deviation of 1.9. The average number of gifted and talented students was 1.5, with a standard deviation of 2.8. Nearly all of the participating comparison-site teachers (96%) were female. One fourth of teachers were between 31 and 40 years of age (24%), one quarter between 41 and 50 (25%), and another quarter were between 51 and 60 (25%). Approximately one fifth (22%) were between 20 and 30 years old. A very small percentage (3%) were 61 years or older. Comparison teachers had taught in a regular teaching position for an average of 13.7 years (standard deviation = 10.5). They had taught in their current district for an average of 10.3 years (standard deviation = 9.8) and in their current school for an average 9.6 years (standard deviation = 9.6). The average number of years teachers taught at their present grade level was 8.3 (standard deviation = 8.2).


44


Appendix C Saxon Math Teacher Implementation Specifics

Appendix C

Videotaped Classroom Observations—Additional Materials and Resources Observers noted the use of additional math materials and resources that were used in the videotaped classroom lessons that were not believed to be a part of the Saxon Math program. Those items are bulleted below by grade level. Kindergarten • • •

• •

• • • • • • • •

•

•

Different songs using patterns/rhythms to learn days of week and months of the year were also used in addition to the Saxon resources. Weather discussed and manipulatives were used to discuss weather topics. Inclusion of good morning song. Also sang months of year in Spanish. Sang days of the week song. Lots of inclusion of music and singing in The Meeting. Inclusion of weather in the meeting. Inclusion of singing state song and recognition of state symbols. Included review of weather and season in the meeting. Also, counted by 5s during the meeting. Class went over menu. They recited their class motto. Went over the weather and placed pictures of current weather on board (i.e., cloud with rain). Recited the months in Spanish also. Counted the number of days of school using bundles of drinking straws. Used a tape/CD of music to say months of the year, days of the week, and to count to 20 during the meeting time. During the meeting, teacher signed to students as to what the next day’s colors would be. During meeting, put clothes on paper bear to show what the weather was like. Used straws to count the number of school days. Included a review of Xs on a 10 frame. Used clock on meeting board to tell time but was not supposed to be a part of the meeting. Used bundles of straws to determine the “number of days.” Verbally included information about President Abraham Lincoln and the Lincoln Memorial when demonstrating penny. Teacher used a poem about money when discussing money. The students read the words of the poem as the teacher pointed to them. When counting by 5s, did a marching exercise. Phonetically sounded out segments, shapes, designs, reviewed these concepts when discussing use of geoboard. Lesson practice took extra time because counting exercises were included. Covered shapes and colors as well. Also, discussed weather using various manipulatives. Did some type of attendance activity—had one student do attendance. Expanded lesson by going over pattern ABBC in colors and shapes, and so forth. After the lesson was complete, the teacher had the students break up into groups to make Gack.

S a x o n E l e m e n t a r y M a t h P r o g r a m E f f e c t i ve n e s s S t u d y

1

Appendix C

First Grade • • • •

Used alternative materials for The Meeting skills (e.g., different weather chart, whiteboard for doing calendar activities). Used other worksheets, activities, and lesson plan (not Saxon) to teach the lesson. It is not known from where the lesson came. The teacher did use the Guided Class Practice sheet that was intended to go with the lesson. Used alternative materials for The Meeting skills (e.g., different charts that were not from Saxon). This school uses the materials for the next grade level (e.g., kindergarten uses first grade, etc.). After the lesson and before Guided Practice, students did a 2-minute timed Fact Practice and then graded their papers. This was not a part of what was to be Fact Practice. Teacher noted they did this every day.

Second Grade •

• •

• •

•

Added a “birthday” component to meeting. Does meeting components in a different order than listed. Has part of The Meeting at meeting board, does last part with children at seats as way to transition to Fact Practice. Uses large blue tiles as an end-of-meeting manipulative. Combines a worksheet with lesson and jumps in with out direction as it is, as she says, “continuing lesson,” meaning students are already familiar with it. Teacher models worksheet both verbally and by writing word problems as basic equations on the whiteboard. She adds one extra “practice” question at the end of the lesson worksheet. After The Meeting, there is a “Problem of the Day,” which is projected onto a television via teacher’s laptop computer. She mixes the fact sheet and The Meeting together. She did an exercise after the lesson but not the Guided Practice. This exercise included some elements from the meeting. After Guided Practice, she does more of The Meeting. This time she has students sit on the floor in a circle. Teacher has students fill out Side B of Fact Practice sheet timed and front side untimed. Used overhead-projector for Guided Practice sheet A. Used some meeting manipulatives but did not have a full meeting board (not recommended size smaller) combined board materials (off to right of whiteboard) and whiteboard interaction. Did not have Saxon money. Uses a drawn representation on whiteboard. During lesson, does not do everything but does most recommended in the lesson. Varies some from script. Uses manipulatives instead of having each student complete it on paper as suggested (one student at a time in front modeling for whole class, students total do this). In morning, teacher gives out an additional two-sided worksheet for students to do for review first thing before meeting. After showing the lesson using oversized money on the board, they do Guided Practice. Other variation: In Guided Practice, teacher goes through problems one at a time, but doesn’t correct answers till end.


2

Appendix C

• •

• • •

•

•

•

• •

•

After the 2-minute time limit for timed Fact Practice, she allows additional 30 seconds (however, none of the children needed it). Uses a projector to help model in the lesson. During Fact Practice, the teacher had students take a Friday assessment (timed) instead of Fact Practice sheet because (she said) of parent/teacher conferences. It was still timed and was like Fact Practice sheet Side A in all other respects, but she had students take Fact Practice sheet A and B home, so I marked it “d.” Lesson 95, however, specifies that Fact Practice sheet A&B is all homework that day, so she is doing the appropriate thing. During lesson has students do the last two problems from lesson script on their own paper and calls on two students to show their work on the board. Fact Assessment Day = no student Fact Practice sheet given. Teacher gives additional “math cards” manipulatives to students for first time during lesson. Not part of given lesson and do not look to be Saxon Math Fact Cards. Has a few other children in addition to student of the day come up to do stuff on the meeting board. At end of meeting has students return to seats and write numbers by 10 (from 187) on what looks like a supplemental worksheet (this takes about 4 minutes and children work silently). Rather than having students Fact Practice answers to a “buddy” (before timed test), the teacher has students say answers in chorus. Has no The Meeting board manipulatives so she writes everything on the whiteboard instead. Sequences from The Meeting to Lesson with a “pocket counting” group activity = four groups. Has a 3-page worksheet (supplemental, not Saxon) after group activity. The Lesson consists entirely of doing the worksheet and putting the results on the board, with teacher walking around and helping. Has some additional math posters up. Uses Saxon tiles and counting sticks to help students count during the lesson (has student-of-the-day hold them out while students count in chorus) instead of following secret number directions exactly. In Fact Practice, she does timed test first. Has students do some problems on whiteboard during the lesson. Has students lead the whole math meeting! In The Meeting, each student has his/her own clock. Teacher does not teach second-grade lesson #96. The Meeting follows #96 directions. Words (name of lesson) mathematicians used to describe how objects move. Teacher clearly tapes different segments on different days, so pinpointing what lesson goes with what segments impossible. Has students sing some of the calendar and number information. In Lesson, put list of foundation geometry. Solids on overhead projector instead of making charts. Does some elements of meeting right before doing Guided Practice while students sit in seats (just calendar). Does a short exercise with Wrap-Ups after Guided Practice and before lunch. Instead of Wrap-Ups, she uses a very similar class exercise. The worksheet used during Fact Practice does not appear to be Saxon and has no Side B but is used as a quasi-timed, “as fast as you can” Fact Practice. Teacher expands upon the Lesson using repetition and additional modeling.


3

Appendix C

• •

•

•

•

• • •

The Meeting—has students recite calendar information as a song. Mixes some spelling into The Meeting (has them spell “Wednesday” by standing in front of class). Note: I think The Meeting started for some time before the tape began running. During Lesson, lets students choose what objects to weigh against the paperclips rather than just the chalk. Tape stopped before Guided Practice completed. She does not have some of The Meeting board materials and writes the appropriate corresponding charts/activities on the whiteboard. Has an “odd number chair” in The Meeting. Instead of lunch/attendance activity, she has a “days we’ve been in school” set of activities. Also has an additional “birthday equation” component in meeting there, students try to think of multiple ways to equal a number (the date of a fellow student’s birthday). She has students keep a math journal that they record stuff in during the Lesson. Uses Wrap-Ups in Guided Practice as an independent exercise. Work tool for when students finish their worksheets and she is helping other students. Teacher goes over Fact Practice questions and answers orally before having them complete the sheet—this is in the Fact Practice instructions. Lesson cut off before entirely completed. During Guided Practice, has some student come up to projector to help with going through answers with the class. Uses an additional whiteboard in back of room for The Meeting board materials and showing examples. Uses an easel with large paper pad to print up The Meeting components and activities. For Fact Practice, uses a game with kids in a circle and flash cards manipulatives instead of Wrap-Ups. Uses overhead projector for Guided Practice.

Third Grade • •

• • •

A timed exercise (not Saxon) is done at beginning of tape before The Meeting. Were some additional handmade posters used in The Meeting. Does a timed exercise (multiplication facts) before The Meeting. This was the Fact Practice, but again she did not introduce anything. Does a newspaper article exercise; they look for ads with grocery items going to the store. They choose items they would like to buy and the prices, then practice adding. This actually is the Lesson—she just did not introduce what she was doing at all so it was confusing. Does both sides of Guided Practice. Uses pieces of candy as incentives during questions/answer periods. Does more recitation with calendar than lesson calls for. During Fact Practice, teacher pairs students to practice together. Extends Fact Practice with an additional worksheet. During The Meeting, adds a lunchcount ratio component.


4

Appendix C

• • •

• •

•

• • •

•

•

• •

FYI: Lesson 95 had fact assessment, not practice, so she follows different rules rather than reading answers or having students read them. The procedure she follows does exactly match the Saxon directions. Had some additional homemade meeting manipulatives (a bus cutout…) did not observe it being used, though. Hands out additional worksheets at the end of The Meeting and has students work on them. Before fact assessment, uses teach fact cards in modified way, has a child stand at the front with cards to call on other students. After fact assessment, she goes over homework from previous night, then she gives a test. Does not appear to be a Saxon test, takes 15 minutes (that 15 minutes is taped). Hands out an additional practice worksheet after Guided Practice. Has a bulletin board filled with tacked-up index cards (seems to be for helping students remember) with math words written on them (like, equally likely). Also has some additional non–Saxon Math posters on the classroom walls. At the end of Guided Practice, teacher hands out a supplemental related worksheet for students to complete on their own in class. Has additional 3-D shape and angle banners above meeting board. Does an additional oral practice component during the Fact Practice using her own fact cards. Adds a practice worksheet into middle of the Lesson so students can practice what they have just learned. Has an additional component called Math Facts Wizard (more review than Saxon offers). Instructor does give 2 to 3 minutes for showing about what was learned, but she does it after the Guided Practice, not after the Lesson. Looks like two additional math posters (non-Saxon) are a graph chart used in conjunction with temperature to make graphs. Used additional supplemental worksheet after the Guided Practice (that portion used in conjunction with Guided Practice worksheet). Reviews homework worksheet that does not look like Saxon or to the appropriate lesson. After this, she does a number sentence exercise with several kids at the board. Does part of The Meeting board on overhead projector, makes significant modifications to The Meeting directions. Tape starts with teacher going over a worksheet on angles and fractions (4 min.). Does not seem to be a Saxon sheet. Does times/multiplication facts (5 min.) orally w/partners (seems to be in place of Wrap-Ups). Number of dyads recite to each other (4 min.). Uses a variety of maps for lessons. Uses math journals in assistance with Guided Practice (ostensibly) to help students retain knowledge. Has students do some of The Meeting components on paper at desks, such as patterns, in clusters, then goes over them with class. This makes up the meeting (major modification). Uses Guided Practice sheet from #125 part 1; 125 part 2 has no Guided Practice section. Teacher mentioned doing Guided Practice after lunch. Uses a white paper easel during meeting to help represent some The Meeting components numerically. Also uses it to show/model components where manipulatives would be used. Instead of The Meeting masters, each student


5

Appendix C

• •

has a small, white dry-erase board and a marker so they can follow along with The Meeting and uses them again for the Lesson. At end of The Meeting, two high school–aged tutors come in and sit by two particular students. She lets students complete Side B of Fact Practice for bonus if they want to, since it is Friday. Uses overhead projector to display a Fact Practice sheet that she fills out as one student at a time reads on a line of the answers one at a time until those problems are solved. They go over incorrect answers, the teacher and the student reading the answers. Uses additional big white sketch pad on easel to help with The Meeting. Has an additional thinker problem to meeting component. Uses cards of some kind while doing her Fact Practice. Instead of Wrap-Up, she hands out cards with numbers on them and says either 2 or 7, while indicating a child. The child called then multiplies number on card with number called.


6

Appendix C

Experimental Teacher Open-Ended Survey Responses Did you use other materials to supplement your mathematics curriculum? If yes, what materials did you use and for what purposes (e.g., what curriculum areas did the material address)? If not, why not? Gr K K K K

K K K K K

K

K K K

K K

K K K K K K K

Response No. Yes, all curriculum areas, further practice for students. I used my own calendar materials for the meeting. I often incorporate quality literature when appropriate to integrate the curriculum. I used songs to enrich the lessons and incorporate other modalities (Dr. Jean Feldman CDs are great). No. Yes, I use practice worksheets for basic facts, practice of writing numerals. Books relating to concept (i.e. Grouchy Ladybug, Evie Carle, to teach telling time to the hour). Worksheets and manipulatives to practice more difficult skills such as time, money, story problems, addition, and subtraction. Yes, Drop In the Bucket (sequencing, correspondence); Addison Wesley (computation skills, time, money, geometric shapes); Scott Foresman (numeral writing). To teach a sphere, cone, and cylinder, I used a globe, a coke can, and an ice cream cone. My lesson materials for the lessons were not ordered or delivered to me this year. Yes, I used materials I had prior to implementing Saxon: counting bears, linking cubes, clocks, real coins, geoboards, tangrams. Yes. Our state requires writing numbers to 30, so I made worksheets to allow for writing 1 through 30 instead of stopping at 20 as Saxon presently does (needed earlier in year). I developed my own positional words lessons. Math Their Way. Math through literature at end of year. I used other programs also to include more “games” to teach the objectives (e.g., Time game, adding game with bears, extra pattern sheets from old chapter math books). Yes, I use worksheets that I have created to reinforce the concepts that I am teaching. Yes. Worksheets from elementary reproducible Web sites for homework. Methods and/or concepts from Investigations math program. Yes, I used materials to support VA SOLs. No, I had all materials at my school I needed. Week-by-week math essentials, provided by NCDPI. I use the manipulatives provided by Saxon. Math meeting, I use stickers on arrows to assist in counting by 10s. Yes. I also incorporated some lessons and activities from Math Their Way. I have some math games that can be sued at our Math Center for independent play and practice.


7

Appendix C

Gr K K K K K 1 1 1

1

1 1 1 2 2 2

2 2 2 2

2

2 2

2

Response No, had enough with Saxon. Yes. Multiple Fact Practice activities and many counting book and other practice sheets. No. We have other hands-on materials that we supplement or theme-based activities. We use Saxon primarily, though. Yes, various supplemental math worksheets. I needed more work time for time, money, addition, and subtraction. Old workbook pages for any concept the children were having trouble with (e.g., counting dollars). Geometry—district adopted curriculum. Yes. I have a wealth of materials—counters, bugs, cars. I often rotate supplies and materials to add a spark to lessons in order to perk interest. I used them in areas of math facts, probability, and graphing. Yes. I used other materials to supplement Saxon Math. Various workbooks for additional Fact Practice, practice for telling time, counting, money, and lots of supplemented activities for place value. Actual money, precision teaching timing sheets (math facts), clocks (time). Supplemental, math their way materials for lesson extensions or areas that need to be covered further. Yes, our school uses Harcourt Math Advantage as a supplement to Saxon. No—enough materials were given with Saxon. We sometimes played games for additional math Fact Practice, clocks, and money. Yes, we also use Math Facts in a Flash. This is a software program that helps with math facts. We also use other math material for a.m. papers. We use these to help with the review of time and measuring, etc. Yes, Accelerated Math is used in addition to Saxon Math. I used the math centers to review prior knowledge such as measurement, addition, subtraction, and word problems. Additional practice sheets sometimes needed. Additional coin sets needed for lessons. No, Saxon Math has plenty of materials; sometimes there is not enough time to cover everything in the lesson (in the proper suggested way). I did use playing cards to practice basic addition and multiplication facts instead of the Wrap-Ups (sometimes). I showed clock slides for telling time with the slide projector (I have had these slides for years). I used worksheets with 2- and 3-digit numbers with regrouping. Also, worksheets for subtracting 2- and 3-digit numbers with regrouping. I use book that have blackline masters to practice 2 and 3 digit adding and subtracting for more practice. Also, I use much more practice counting money. I added a problem-solving unit. I would introduce several problem-solving strategies. Then students would solve problems daily using the strategies. Depending on the lesson, I would use other manipulatives to teach to all the multiple intelligences in my room. We use other materials for more practice of individual skills (e.g., whole sheet of borrowing, more graphing practice).


8

Appendix C

Gr 2 2 2 2

2 2 2

2 2

2 2 2 2 2 2

2 2 2

3 3 3 3

Response Transparencies of the Guided Practice were used to help students with disabilities. Bingo games for addition, subtraction, and telling time used to supplement learning. I used various regrouping practice sheets in addition to Saxon materials. Yes. I used games, bingo, chalkboard challenges, Bellwork, math trade books, etc. I used them for money, clocks, measurement, regrouping, and fractions. I used math worksheets from other sources when I thought children needed more practice. I used the made worksheets that were more in VA SOL format. I made a pretest and posttest each 6 weeks in SOL format. It varied. If I saw an area of weakness in my classroom, I would pull more work on that so they could master it. Worksheets—place value, regrouping, time, money. I used extra worksheets in areas students were having difficulty in, along with extra instruction. We ordered math manipulatives that went with the lessons (pattern blocks, dominoes, color tiles, etc.) I use math worksheets that I make or copied from old math series to practice the skills learned each week. We do hands-on activities (measuring objectives, money to count, games ordered) to practice skills. Marcy Cook Math—manipulates the numbers. Math Warm-Ups: Developing Fluency in Math, Grades 2 and 3—Prufrock Press. Used occasionally in class or for homework; these short 10-problem lessons reviewed such things as fact families, money, clock work, and various analogies. Yes. I used supplemental worksheets/activities to reinforce the major concepts covered by Saxon. Yes, because of the state K–2 assessment. Saxon doesn’t touch on all of the objectives the students have to know. Yes, I made a chart for a “Brain Boggler” (problem area or tricky problem). Base 10 blocks, plastic coins. I used addition/subtraction games to reinforce facts. TERC, area, shapes, halves, symmetry. Used while some students were involved in other testing. Yes. I supplemented the mathematics curriculum with the NC standards course of study. Saxon doesn’t cover all of our standards to help prepare the students for EOGs. I used the week-by-week essentials mathematics to help with problemsolving skills. I’ve used centers with extension of what has been learned and taught in SM. Teacher made. I also bought small clocks to help with teaching time. Yes, we have Math Advantage workbooks at our school to help supplement Saxon. We do this so we can meet our benchmarks on schedule since the Saxon program doesn’t. Yes, more drill in multiplication sooner. We were working toward a goal of 100% knowing 0-12’s before school ended—we made it! Computer drill. Dry-erase marker boards—on a regular basis I am able to observe students understanding immediately. Yes, I use supplemented materials in all areas. Star Math—Accelerated Math computer program. My school uses Math Facts in a Flash and Accelerated Math to supplement our curriculum. We are trying to boost our math scores.


9

Appendix C

Gr 3 3 3

3

3 3 3

3

3

3 3

3 3

3 3

3 3 3

Response Yes, Accelerated Math. I used this program to reinforce concepts that had been taught in Saxon Math. I used multiplication bingo to help students review multiplication facts. Yes, I did use other materials to supplement Saxon. I pulled other worksheets from my files sometimes to provide additional practice for some skills (elapsed time, telling time, long division, 3- and 4-digit subtraction and addition, used a money kit for counting change and money amounts). Not really. I used playing cards for a change of pace when practicing basic addition and multiplication facts. Sometimes I would send home extra computation problems taken from other math series for extra practice. More computation practice was required. I do use extra multiplication and sometimes clock games as incentives to make these more fun. Rectangular flashcards for addition, subtraction, multiplication, one set per student for take-home use/study. Rectangular flashcards (large size) for whole-group use. We also used the Saxon triangular shaped set. Multiplication bingo games for learning reinforcement. I used some accelerated pages taken from other resources for my highest math achievers. Computer program that dealt with math concepts. Used Saxon 54 much of the year with majority of the students. Others used Saxon 3—one child used HEATH. Heath assessment with child with language disability. One student also uses Math 64 book. Sometimes students needed extra practice on some skills—like multiplying 1 digit by 2 or 3 digits with regrouping. I would run off sheets of practice problems from a reproducible book. There were no areas of the curriculum that weren’t covered that I identified. I just need some extra practice of some skills. Yes, I used some Excel Math sheets for added practice in geometry and solving word problems. Third grade teachers were told to use Saxon Math as supplement this year. However, I would do the math meeting everyday, and teach Saxon Math lessons that went along with our school’s math pacing guides. Yes, extra multiplication papers, measuring papers time Yes, I pulled resources to reinforce the lesson taught. I especially had to pull resources on fractions—equivalent and lowest terms. Also had to pull resources for geometry. Shorten the lessons. I spent 1 to 1.5 hours or more some days on The Meeting, Lesson, and Guided Practice. Yes. Materials are mostly created by one. For example, combinations are an AZ standard. I also introduced some concepts early in minilessons or during math meeting. No, because I was hoping Saxon Math was all inclusive for the material needed to be covered. Yes, I used the Measuring Up and Silver Burdett Gin for lessons Saxon did not provide and also for additional practice. I used some books (picture) to clarify concepts or further explain ideas with geometry.


10

Appendix C

3 3 3

3 3 3

3 3 3

Materialists teach more about geometric shapes. Use Study Island computer program to teach some objectives. Multiplication program—to help students learn facts (reinforcement). Flash cards. Fractional plastic shapes—fractions; Small Judy clocks—clocks; Coin cups— money, change, counting coins; Math games we made to reinforce Fact Practice. We made additional review dittos for money, multiplication, division, etc. Read it! Draw it! Solve it! Problem solving activities. Investigations—flips, turns, and area; Turtle paths to increase awareness of geometric sense. Extra worksheets to cover the SOLs Saxon missed. I used state resources to supplement instruction (NC Strategies for Instruction, Buckle Down). I also used additional daily math practice sheets and daily word problems. Yes, MacMillan/McGraw Hill, utips online testing, other math workbooks. Mainly supplemental multiplication and division papers. I used Judy clocks to further instruct on time concepts. I also use base 10 blocks to identify place value and unifix cubes for division concepts.


11

Appendix C

What aspects of the Saxon Math program made you or your school decide to adopt the program? Gr K K K K

K K K K K K K K K

K K K K K K K K K K K K K K 1 1 1 1

Response I was not here. Saxon Math program was already in place at this school when I was hired. Very easy to follow, well-written lessons. We enjoyed the Saxon Phonics program so much that we were curious to see how our students would do with the Saxon Math program. The scripted program (Direct Inst.) works really well with our student population of over 70% free/reduced lunch. It is scripted, has distributed practice, and teaches skills better than other programs, for example, story problems. Teacher-friendly lessons are laid out well. Recommendations from other teachers. I don’t know, it was done before I had a choice. Repetition, recommended by other teachers. Hands on materials, Side B skills reinforcement, spiraling of skill development. The repetition of the program really helps students grasp the skills. I was not a part of the adoption process. It was district adopted. We observed this program at another school with similar demographics. They had high test scores in math because of the use of this program. We liked the use of manipulatives. Direct instruction, manipulatives, lesson sequencing. The lessons are planned and ready to be taught. A teacher encouraged our school to use it because she said it was successful at the other school she worked at. I’m not sure. Saxon was adopted prior to my employment at this school. Consistency throughout Grades K–3 District mandated. The use of manipulatives, hands on, big worksheets with class. Saxon was already in place when I joined staff. County decision. The flow, logic, and repetitiveness of Saxon Math are a definite positive. Guided lessons and practice. Spiral learning—not chapters. Consistency is great! Do not know—district wide. Our district wanted to implement it but our school kept the program in kindergarten because as teachers we loved the program. It is worthwhile to our kinder program. The teachers liked the way the lessons were scripted. Recommendation of another teacher. I liked the repetition of math facts. I used the Saxon Phonics and loved it. I went to a presentation on Saxon Math and talked to the other teachers about using it. The reinforcement of homework nightly and their parents’ involvement in the program. Saxon is very structured. This program works well with small children. The new increment taught each day was great as well as the review activities. Saxon is a building program which is great. The homework portion helped parents be aware of what I was teaching at school.


12

Appendix C

Gr 1 1 1 1 1 1 1

1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 2 2 2 2 2 2

Response Comprehensive, teacher friendly, spiraling. The review of each concept. The distributive practice of skills. The many things that are covered. Recommendations from other teachers. Teacher-friendly lessons are laid out well. The fact that it is complete, incremental, and fun for the children. I was not present at this school when the school adopted the program. The structure and repetition (increasing in difficulty) is unique and very well designed. The assessments are timely and test the material taught. The whole program is ready to go from day one! I love that everything builds. There is a repetition factor. Everything is there for you. The repetitive nature of the program, the set terminology to follow, and the structured nature of the program. We liked the morning meeting and spiral review. The district made the decision to adapt the program for all schools. It’s very repetitive and consistent—covers all SOL. The majority of lessons fit our state standards. Review built in. Ease of use. To be honest, I am not sure the principal decided on this program. I guess he liked what he saw. He is happy with the results. I’m not involved in determining our school’s adopted curriculum. I’m not sure. Saxon was adopted prior to my employment at this school. The day-to-day building on knowledge acquired the previous day. Lots of review on worksheets. N/A The program spirals and builds on itself. It aligns to our TEKS (state requirements). Scripted, sequenced, preferred by upper grade teachers. We had no choice. We were told by the county to use it. How it reviews each day and it keeps building. How it uses different methods for teaching concepts. If a teacher is absent a sub can continue on with lesson. The tight spiral and repetition of practice of Fact Practice—constant assessment tests. The structure and repetitive nature of the program. Saxon follows our Utah core for math. We also like the homework that goes home each night. Its easy to follow, implement, and determine the needs of each child. Scripted lessons, assessments. The adoption of the Saxon program was a district/regional decision. I am not sure. It was adopted before I was employed by the district. I was not teaching here when this was adopted. I was not here when the program was adopted. All teachers are teaching the same content, so all students are sure to get the same instruction. Repetition, accuracy. Teacher friendly, good use of manipulatives helped fill in gaps of things not being taught in old series.


13

Appendix C

2

2

2 2 2 2

2 2 2 2 2 2 2 2 2 2 2 2 2 2 2

2 2 2 3 3

Another teacher’s child had the Saxon Math program in her school. We were impressed with how the program worked (meeting, Fact Practice, homework). The fact that the class practice and homework were similar helped the students. We also liked the review from previously taught lessons. I liked not having hard-cover books! The math skills and objectives that the Saxon Math program contained helped us prepare for our Georgia Criterion Reference Test and the Iowa Test of Basic Skills (that we used to give, but no longer give to our students). Repetition of skills, daily Fact Practice. Results from surrounding schools proved Saxon was effective. We also had Saxon in middle school and wanted to have the same series for all the grades. It spirals and gives lots of practice of many skills all year. Once you add you don’t leave that skills and go on to the next. The thoroughness, repetition and overlapping of concepts, the highly structured comprehensive organization of the program, adequate practice and manipulative materials. We liked the spiraling review. The district selected it as a way of maintaining a consistent program throughout the entire district. The continual review of skills and program format would help all our students improve in math. We felt like it correlated well with the NC curriculum. The school liked how concepts were repeated on the Guided Practice and also our students were showing good results. How the program repeats even in later lessons many skills. The way the skills are taught, how they often connect to others, some making understanding easier. Administration The program was well structured and we liked the incremental development. The organization of the lessons, the building-block approach to skills from one grade level to the next. The repetition throughout the program beginning in kindergarten and going up. It’s been successful with similar populations. The sequential building of skills, manipulatives, and user-friendly resources, materials, etc., for both teacher and students. The layout of Guided Practice with a homework side. The constant practice of a variety of skills. The division made this decision. I believe that Saxon Math provides an excellent foundation in mathematics skills. Our school uses it through all grades for the consistency and the foundation it provides. Review, across the board (throughout the district). We’ve had good end-of-level test scores throughout the district. I love the structure! The fact that Saxon Math uses the same concepts being taught on state testing. I am not sure why my school adopted it. My principal said is was required. The application to making math relative. The drill and completeness of the materials needed to have success. N/A, I was not teaching in this district when the adoption was made.


14

Appendix C

3 3 3

3 3

3

3 3

3

3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3

The scripted format. The teachers like it. It’s easy to implement. The fact that the program is scripted. We wanted to make sure we used the same terms across the board and to assure that all students are being taught the same concepts. The ease of using the program and the hands-on lessons. Our district was ready for a change. We wanted a series that allowed our students the opportunity to learn more skills. In traditional math series, you learn by units and may not see another practice on that particular skill or concept until to get to a chapter review. We were really sold on the amount of skills that were taught with the distributed practice built into the program. We saw an example of a student’s homework page. We liked the way there was a variety of problems and what was being taught. We had to pick a new math series and we decided to try the Saxon program. It was piloted by one teacher per grade and then adopted by our county. We liked the timed fact sheets, the homework on the back of the practice sheets, the materials and manipulatives were easily available. Also the high school was already using Saxon and they liked it. The repetitive, structured nature in which concepts were presented and reviewed. We felt with Saxon we would be less apt to miss key math learning. It’s a complete package. Don’t know. I wasn’t involved in adoption. I believe it was the thoroughness and ease of implementation. We really like the ongoing review. With other programs the students forget the first of the year by the end of the year. All teachers would be following an organized program. The structure of the program and the success rate of other schools that had used the program. Not sure, it was given to me to use. We all like the spiraling that Saxon Math offers. I was not part of making this decision. I don’t know. Program is well structured; liked the incremental development. N/A It was a decision made by our school division. It fits into our curriculum and challenges the students at the appropriate level. Repetition of objectives to have mastery. The amount of review and in-depth coverage of each topic. Gives you a lot of guidance in each lesson (good for new teachers). Repetition was excellent. Progression of lessons is in logical order. Higher assessment scores recorded on state tests. Ease of material use. Consistency across grade level. Very complete lessons—very well organized. We didn’t make accreditation The spiraling curriculum, the constant reviewing of concepts. Daily review of previous skills, daily homework, good coverage of Utah core. The repetition of concepts, its consistency, raise test scores (and it did). The repetition of the concepts also a district by area adopted program.


15

Appendix C

What do you like best about the Saxon Math program? Gr K K K K K K K K K K K K K K K K K K K K K K K K K K K

K 1

Response Easy preparation, manipulatives provided, great lessons, meeting format. Student practice sheets are already prepared. Easy to follow—if a sub had to come in, they could easily teach the lesson. I like the focus on handwriting (i.e., correct formation of numerals). I like the repetition/review of prior lessons throughout the year. All of the above. Manipulatives, repetition, the way the lesson is written, easy for a sub, too. Manipulatives Repetition reinforcement built into the program. Covers a wide range of mathematical concepts kindergarten children enjoy learning about “Congruent” and Symmetrical” concepts. The fun ways of teaching math using a variety of great manipulatives, and the worksheet practice really reinforces the skills. I liked the scripted lessons and the layout or format of lessons. I liked the sequence of lessons, ways the materials were organized, and the lesson practice sheets. The Meeting. The manipulatives, continued practice. Most the of the lessons correlate with the NC course of study. I also loved the use of manipulatives followed by a worksheet to reinforce the concepts taught. Direct instruction, review sequencing. The lesson is planned. The worksheets help them get extra practice. The program keeps the parents informed. They get the review all year. I do like the review Saxon builds on. A wide variety of topics. The Meeting—repetition of skills and structure. Step by step worksheet, hands on materials. Build on skills, manipulatives, hands on. I like using the manipulatives and then applying those skills used for the paper/pencil activity. Availability and variety of materials. Easy to follow. Easy for substitute—premade worksheets. The layout is very specific. Objectives are clear! Meeting board. I like how organized the program is. I like having the worksheets (lesson practices) already made and ready to use. I like the easy format to follow for most of the lessons with the materials already made. I liked the scriptedness, also. Repetition of math facts, counting, odd, even, etc. I like the lessons and the fact that we use manipulatives a lot.


16

Appendix C

1

1

1

1 1 1 1 1 1 1

1 1 1 1 1 1 1 1 1 1

1 1 1 1 1

1. Repetition of skills learned. 2. Sequence is natural. 3. Students enjoy the lessons. 4. Test reflect skills taught. I like the fact they are taught to add and subtract together on a Fact Practice. Also, the problem solving skills they learn are fantastic! I like the mystery bag problem solving second mental computation. The students really get involved in this and enjoy it. Most everything I need is provided in the kit. It is great for a substitute to follow. A small increment if new learning and the continual review is so easy for the students to follow. Ease, fact instruction. Materials are ready made for the teacher. The concepts are reviewed often. Children feel successful and not afraid of math. Repetition, manipulatives, the way lesson is written, easy for a sub, too! The scripted lessons that meet all our state standards and then some. For the younger grades, I like all the handouts that are ready to go. I love the math Meeting, which reinforces skills needed in daily life and taught in the lessons. I think the manipulatives are awesome, it is so nice they are included in the program. All skill levels find satisfaction and a sense of accomplishment in the daily lessons and homework! How it helps you plan ahead. The repetitive nature, the easy to follow guide, and the already prepared worksheets. I liked that Saxon Math always reviews previous concepts in every lesson and then builds on new concepts. I like the meeting time to preteach skills, the Fact Practice for automaticity, and the way it constantly reviews concepts. It is structured. Repetition. Manipulatives for each lesson. Wrap-Ups. Kid Fact Cards. Lessons are easy to implement. Manipulatives are varied. Children seem to enjoy math. That every thing is laid out for you, it gives you all you need for each lesson, and it is easy to expand on. The “Meeting” because it gets our day off to a great start. I like the review Saxon does on the guided and homework practice. The material is taught in a systematic way. The approaches to solving problems are logical and easy for students to use. It allows all students to feel successful. I like the continuous reinforcement of previously learned concepts. I also liked the routine of the morning meeting. TE is set out in an easy to read format. Program follows a logical, spiraling order. Vocabulary is consistent throughout. Manipulatives, math facts sequence, weekly test. The basic skills that are taught.


17

Appendix C

1 1 1

1 1 1 1 2 2 2 2 2 2 2 2

2 2

2 2 2 2 2 2 2 2 2 2 2 2 2

No Wrap-Up for first grade—too hard for them to do. The tight spiral and repetition of practice of Fact Practice—constant assessment tests. I like the gradual introduction of skills and the fact that they are constantly being revisited. The fact that you have just about everything you need for a lesson at your finger tips is great too! I love the daily repetition of learned skills. The child has homework each night that reinforces the skills that were taught that day. Covers most standards, easy to assess. The program is very interacted, well planned, and thoroughly covers our first-grade curriculum standards. Very thorough. The preplanned lessons. A lot of hands on manipulatives for students. Parentfriendly home sheet. I like how you walk through every step with the students and how the homework is setup for the parents and students. Manipulatives and practice are the things I like the best. It is very organized and allows parents to be involved. Math Meeting good morning practice. Each day do Fact Practice and lesson. The scripted lesson, easy for a sub to come into the classroom; daily Fact Practice; the organization of the program; assessments after every five lessons taught. My favorite part about the Saxon Math program is the way the math skills are introduced and then reviewed throughout the year. The review helps the student retain the knowledge they have learned. Repetition of skills, daily Fact Practice. I like repetition of Saxon. Concepts are included in the daily lessons for several weeks. Students are less likely to forget concepts. I also like the Fact Practice. Daily practice benefits the students so much. Very easy to plan for—everyday routine is good for students to know and follow. Plenty of materials, the way concepts are reintroduced and reviewed, assessments that are included. The connection between 10s and 1s and dimes and pennies and the spiraling review is what we like best. I like that it builds upon itself. It also teaches the skills in small, manageable bites, so all students can be successful. I like the Guided Practice on one side and homework on Side B. Continual review of skills. Fact Practice sheets and timed test. It builds on itself. The students enjoy it. They are excited about Math. The repetition of skills. Spiral. The students are successful and they all love math. I can adjust the pace and depth of the lesson to suit specific groups of students. Easy to teach, repetition. I love the spiraling! The facts practice is terrific, too. The sequent of skills, repetition, and manipulatives.


18

Appendix C

2 2 2 2 2 2 3 3 3 3 3 3

3

3 3 3

3 3 3 3 3 3 3 3 3 3 3

The layout of Guided Practice with a homework side. The constant practice of a variety of skills. I liked the scripts and the Fact Practice. The Wrap-Ups were fun and the students enjoyed them the most. I like how Saxon Math is organized for teachers. Plus it is a sequential program. Consistency of review, manipulatives, and a variety of skills taught. It pretty much fits our core. I like the Fact Practice and guided lesson. I have always taught math using the calendar. The morning meeting is also good. I like how there is constant review of materials. This helps the students retain information better. What I stated above. Spiral review, easy planning, consistency. Repetition. Great review throughout the year. Hands-on activities to present lessons. Teacher friendly. The constant review of facts and skills to make sure they are mastered. I like the wide range of skills that are taught throughout the school year. The Guided Practice pages provide distributed practice on the skills that have been previously taught. I also like that my students are having to do problem solving, measurement, and Fact Practice on a daily basis. The lesson plans for each lesson also made it easy to plan for instruction. I like the scripted lessons. It’s easy to follow—the way lessons are very organized. I also like the way story problems are taught and how the problem of the day from the math Meeting fits into that day’s lesson. I also like the fact that students must make corrections—although this can be time consuming sometime. The Meeting board is my favorite component of Saxon Math. Besides all of the things mentioned in number 2, I liked the repetition of things that have already been introduced. I also find the Math Meetings very helpful. Ease of use for teacher/substitute teacher. I’m confident that concepts are “covered” well. All materials are at my fingertips. Easy to track student progress and discover weak areas in their learning or my teaching. Review of concepts, I like the length of lesson. With few exceptions, everything I need is there. Also, we are sure that all students are exposed to the same work. The small steps. Children seem to really understand the concepts much quicker. It is well structured, repetitive, and covers all of the SOLs. The Meeting board. Elapsed time and measurement are thoroughly taught. I like the way the materials are copied (handouts, Guided Practice, tests). Students are successful; all the students love math; no “math anxiety.” The direct instruction of method is helpful for students. I liked The Meeting board. It Reviews Material Constantly. Students Cannot Forget What They Have Learned Because It Is Constantly Brought Back Into Problems For Review Or Further Development.


19

Appendix C

3

3 3 3 3 3 3 3 3

Good job of covering third grade fact sheets for multiplication objectives for the state of NC. Meeting-teaching of objectives-skin counting, telling time, patterns (students like math). Amount of reading involved, which is just like third-grade EOC. Saxon gives lots of great strategies for various topics. Saxon revisits concepts taught on a more day to day basis. Lesson plans are excellent for substitutes. The dittos are included. Large charts available. Answers to tests and daily worksheets provided. The sequence helps student build on prior knowledge. Previously learned concepts are reviewed before moving on higher levels. Lesson plans are very complete. The consistency. The pacing and review of content. The reasons I stated in answer 2. I love the Side A/B component, easy to use, kids are learning the concepts. The repetition of skills and the daily math facts.


20

Appendix C

What changes, if any, should be made to the Saxon Math program? Gr K K

K K

K K K K

K K K K K

K

K K K

Response None. 1. More emphasis on handwriting practice of individual numbers. 2. More emphasis on addition and subtraction problems. 3. Arrangement of the introduction of time and money (done too early in school year). Happy with program. There is a lot of passing out and taking up of materials that is time-consuming but necessary because kindergarteners tend to focus on what is in front of them (within reach) rather than on the teacher. I do appreciate the repetition/review of the lessons; however, I would like to completely cover a “unit” or topic rather than skipping around so much. In other words, I would like to teach “pattern” completely before moving on to something else. Extend the kindergarten curriculum to accommodate all day kindergarten. We finished in March. Spend more time on a specific task; don’t introduce a concept and move on only to bring it back up 3 weeks later. Possibly harder skills in kindergarten. Worksheet’s lines for writing are too large for students to write properly. Companies always think BIG for kdg. This is not always true. The “big” spaces are harder for kdg. To control. Need additional computational worksheets. Possible supplementary book for higher students. Overall, I love the program. Supplementary lessons for pictographs, estimation, time, and money. None. I would like to see subtraction included in kindergarten curriculum as well as more lessons on graphing, patterning, and greater than/less than concepts. At the end, an introduction of the next-grade level format and practice. I would like to see more addition and subtraction number problems. Ordinal numbers need to include 1st through 10th. Positional words activities should be included as lessons—over, under, beside, in front, behind, above, around, in, on. Increase writing numbers to 30 instead of 20 earlier in the year. Our state no longer requires money and time as a kindergarten objective, so less emphasis should be placed on it in kindergarten but more in first grade. More supplemental papers. I have the hands-on format, but they also need to be able to master concepts on paper. I like the first grade, etc., daily papers with review and concept building, I wish kindergarten could have something at their level, especially middle of the year. I would like to see more “games” put in the lessons. Saxon needs to be correlated to grade-level standards more, especially in kindergarten. Supplemental materials can be more challenging for students who have previous knowledge or have grasped concept. One topic can be studied long term instead of stepping around, review at end of year.


21

Appendix C

Gr K

K K K K K K K K

K 1 1 1 2 2 2 2 2 2

2 2

2 2 2

Response This program doesn’t match the SOLs we are required to teach. The program jumps from topic to topic. Kindergarten students need more repetition of skills in the lessons. Focusing on a skill for several days would benefit students. More group work. No suggestions. None. More hands-on activities. More high-level activities for those ready to advance a step beyond the regular lesson. Move at a slightly faster pace and offer challenging activities to enhance learning w/advanced students. Add more number lessons, more counting practice, sorting lessons. Sometimes I feel the Saxon lesson has nothing to do with the lesson and the students are confused. I know it all ties together so I am pleased with the program. However, if we are working on shapes for the lesson the worksheet should have to do with shapes. I believe Saxon should focus more on time, addition, subtraction, and money. I’d like to use more practice in place value. Other than that, I LOVE Saxon! Several areas are covered quickly, areas that are addressed in one lesson and then left for only the Guided Practice. None. Help with additional practice or ideas for intervention for kids who are struggling with a concept. It takes way to long to make it through everything. I think lessons part 1 and part 2 should be two separate lessons. We are asked to do both in one day. Some challenging skills taught right at end of second grade, I feel need to be taught earlier to give additional practice, otherwise skills usually lost over summer. Third grade has a booklet with answers to all the assessments. I would like that for second grade, too. I would like for the subtraction facts to be introduced earlier in the school year. Also, addition, adding 2-digit numbers with regrouping should be introduced earlier. I would like to have more adding and subtracting problems with regrouping. Include more measurement such as grams, kilograms, liter, ounce, pound, quart, gallon, etc. More practice on new skills taught that day. More storage tubs, the sheets and folders don’t fit in what’s provided. I would like to see more of a variety in story problems to allow students to use more problem-solving strategies. I would also adjust to the higher and lower students in a room. Challenge the higher ones and adjust the amount of reading for lower readers. Too much review at early part of year. Cut back on the amount of printed dialogue for the teacher to wade through every day; just print key phrases, points, etc. Keep up the good work. More regrouping, money, and time practice.


22

Appendix C

2

2

2

2 2

2 2 2 2 2 2 2

2 2 2

2

2

3

3 3

It is very easy for far too long. There is no real support for gifted students. They are simply not challenged by the program. Content important to grade-level standards is often very late in being presented. Takes too much time to do all components of the program every day. I would like a pretest and posttest made in VA SOL format to use each 6 weeks or at least three times a year. Shortening the Math Center Time. After a while, some things are not taught any longer (such as fact families). However, the students are expected to know this on their end of year assessments. I would like to see place value taught without the use of money. Students seem to have trouble transferring the information. More word problems—maybe extra pages of only word problems. More information and practice on measuring cups, pints, grams, kilograms, pounds, ounces, etc. Get to borrowing/subtraction and carrying double-digit adding quicker in the year. I’m happy with the program. I would place less emphasis on The Meeting board part of the lesson and develop more hands-on activities. Make Saxon more state based. Correlate with NC SCOS. There needs to be an integration of problem solving—not just equations—and p.s. strategies. Second grade needs a final assessment (comprehensive). Some lessons at the second-grade level just touch on a concept and (I believe) never bring it up again. Maybe those could be left for the time when students use them more. It did not meet all of the SOL objectives; more place value and number sense needs to be provided initially. Saxon Math needs to incorporate more problem-solving strategies for second grade to match the end of the grade tests. I have been concerned that the children need more challenge in different ways of thinking through problem solving. They depend a lot on teacher guidance instead of thinking on their own. It needs to be adjusted for teachers teaching two grades at the same time (i.e., 1–2, 2–3, combination grades). I have had difficulty with the homework aspect of the program. It is reliant upon parent support and an aide or other adult in the classroom. This is not practical. My students complete the homework side in class where they can ask for help. Unfortunately, parents aren’t always there. I do not like the way the drill sheets are the same problems in the same order. Maybe there is a theory behind this, but I think it is a crutch for some. I also think they need some mixed function drill work. Very repetitive for third graders. More practice in regrouping with subtraction.


23

Appendix C

3

3 3 3

3

3 3 3

3 3

3 3 3 3

I feel the program goes too slow at the beginning of the year and then too fast at the end of the school year. I would like some of the material presented earlier in the year so that students are more prepared for testing in April. More time to teach complex skills. The lessons need to be more uniform in the length of time it takes to teach them. I think on the lessons where the written assessment is given there shouldn’t be an additional lesson to teach. I sometimes found it to be too many things to do during the allotted math time. There are some skills that I feel our weaker math students need more initial practice before moving on to the new skill. Maybe providing a math master for more practice would be helpful for the first instruction on difficult skills (elapsed time, clocks, long division, 2–3 digit addition/subtraction/multiplication). Maybe cutting back from 135 lessons to 130 or 125 to allow extra time to practice difficult skills. I found it difficult to find the time to complete all of the oral assessments when students need to be assessed one at a time. I’m not sure what a better alternative would be. Storage contains—three containers do not hold all of the lessons. Would it be possible to get one or two more? Basic multiplication facts introduced earlier—is this possible? Some kind of graph for Fact Practices so that the students could keep track to “see” how well they are learning the basic addition, subtraction, multiplication, and division facts. A monthly newsletter with tips, etc., of what ideas, problems, etc., other teachers have … share ideas … helpful hints … question-and-answer corner. Additional computation practice should be added. On the day of assessments, we have too much material to get through in our allotted math period. It usually takes 2 days to get all of the material finished It is not challenging enough for top math students. They are bored easily because they grasp math concepts easily. It is too fast-paced for low-achieving math students. They need more practice at the beginning. I feel the time crunch. Saxon requires at a minimum, 1 ½ hours per day. I feel the need to complete as many lessons as humanly possible each year. (On the assessment lessons, we used 2 days, i.e., Lesson #115, Part 1 on Monday and Lesson #115, Part 2 on Tuesday). We needed that much time. Language doesn’t always match the state tests. Language is too difficult for lower level learners. I would really like a textbook for third grade. I’ve taught both third and fourth grades. The quantity of materials to learn is huge and it would be helpful for the students to begin to learn the organization aspects of the program while the amount of daily work is less. Extra practice sheets or extra tests for those who don’t achieve 80%. Add another test every 10 days with multiple-choice answers. More in multiplication, time, measurement. The charts (masters) are not large enough for the large class to see. I frequently use transparencies instead. The program in Grade 3 seems to start out too slowly, accelerate around November, and too many new concepts are introduced too late for standardized testing. I haven’t decided whether I like the spiraling. Also, the tests and practice look so much alike, many students don’t recognize the concepts when presented differently.


24

Appendix C

3 3

3 3 3 3

3

3 3

3

3

The program is great. The curriculum needs to incorporate standardized test format and include more metric material. Students only gain exposure to concepts and are not given enough problems to gain mastery. If time could be adapted it would be great. We have 1 hour of math a day, and it is impossible to follow the lesson as outlined in that amount of time. More geometry. More time for math meetings. Manipulatives, coins, fractions, clocks needed. A booklet of additional dittos for reteaching. Different classes have different needs so additional worksheet booklet would be helpful. Overhead lays would be nice to buy. Slight decrease in the amount of homework. Lessons should be gauged so they take about the same length of time. Currently, third-grade lessons ranged from 10 minutes to 45 minutes. That makes it hard to plan for new teachers, at a new grade level, or when first beginning to teach Saxon. More lessons/practice with fractions, measurement (capacity, weight, distance, etc.) and multi-step word problems. Add math drill pages for regroup (addition and subtraction). Add math drill pages for 4–5 digit addition and subtraction problems. Change the fact card-triangles. Go back to flash cards. More supplemental pages, not enough meat the kids need more practice on some concepts—long division. Multiplication is introduced in a way that doesn’t make sense, too late in the year. That certain concepts are more aligned w/district benchmarks, such as multiplication, measurement, and regrouping.


25


Appendix D Instrumentation and Study Correspondence

Appendix D

Sample Letter of Participation (Experimental Sites) Dear Principal; We are pleased to inform you that your school has been randomly selected to participate in a national study of the effectiveness of the Saxon Math program. The Institute for the Advancement of Research in Education (IARE) at AEL is conducting this research during the 2005–2006 school year. This important study will examine the effect of Saxon Math on student math achievement and will help the program’s publisher, Harcourt Achieve, improve its product. Participation in this study will involve the following: • • • • • •

Harcourt Achieve will appoint one of your staff based on your recommendation to coordinate data-collection efforts during the study year. The site coordinator will receive a stipend of $500. One teacher each from kindergarten through third grade who is using Saxon Math will be randomly selected for participation. Each teacher will receive a stipend of $300. Teachers will be asked to complete a survey and participate in a brief interview. Participating teachers will work with the site coordinator to videotape one Saxon Math lesson to allow us to analyze fidelity of program implementation. A random sample of students in the classrooms of participating teachers will complete a pre- and posttest of the Stanford Diagnostic Mathematics Test, Fourth Edition (SDMT). We ask that Saxon Math assessment data from randomly sampled students in the classrooms of participating teachers be shared with us throughout the study.

Please be assured that we are committed to ensuring the anonymity and confidentiality of all study participants. No names or identifying information will ever be reported, and all data will be presented in the aggregate. We look forward to working with you on this important study! Please let us know whether or not you are willing to participate in this study by submitting your reply online at the following URL by Friday, May 27: http://www.ael.org/saxon/ If you have any questions about this study, please feel free to contact Caitlin Howley at 1-800624-9120 (ext. 5459) or [email protected]. Sincerely,

Caitlin Howley, PhD Research & Evaluation Specialist


2

Appendix D

Sample Letter of Participation (Comparison Sites) [Insert Date]

Dear Principal; The Institute for the Advancement of Research in Education (IARE) at AEL is conducting a national study on the Saxon Math program. We would like to invite your school, as a nonuser of the Saxon Math program, to participate as a comparison site to schools that are using the Saxon Math program. This research will take place during the 2005–2006 school year. Participation in this study will involve the following: • • • •

Recommendation of one of your staff to coordinate data-collection efforts during the study year. The site coordinator will receive a stipend of $500. One teacher each from kindergarten through third grade will be randomly selected for participation. Each teacher will receive a stipend of $200. Teachers will be asked to complete a 20–30 minute survey. A random sample of students in the classrooms of participating teachers will complete a pre- and posttest of the Stanford Diagnostic Mathematics Test, Fourth Edition (SDMT) (95–110 minutes administration time).

Please be assured that we are committed to ensuring the anonymity and confidentiality of all study participants. No names or identifying information will ever be reported, and all data will be presented in the aggregate. We look forward to working with you on this important study! Please let us know whether or not you are willing to participate in this study by submitting your reply online at the following URL by [insert date]: http://www.ael.org/saxon/. Our staff will also be following up with you via a telephone call within the next couple of days to answer any questions you may have as well as solicit your participation response. If you have any questions about this study, please feel free to contact Kimberly Good at 1-800-624-9120 (ext. 5449) or [email protected]. Sincerely,

Kimberly Good, PhD Research Associate


3

Appendix D

Memorandum of Understanding (Experimental Sites)

Edvantia, Inc. P.O. Box 1348 Charleston, WV 25325-1348 p: 304.347.0400 or 800.624.9120 f: 304.347.0487 www.edvantia.org

Memorandum of Understanding National Study on the Effectiveness of the Saxon Mathematics Program This memorandum of understanding (MOU) details the responsibilities your school agrees to undertake, as a user of the Saxon Mathematics Program, agrees to as a participant in the national study on the effectiveness of the Saxon Mathematics Program being conducted by Edvantia, Inc. (formerly AEL). Please read the following agreements, sign at the bottom of page 2 and return the signed form to Edvantia, Inc. 1. The school will complete and return to Edvantia, Inc. the participating school information form which requests the designation of a site coordinator and contact information and the names of kindergarten through third grade teachers. 2. The site coordinator will be responsible for distributing the pretests and posttests to the teachers. Each of the four teachers will be responsible for administering the pre- and posttests to their students. The site coordinator will collect the completed tests and materials from the teachers and return the data to Edvantia, Inc. for analysis. 3. The site coordinator will be responsible for collecting from the teachers a sample of Saxon Mathematics Program student-assessment data. The site coordinator will submit the student assessment data to Edvantia, Inc. for analysis. 4. The teachers will be responsible for completing forms provided by Edvantia, Inc. requesting demographic information on the students. This information will include the following: age,

gender, free or reduced-price lunch status, special education status, English language learner status and racial identification. The site coordinator will be responsible for collecting this information from the teachers and submitting the data to Edvantia, Inc. 5. The site coordinator will be responsible for distributing the parental consent forms which are necessary to be completed by parents allowing Edvantia, Inc. to use the pre- and posttests results and the sample of Saxon Mathematics student-assessment data as a part of the study. The four teachers will be responsible for ensuring that the parental consent forms are given to the students and returned to school after having been signed up the parents.


4

Appendix D

6. The site coordinator (or a designee) will be responsible for videotaping one Saxon Mathematics Program lesson for each of the four participating teachers. The site coordinator will send the videotaped lessons to Edvantia, Inc. for analysis. 7. Each of the four participating teachers will be responsible for participating in one telephone interview with Edvantia, Inc. 8. Each of the four participating teachers will be responsible for collecting a sample of student assessment data from the Saxon Mathematics Program. The site coordinator will collect the student-assessment data from the teachers and return the data to Edvantia, Inc. for analysis. 9. The site coordinator will be responsible for distributing a survey developed by Edvantia, Inc. to each of the four participating teachers. The four participating teachers will be responsible for completing the survey. The site coordinator will collect the completed surveys from the teachers and return the surveys to Edvantia, Inc. for analysis. Edvantia, Inc. will provide all required testing materials and instrumentation. This includes the pre- and posttests, blank videotapes, and survey. Edvantia, Inc. will also assume all postage costs. Upon completion and return of the pretest data, the site coordinator and four participating teachers will receive 50% of the honorarium. The site coordinator will receive $250 and each of the teachers $150 at this time. Upon completion and return of all other data-collection requirements (posttest data, teacher interviews, teacher surveys, sample of student assessment data from the Saxon Mathematics Program), the site coordinator and four participating teachers will receive the other 50% of the honorarium. The site coordinator will receive $250 and each of the teachers $150. The total amount to be paid to the site coordinator is $500 and the total amount to be paid to each participating teacher is $300 for fulfilling all requirements of the study as described in this MOU. Your signature indicates that you agree to abide by these requirements. Please sign and return this MOU as soon as possible and either fax it to my attention (304.347.0489) or mail it to the address listed above. If you have any questions about this study, please feel free to contact Kimberly Good at 1.800.624.9120 ext. 5449 or [email protected].

Principal’s Signature

Date

School Name (please print)

Edvantia, Inc. Representative’s Signature


Date

5

Appendix D

Memorandum of Understanding (Comparison Sites)


Memorandum of Understanding National Study on the Effectiveness of the Saxon Mathematics Program This memorandum of understanding (MOU) details the responsibilities your school agrees to undertake, as a nonuser of the Saxon Mathematics Program, agrees to as a participant in the national study on the effectiveness of the Saxon Mathematics Program being conducted by Edvantia, Inc. (formerly AEL). Please read the following agreements, sign at the bottom of page 2, and return the signed form to Edvantia, Inc. 1. The school will complete and return to Edvantia, Inc. the participating school information form which requests the designation of a site coordinator and contact information and the names of kindergarten through third-grade teachers. 2. The site coordinator will be responsible for distributing the pretests and posttests to the teachers. Each of the four teachers will be responsible for administering the pre- and posttests to their students. The site coordinator will collect the completed tests and materials from the teachers and return the data to Edvantia, Inc. for analysis. 3. The teachers will be responsible for completing forms provided by Edvantia, Inc. requesting demographic information on the students. This information will include the following: age,

gender, free or reduced-price lunch status, special education status, English language learner status, and racial identification. The site coordinator will be responsible for collecting this information from the teachers and submitting the data to Edvantia, Inc. 4. The site coordinator will be responsible for distributing the parental consent forms which are necessary to be completed by parents allowing Edvantia, Inc. to use the pre- and post-tests results as a part of the study. The four teachers will be responsible for ensuring that the parental consent forms are given to the students and returned to school after having been signed by the parents.

5. The site coordinator will be responsible for distributing a survey developed by Edvantia, Inc. to each of the four participating teachers. The four participating teachers will be responsible for completing the survey. The site coordinator will collect the completed surveys from the teachers and return the surveys to Edvantia, Inc. for analysis.


6

Appendix D

Edvantia, Inc. will provide all required testing materials and instrumentation. This includes the pre- and posttests and survey. Edvantia, Inc. will also assume all postage costs. Upon completion and return of the pretest data, the site coordinator and four participating teachers will receive 50% of the honorarium. The site coordinator will receive $250 and each of the teachers $100 at this time. Upon completion and return of all other data-collection requirements (posttest data and teacher surveys), the site coordinator and four participating teachers will receive the other 50% of the honorarium. The site coordinator will receive $250 and each of the teachers $100. The total amount to be paid to the site coordinator is $500, and the total amount to be paid to each participating teacher is $200 for fulfilling all requirements of the study as described in this MOU. Your signature indicates that you agree to abide by these requirements. Please sign and return this MOU as soon as possible and either fax it to my attention (304.347.0489) or mail it to the address listed above. If you have any questions about this study, please feel free to contact Kimberly Good at 1.800.624.9120 ext. 5449 or [email protected].

Principal’s Signature

Date

School Name (please print)

Edvantia, Inc. Representative’s Signature


Date

7

Appendix D

Teacher Consent Form (Experimental Sites)


TEACHER CONSENT FORM National Study on the Saxon Mathematics Program October 2005 Dear Teacher: Thank you for participating in this national study on the effectiveness of the Saxon Mathematics Program. Your school is one of nearly 40 schools across the country that has been selected to participate in this study. This study is being conducted at the kindergarten through third- grade levels in one classroom at each of those grade levels. You were randomly selected from your grade level to participate. The study is being conducted by Edvantia, a nonprofit research and development organization, during the 2005–2006 school year. This important study will examine the effect of Saxon Math on student math achievement and will help the program’s publisher, Harcourt Achieve, improve its product. Your participation in this study will involve the following: • • • • •

Completing a 20–30 minute survey and participating in a brief interview. Working with your site coordinator to videotape one Saxon Math lesson to allow us to analyze fidelity of program implementation. Administering a standardized math proficiency test in the fall and spring (80 minutes administration time, kindergarten fall testing time is 30 minutes) to your students. Sharing with Edvantia Saxon Math assessment data from students in your classrooms. Sharing with Edvantia demographic information on your students. This information will include the following: age, gender, free or reduced-price lunch status, special education status, English language learner status, and racial identification.

Upon completion and return of the math pretest data, you will receive 50% of the honorarium ($150). Upon completion and return of all other data-collection requirements (math posttest data, teacher interviews, teacher surveys, sample of student assessment data from the Saxon Mathematics Program), you will receive the other 50% of the honorarium ($150). The total amount to be paid to you is $300 for fulfilling all requirements of the study as described in this consent form.


8

Appendix D

By signing and returning this form, you are giving permission to participate in the abovementioned data-collection activities. By signing this form, you are also giving us permission to use the information you provide in a report we write about the results of our study. Your responses will be kept anonymous and confidential. That is, your responses to the survey and interview will never be associated with you, and data will only be reported in the aggregate. The videotaped lessons will be reviewed in terms of how the Saxon Mathematics Program is being implemented. Your name will not be associated with the videotape and data will only be reported in the aggregate. Only authorized Edvantia staff will access the data strictly for the purposes of organizing and analyzing the results. Data will be stored on a password-protected computer file for 3 years past the termination of this project. Finally, your signature on this form indicates that you understand that you are free to withdraw participation in the study at any time by notifying myself, Dr. Kimberly Good, and that there will be no negative consequences from Edvantia as a result of this withdrawal. Data collected for research purposes are stored in compliance with ISO 17799 requirements for access, security, and redundancy. Data are stored in an encrypted format in a centralized, electronically and physically secure server at Edvantia for a period not to exceed 5 years. All electronic data of a personal nature are safeguarded and available only to those project leaders, staff, and technologists having a need to know within the specific criteria as set forth in the approved project plan. There are no known risks associated with participating in these research activities that are greater than those ordinarily encountered in daily life. Should you feel too anxious, however, please notify me immediately. Information obtained during this study will be held in strictest confidence, with the exception that if a researcher obtains clear evidence of unlawful behavior that could result in physical or mental damage to a minor, the researcher is required by statute to report such evidence to the authorities. Therefore, please notify me, Kimberly Good, at 1.800.624.9120 ext. 5449 of questions or concerns that you may have prior to signing the consent form. If, during the course of the research activities, a question or concern arises, please do not hesitate to ask. Please complete and sign this form. Then return the consent form to the person designated as the site coordinator. If you have any questions or concern, please feel free to contact me, Dr. Kimberly Good, at 1.800.624.9120 ext. 5449, or by e-mail at [email protected]. For information on your rights, contact Dr. Merrill Meehan, Edvantia IRB Chair, 1031 Quarrier Street, 6th Floor, or at 1.800.624.9120 ext. 5432. Thank you!

Teacher’s Name: Grade Level: School Name: Teacher’s Signature: Date: Edvantia researcher signature: _______________________________ Date: _________________


9

Appendix D

Teacher Consent Form (Comparison Sites)


TEACHER CONSENT FORM National Study on the Saxon Mathematics Program October 2005 Dear Teacher: Thank you for participating in this national study on the effectiveness of the Saxon Mathematics Program. Your school is one of nearly 30 schools across the country that has been selected to participate in this study as a comparison site (a nonuser of the Saxon Mathematics Program). This study is being conducted at the kindergarten through third-grade levels in one classroom at each of those grade levels. You were randomly selected from your grade level to participate. The study is being conducted by Edvantia, a nonprofit research and development organization, during the 2005–2006 school year. This important study will compare student math achievement in schools using the Saxon Mathematics Program as compared to schools using other mathematics programs. Your participation in this study will involve the following: • • •

Completing a 20–30 minute survey. Administering a standardized math proficiency test in the fall and spring (80 minutes administration time, kindergarten fall testing time is 30 minutes) to your students. Sharing with Edvantia demographic information on your students. This information will include the following: age, gender, free or reduced-price lunch status, special education status, English language learner status and racial identification.

Upon completion and return of the math pretest data, you will receive 50% of the honorarium ($100). Upon completion and return of all other data-collection requirements (math posttest data, teacher interviews, teacher surveys, sample of student assessment data from the Saxon Mathematics Program), you will receive the other 50% of the honorarium ($100). The total amount to be paid to you is $200 for fulfilling all requirements of the study as described in this consent form. By signing and returning this form, you are giving permission to participate in the above-mentioned datacollection activities. By signing this form, you are also giving us permission to use the information you provide in a report we write about the results of our study. Your responses will be kept anonymous and confidential. That is, your responses to the survey will never be associated with you, and data will only be reported in the aggregate. Only authorized Edvantia staff will access the data strictly for the purposes of organizing and analyzing the results. Data will be stored on a password-protected computer file for 3 years past the termination of this project. Finally, your signature on this form indicates that you


10

Appendix D

understand that you are free to withdraw participation in the study at any time by notifying myself, Dr. Kimberly Good, and that there will be no negative consequences from Edvantia as a result of this withdrawal. Data collected for research purposes are stored in compliance with ISO 17799 requirements for access, security, and redundancy. Data are stored in an encrypted format in a centralized, electronically and physically secure server at Edvantia for a period not to exceed 5 years. All electronic data of a personal nature are safeguarded and available only to those project leaders, staff, and technologists having a need to know within the specific criteria as set forth in the approved project plan. There are no known risks associated with participating in these research activities that are greater than those ordinarily encountered in daily life. Should you feel too anxious, however, please notify me immediately. Information obtained during this study will be held in strictest confidence, with the exception that if a researcher obtains clear evidence of unlawful behavior that could result in physical or mental damage to a minor, the researcher is required by statute to report such evidence to the authorities. Therefore, please notify me, Kimberly Good, at 1.800.624.9120 ext. 5449 of questions or concerns that you may have prior to signing the consent form. If, during the course of the research activities, a question or concern arises, please do not hesitate to ask. Please complete and sign this form. Then return the consent form to the person designated as the site coordinator. If you have any questions or concern, please feel free to contact me, Dr. Kimberly Good, at 1.800.624.9120 ext. 5449, or by e-mail at [email protected]. For information on your rights, contact Dr. Merrill Meehan, Edvantia IRB Chair, 1031 Quarrier Street, 6th Floor, or at 1.800.624.9120 ext. 5432. Thank you!

Teacher’s Name: Grade Level: School Name: Teacher’s Signature: Date: Edvantia researcher signature: _______________________________ Date: _________________


11

Appendix D

Parent/Guardian Consent Form (Experimental Sites)


PARENT/GUARDIAN CONSENT FORM National Study on the Saxon Mathematics Program October 2005 Dear Parent: Edvantia is a nonprofit research and development organization which is conducting a national study on the effectiveness of the Saxon Mathematics Program. Your child’s school is one of nearly 40 schools across the country that has been selected to participate in this study. This study is being conducted at the kindergarten through third-grade levels, and your child’s classroom has been selected to participate. In order to measure the impact of the Saxon Mathematics Program on student achievement, we are collecting Saxon Mathematics Program student-assessment data throughout the school year (i.e., math tests that are administered by your child’s teacher) and administering a standardized math proficiency test The standardized math proficiency test will be administered in the fall and again in the spring so that we can assess any gains in student achievement that occurred over the school year. Students whose parents give their permission for participation will be administered the test by their teacher one time in the fall and again in the spring. The test will last approximately 80 minutes (kindergarten fall testing time is 30 minutes). For data-analysis purposes, we are also requesting to collect the following information about your child: age, gender, free or reduced-price lunch status, special education status, English language learner status, and racial identification. Please sign and indicate below whether you would (check the box marked “yes”) or would not (check the box marked “no”) like your child to participate in this study. By checking the box marked “yes,” you are giving permission for your child to take part in both the pre- and posttest, for us to collect a sample of Saxon Mathematics Program student-assessment data, and to collect the background information on your child detailed in the paragraph above. By checking the box marked yes, you are also giving us permission to use the information your child provides in a report we write about the results of our study. Your child’s answers will be kept anonymous and confidential. That is, your child’s answers on the pre- and posttests, the student assessment data and the student demographic data will never be associated with your child’s name, and data will only be reported in the aggregate. Only authorized Edvantia staff and Harcourt Assessment staff will access the data strictly for the purposes of organizing and analyzing the results. Data will be stored on a password-protected computer file for 3 years past the termination of this project. Finally, your signature on this form indicates that you understand that you are free to withdraw permission for your child’s participation in the study at any time by notifying the teacher, and that there will be no negative consequences from Edvantia as a result of this withdrawal. Saxon Elementary Math Program Effectiveness Study

12

Appendix D

Data collected for research purposes are stored in compliance with ISO 17799 requirements for access, security, and redundancy. Data are stored in an encrypted format in a centralized, electronically and physically secure server at Edvantia for a period not to exceed 5 years. All electronic data of a personal nature are safeguarded and available only to those project leaders, staff, and technologists having a need to know within the specific criteria as set forth in the approved project plan. There are no known risks associated with participating in these research activities that are greater than those ordinarily encountered in daily life. Should your child feel too anxious, however, please have him or her notify their teacher immediately. Information obtained during this study will be held in strictest confidence, with the exception that if a researcher obtains clear evidence of unlawful behavior that could result in physical or mental damage to a minor, the researcher is required by statute to report such evidence to the authorities. There are no expected direct benefits for participating respondents. Nor will there be any compensation given by Edvantia to the students to participate in these research activities. You have the right to have any questions or concerned addressed before consenting to participate in any research. Therefore, please notify me, Kimberly Good, at 1.800.624.9120 ext. 5449 of questions or concerns that you may have prior to signing the consent form. If, during the course of the research activities, a question or concern arises, please do not hesitate to ask. Please complete and sign this form. Check either the box marked “yes” (my child has permission to participate in the study) or the box marked “no” (I don’t want my child to participate in the study). After you have signed the form, please read the attached Child Assent Form to your child to make sure he or she is comfortable participating in the study. If your child would like to participate in the study, check the box marked “yes” (my child would like to participate in the study). If your child would not like to participate in the study, check the box marked “no” (my child does not want to participate in the study). Then ask your child to return the consent form to his or her teacher. If you have any questions or concern, please feel free to contact me, Dr. Kimberly Good, at 1.800.624.9120 ext. 5449, or by e-mail at [email protected]. For information on your rights, contact Dr. Merrill Meehan, Edvantia IRB Chair, 1031 Quarrier Street, 6th Floor, or at 1.800.624.9120 ext. 5432. Thank you! Child’s Name: Child’s Grade: School Name: Parent or Guardian Name: Address: Parent or Guardian Signature: Date:

Yes, I give my permission for my child to participate in the Saxon Mathematics Program Study. No, I do not want my child to participate in the Saxon Mathematics Program Study.

Yes, my child would like to participate in the Saxon Mathematics Program Study. No, my child does not want to participate in the Saxon Mathematics Program Study.

Edvantia researcher signature: _______________________________ Date: _________________ Saxon Elementary Math Program Effectiveness Study

13

Appendix D

Parent/Guardian Consent Form (Comparison Site)


PARENT/GUARDIAN CONSENT FORM National Study on the Saxon Mathematics Program October 2005 Dear Parent: Edvantia is a nonprofit research and development organization which is conducting a national study on the Saxon Mathematics Program. Your child’s school is one of nearly 30 schools across the country that has been selected to participate in this study as a comparison site (a nonuser of the Saxon Mathematics Program). In order to measure the impact of the Saxon Mathematics Program on student achievement for schools that are using the Saxon Mathematics Program and schools using other mathematics programs, we are administering a standardized math proficiency test to kindergarten through third-grade students in selected classrooms at your child’s school. Your child’s classroom has been selected to participate in the study. The standardized math proficiency test will be administered in the fall and again in the spring so that we can assess any gains in student achievement that occurred over the school year. Students whose parents give their permission for participation will be administered the test by their teacher one time in the fall and again in the spring. The test will last approximately 80 minutes (kindergarten fall testing time is 30 minutes). For data-analysis purposes, we are also requesting to collect the following information about your child: age, gender, free or reduced-price lunch status, special education status, English language learner status, and racial identification. Please sign and indicate below whether you would (check the box marked “yes”) or would not (check the box marked “no”) like your child to be considered as one of the eight students to participate in this study. By checking the box marked yes, you are giving permission for your child to take part in both the standardized math proficiency test and to collect the background information on your child detailed in the paragraph above. By checking the box marked yes, you are also giving us permission to use the information your child provides in a report we write about the results of our study. Your child’s answers and the student demographic data will be kept anonymous and confidential. That is, your child’s answers on the pre- and posttests will never be associated with your child’s name, and data will only be reported in the aggregate. Only authorized Edvantia staff and Harcourt Assessment staff will access the data strictly for the purposes of organizing and analyzing the results. Data will be stored on a password-protected computer file for 3 years past the termination of this project. Finally, your signature on this form indicates that you understand that you are free to withdraw permission for your child’s participation in the study at any time by notifying the teacher, and that there will be no negative consequences from Edvantia as a result of this withdrawal. Saxon Elementary Math Program Effectiveness Study

14

Appendix D

Data collected for research purposes are stored in compliance with ISO 17799 requirements for access, security, and redundancy. Data are stored in an encrypted format in a centralized, electronically and physically secure server at Edvantia for a period not to exceed 5 years. All electronic data of a personal nature are safeguarded and available only to those project leaders, staff, and technologists having a need to know within the specific criteria as set forth in the approved project plan. There are no known risks associated with participating in these research activities that are greater than those ordinarily encountered in daily life. Should your child feel too anxious, however, please have him or her notify their teacher immediately. Information obtained during this study will be held in strictest confidence, with the exception that if a researcher obtains clear evidence of unlawful behavior that could result in physical or mental damage to a minor, the researcher is required by statute to report such evidence to the authorities. There are no expected direct benefits for participating respondents. Nor will there be any compensation given by Edvantia to the students to participate in these research activities. You have the right to have any questions or concerned addressed before consenting to participate in any research. Therefore, please notify me, Kimberly Good, at 1.800.624.9120 ext. 5449 of questions or concerns that you may have prior to signing the consent form. If, during the course of the research activities, a question or concern arises, please do not hesitate to ask. Please complete and sign this form. Check either the box marked “yes” (my child has permission to participate in the study) or the box marked “no” (I don’t want my child to participate in the study). After you have signed the form, please read the attached Child Assent Form to your child to make sure he or she is comfortable participating in the study. If your child would like to participate in the study, check the box marked “yes” (my child would like to participate in the study). If your child would not like to participate in the study, check the box marked “no” (my child does not want to participate in the study). Then ask your child to return the consent form to his or her teacher. If you have any questions or concern, please feel free to contact me, Dr. Kimberly Good, at 1.800.624.9120 ext. 5449, or by e-mail at [email protected]. For information on your rights, contact Dr. Merrill Meehan, Edvantia IRB Chair, 1031 Quarrier Street, 6th Floor, or at 1.800.624.9120 ext. 5432. Thank you! Child’s Name: Child’s Grade: School Name: Parent or Guardian Name: Address: Parent or Guardian Signature: Date:

Yes, I give my permission for my child to participate in the Saxon Mathematics Program Study. No, I do not want my child to participate in the Saxon Mathematics Program Study.

Yes, my child would like to participate in the Saxon Mathematics Program Study. No, my child does not want to participate in the Saxon Mathematics Program Study.

Edvantia researcher signature: _______________________________ Date: _________________ Saxon Elementary Math Program Effectiveness Study

15

Appendix D

Student Assent Script

Saxon Mathematics Program Study Child Assent Script Your school is participating in a national study to research mathematics programs. We are asking students just like you to complete a math test. The test will take about 80 minutes to complete (30 minutes for kindergarten students). Your teacher will be administering the test to you. We are also asking your teachers to provide us with some information about you such as your name, birth date, gender, and similar information. This information will help us with the study. The test is similar to other tests that you take in school. This test will not be graded, but we would like you to try your best on it. You should not feel more scared or nervous than you might with other tests. If you do, please let your teacher know immediately. Also, if you start the test but do not feel comfortable finishing, you can let your teacher know. It is okay to let a teacher know if you become uncomfortable. You will not get in trouble if you need to stop and nobody else will know. Researchers at Edvantia will study the results from your and other students’ tests—not your teacher or your parents or guardians. They will keep track of the answers of all of the kids from your school on my computer with special passwords. Nobody from your school will see them. Also, Edvantia researchers will not be keeping your name on a computer. After 5 years, they will make sure that their computer files are erased. When they discuss findings from this study, they will not give out your, or anyone else’s, name. We, your parents or guardians, have signed a form giving your teachers permission to give you the test and to collect information on you (e.g., name, birth date, gender, etc.). However, it is up to you to decide whether you really will participate. You do not have to participate in this test. Taking this test will not immediately improve your school experience. But if you decide to participate, it could help people find out how to make students’ learning experiences better. A special group of education professionals has looked at this study and has made sure that this study was safe for you to participate in. They want to make sure that you are safe and comfortable and that your test answers stay private. Do you have any questions that will help you decide whether to participate in this study? Do you have any questions about how safe this study is? If you do have any questions, please ask them of Edvantia researcher, Dr. Kimberly Good (1.800.624.9120, ext. 5449). For information on your rights, contact Dr. Merrill Meehan, Edvantia IRB Chair, 1031 Quarrier Street, 6th Floor, or at 1.800.624.9120 ext. 5432. If you are willing to participate in the study, we, your parents or guardians, will check the box marked “yes,” my child would like to participate in the Saxon Mathematics Program Study. If you do not want to participate in the study, we, your parents or guardians, will check the box marked, “no, my child does not want to participate in the study.” Thank you! Saxon Elementary Math Program Effectiveness Study

16

Appendix D

Site Coordinator Pretest Administration Instructions

_______________________________________________________ Edvantia, Inc. P.O. Box 1348 Charleston, WV 25325-1348 p: 304.347.0400 or 800.624.9120 f: 304.347.0487 www.edvantia.org

To:

Saxon Math Study Site Coordinators

From: Kimberly Good, PhD, Research Associate Date: October 26, 2005 Re:

Site Coordinator Stanford Achievement Test, Ninth Edition Pretest and Student Demographic Form Information

This mailing contains the following testing materials as well as the Student Demographic Information Forms: 1. Teacher Pretest Information Memo Distribute one Teacher Pretest Guidelines from Edvantia to each participating teacher. 2. Participating Students Listing The participating student listing identifies the students that have been randomly selected to participate in the study at each grade level. There is a maximum of eight students per grade level. These students were selected from students’ whose parents signed and returned the consent forms indicating permission for their child to participate in the study. Distribute one Participating Student Listing to each teacher. 3. Directions for Administering Booklets Distribute the appropriate grade level Directions for Administering booklet to the participating teachers. They are labeled by grade level, and there is one booklet for each teacher. 4. Test Booklets Distribute the appropriate test booklets to the participating teachers. The test booklets are labeled by grade level (Kdg = SESAT 1, first grade = SESAT 2, second grade = PRIMARY 1/Abbr, third grade = PRIMARY 2/Abbr). Each teacher should receive 8 test booklets corresponding to the grade level they teach. Note: Teachers will only be administering the math test. 5. Rulers Distribute the rulers to the participating teachers. Each second- and third-grade teacher should receive eight rulers.


17

Appendix D

6. Test Security Affidavit Distribute a Test Security Affidavit form to each teacher. Each teacher must sign and return the Test Security Affidavit to you. The site coordinator must also read and sign the Test Security Affidavit. The signed affidavits must be returned to Edvantia. 7. Student Demographic Information Form Distribute one Student Demographic Information Form to each participating teacher. Teachers should complete the requested information requested for the eight students in their classroom selected to participate in the study. The teacher may have to contact the school office to obtain some of the information. In the event some information is not able to be shared due to school or district policy, please indicate “NA” (not available) in the appropriate column(s). The Student Demographic Information Forms must be included with the test booklets and related documents. General Information The testing window for administering the SAT 9 mathematics test in the participating K–3 classrooms is October 31–November 22. However, we encourage the teachers to administer the test as early in that timeframe as possible in order to have the maximum time between the pre- and posttest. Students will mark all responses in their test booklets. The site coordinator should collect all test booklets from the teachers (both completed and extra test booklets, if applicable) and Directions for Administering booklet from each teacher. The test booklets should be returned to Edvantia immediately upon conclusion of testing. Please rubber band each classroom set of completed test booklets together. If applicable, separately rubber band each set of unused grade-level test booklets. It is critical that all test booklets and the Directions for Administering booklets be returned to Edvantia. Include the signed affidavits, Directions for Administering booklets and the completed Student Demographic Information forms to Edvantia along with the test booklets. Please return all test booklets and related documents and the completed Student Demographic Information Forms by November 30. After you have assembled all data and information mentioned above, please return them to Edvantia using the UPS envelope and label provided. Enter your name and school’s address in the return envelope. Call the UPS number on the label provided to schedule a pick-up at your school. All shipping charges are being handled by Edvantia. The delivery to Edvantia will be overnight, therefore you need to have the test booklets and related documents and the completed Student Demographic Forms collected and sent by November 29. We prefer that all materials be submitted at one time. All materials must be returned to Edvantia by November 30. Upon receipt of all materials, Harcourt Achieve will process the first one half of stipend checks for the site coordinators and the participating teachers. The checks will be mailed to your home address. If you have any questions about the contents of this memo or the study, please feel free to contact me at 1.800.624.9120 (ext. 5449) or [email protected].


18

Appendix D

Teacher Pretest Administration Instructions


To:

Saxon Math Study Participating Teachers

From: Kimberly Good, PhD, Research Associate Date: October 26, 2005 Re:

Teacher Stanford Achievement Test, Ninth Edition Pretest and Student Demographic Form Information

You should have received the following materials for administering the Stanford Achievement Test, Ninth Edition (SAT 9). Although you have the full battery of SAT 9 tests, you will only be administering the math subtest. This packet also contains the Student Demographic Information Form. 1. Participating Students Listing The participating student listing identifies the students that were randomly selected to participate in the study at each grade level. There is a maximum of eight students per grade level. These students were selected from students’ whose parents signed and returned the consent forms indicating permission for their child to participate in the study. 2. Directions for Administering Booklet Please review the information contained in the Directions for Administering booklet. It is particularly important to familiarize yourself with the section titled “General Directions for Administering.” 3. Test Booklets Only the eight students randomly selected to participate in the study will be tested. You will only be administering the math test. The test booklets are labeled by grade level (Kdg = SESAT 1, first grade = SESAT 2, second grade = PRIMARY 1/Abbr, third grade = PRIMARY 2/Abbr). The total testing time for the kindergarten math pretest is 30 minutes. The total testing time for the grades 1–3 pretest is 80 minutes. The testing time does not account for the preparation time. For grades 1–3 there are two math subtests: Problem Solving and Procedures. It is advised that you administer the Problem Solving subtest one day and the Procedures subtest a second day. The testing window for administering the SAT 9 mathematics test in your classroom is October 31– November 22. However, we encourage you to administer the test as early in that timeframe as possible in order to have the maximum time between the pre- and posttest. Students will mark their responses to the math subtest in the test booklet. Students will need a number 2 pencil to complete the test. Saxon Elementary Math Program Effectiveness Study

19

Appendix D

You will also need to complete the information on the back side of the test booklet for each of the eight students (student name, school, gender, grade, teacher, district and date of birth). You do not need to complete the student number or other information boxes. 4. Rulers You will receive a ruler for the 8 students being tested (applicable only for second and third grades). Specific instructions on when and how to use these are found in the Directions for Administering booklets. 5. Test Security Affidavit Read and sign the Test Security Affidavit. 6. Student Demographic Information Form Complete the requested information requested for the eight students in your classroom whom were randomly selected to participate in the study. You may need to contact the school office to obtain some of the information. In the event some information is not able to be shared due to school or district policy, please indicate “NA” (not available) in the appropriate column(s). The Student Demographic Information Form must be included with the test booklets and related documents. General Information Return (1) the completed test booklets (extra test booklets, if applicable), (2) signed Test Security Affidavit, (3) Directions for Administering booklet, and (4) completed Student Demographic Information Form to your site coordinator immediately after testing. It is critical that all test booklets (both completed and unused, if applicable) and the Directions for Administering booklet be returned to your site coordinator. Return these items to your site coordinator on or before November 29 (or the date specified by your site coordinator). After Edvantia has received all testing materials and Student Demographic Information forms from your school (see paragraph one in this section), Harcourt Achieve will process the first one half of stipend checks for the site coordinators and the participating teachers. The checks will be mailed to your home address. If you have any questions about the contents of this memo or the study, please feel free to contact me at 1.800.624.9120 (ext. 5449) or [email protected].


20

Appendix D

STANFORD ACHIEVEMENT TEST SERIES: NINTH EDITION TEST SECURITY AFFIDAVIT I acknowledge that I will have access to Form S of the Stanford Achievement Test, Ninth Edition (the “Test”) for the purpose of administering the Test solely for a research study to evaluate the relationship between student achievement and the Saxon skill-based math program. I understand that the Test materials are highly secure, and it is my professional responsibility to protect their security as follows: 1)

I will not divulge the contents of the Test, generally or specifically, to anyone.

2)

I will not copy any part of the Test materials whatsoever.

3)

I will limit access to the Test materials by examinees only to actual testing periods.

4)

I will return all Test booklets and Directions for Administering booklets to Edvantia.

5)

I will destroy all other Test materials upon conclusion of testing (if applicable).

________________________________________ Signed _______________________________________ Print Name _______________________________________ Position _______________________________________

Agency/School _______________________________________ Address _______________________________________ City/State _______________________________________ Date Saxon Elementary Math Program Effectiveness Study

21

Appendix D

Student Demographic Data-Collection Sheet

____________________________________________________________ Edvantia, Inc. P.O. Box 1348 Charleston, WV 25325-1348 p: 304.347.0400 or 800.624.9120 f: 304.347.0487 www.edvantia.org

National Study on the Saxon Mathematics Program

Please complete the information requested in the table for each student in your classroom. • • • • • • •

Please print the last name and first name of the student. Indicate the birth date of the student (mm/dd/yy) (e.g., 12/22/98) Indicate the gender of the student; enter “M” for male or “F” for female. Indicate whether the student is eligible for free or reduced-price lunch; enter “Y” for eligible or “N” for not eligible. Indicate whether the student is identified as special education; enter “Y” for special education or “N” for not special education. Indicate whether the student is an English language learner (ELL); enter “Y” for ELL student or “N” for not ELL student. Indicate the race of the student. Use one of the following codes: B = Black or African American, Non-Hispanic H = Hispanic or Latino W = White, Non-Hispanic N = American Indian or Alaskan Native A = Asian P = Native Hawaiian or Other Pacific Islander M = Multiracial O = Other (please specify)

This information will be kept confidential and used exclusively by Edvantia staff as a part of data analysis for the study. Upon completion of the information requested, please return this form to your site coordinator, and he/she will send it to the proper representative at Edvantia.

(over) Saxon Elementary Math Program Effectiveness Study

22

Appendix D

School: Teacher Name: Grade Level:


23

Racial Identification

English Language Learner (ELL) Status

Birth Date

Special Ed Status

Student First Name

Free/Reduced Lunch Status

Student Last Name

Gender

Student Demographics

Appendix D

Instructions for Videotaped Classroom Lessons

____________________________________________________________ _ Edvantia, Inc. P.O. Box 1348 Charleston, WV 25325-1348 p: 304.347.0400 or 800.624.9120 f: 304.347.0487 www.edvantia.org To:

Saxon Math Study Site Coordinators and Teachers

From: Kimberly Good, PhD, Research and Evaluation Specialist I Date: [Insert] Re:

Instructions for Videotaped Saxon Math Lesson

To assess the fidelity of implementation of the Saxon Math program, we are requesting that each participating teacher be videotaped teaching one Saxon Math lesson. The dates for which you have been asked to provide a videotaped classroom lesson for the National Study on the Saxon Math program are [insert 3 week videotaping dates]. This memo provides specific instructions for conducting the videotaped classroom lesson of each participating teacher. Site coordinators— please distribute a copy of this memo, a videotape, and the Videotaped Classroom Observation Information Form to each participating teacher. Please ensure each participating teacher videotape contains one complete Saxon Math lesson. The videotape should include the entire span of time that each teacher devotes to teaching math over the course of the day of taping, including both The Meeting and the Lesson, which may also include Guided Practice and Fact Practice. If necessary, you may have to schedule multiple videotaping sessions if the teacher conducts The Meeting at one point of the school day and the Lesson at another time. Teachers should teach the lesson as they normally would. The purpose of the videotaped observation is not to assess the extent to which the teacher follows the script or to make judgments on the performance of the teacher relative to the Saxon Math program. Instead, the purpose is to examine the ways the Saxon Math program is implemented across schools and classrooms participating in the study. Therefore, if teachers use other math materials in addition to the Saxon Math program during math class time, please include this as a part of the videotaped observation. The focus of the videotape should primarily be on the teacher teaching and not the students. However, as The Meeting and Lesson are being conducted, please pan the room with the camera occasionally so that student reaction and interaction is captured. Also, take a video sweep of math resources being used (e.g., math posters and related items hanging on the wall, use of The Meeting materials). Each participating teacher should be videotaped on a separate VHS tape. Each VHS tape needs to be labeled with the school name, teacher name, and grade level. Each teacher must complete one Videotaped Classroom Observation Information Form.


24

Appendix D

Upon completion of all videotaping, return the VHS tapes and Classroom Videotaped Observation Information Forms to Edvantia via UPS in the same box. Complete the information on the UPS shipping label that has been enclosed with this mailing. Call UPS to schedule a pick-up at your school. The VHS tapes should be returned to Edvantia by [Insert date]. In April, Edvantia staff will contact the site coordinators to schedule telephone interviews with each participating teacher. These interviews are to collect additional information on the implementation of the Saxon Math program at the classroom level.


25

Appendix D

____________________________________________________________ Edvantia, Inc. P.O. Box 1348 Charleston, WV 25325-1348 p: 304.347.0400 or 800.624.9120 f: 304.347.0487 www.edvantia.org

National Study on the Saxon Mathematics Program Videotaped Classroom Observation Information Form Please complete and return this form to your site coordinator, and he/she will send it to the proper representative at Edvantia. School Name:

________________________________________________

School Address:

________________________________________________

Teacher Name:

________________________________________________

Grade Level:

________________________________________________

Number of Students in Classroom: _______________________________________ Date:

_______________________________________________

Lesson number and topic:

__________________________________________

Videotaping time of day: __________________________________________ (e.g., 8:45–9:15 a.m., 2:10–2:35 p.m.) Total length of time devoted to this Math Meeting (in minutes): _______________ Your judgment on how long it should take for this Math Meeting (in minutes): _________________________________ Total length of time devoted to teaching this Lesson (in minutes): ______________ Your judgment on how long it should take to teach this Lesson (in minutes): _________________ Number of years (including the 2005–2006 school year) you have been teaching the Saxon Math program: __________________________________________


26

Appendix D

Kindergarten Innovation Configuration Matrix (ICM) Edvantia Reviewer Date Directions: Refer to the print version of the Saxon Math Meeting and lesson you are observing. Completion of the ICM requires familiarity with the skills that are intended to be included in The Meeting, the manipulatives to be used in both The Meeting and the Lesson, as well as a general understanding of the script to be followed in teaching The Meeting and Lesson. For each component of the Saxon Math Meeting and Lesson shown in the matrices below indicate the descriptor best illustration your analysis of the videotaped classroom observation. For each component, indicate the variation in teaching that you observe (a, b, c, d) in the column titled “Variation.” If a particular component was not observed, indicate “e” in the Variation column. Teacher Name School Name Lesson Number A.

The Meeting Indicate the length of time (in minutes) devoted to The Meeting. Check if The Meeting was not included as a part of the videotaped classroom observation.

If The Meeting was a part of the videotaped classroom observation, complete the following section. If The Meeting was not a part of the videotaped classroom observation, skip to the Lesson.

©2006 by Edvantia, Inc. All rights reserved Saxon Elementary Math Program Effectiveness Study

27

Appendix D

(a)

(b)

(c)

(d)

The Meeting focus is on reviewing previous lesson concepts. The Meeting focus is on only one to two skills to be taught in The Meeting, or there is no practice of meeting skills. Manipulatives and resources are not used as a part of The Meeting.

1

The Meeting focus is entirely on practice of skills intended to be taught in The Meeting.

The Meeting focus is on the majority of skills to be taught in The Meeting.

The Meeting focus is on one half of the skills to be taught in The Meeting.

2


Manipulatives and resources are used as a part of The Meeting, although they are not appropriate for the skills being taught.

3

Teacher frequently asks questions of students.

4

Teacher frequently uses appropriate amount of wait time for students to respond to questions.

5

The Meeting is conducted in an orderly manner. Only one student speaks at a time unless the teacher asks for a choral response. All students are focused and on task.

The appropriate manipulatives and resources are used as a part of The Meeting, although they may not be the specific ones recommended by Saxon Math. Teacher asks a moderate number of questions of students. Teacher sometimes uses appropriate amount of wait time for students to respond to questions. For the most part, The Meeting is conducted in an orderly manner. Occasionally, students respond to questions without being called upon. Occasionally, multiple students speak

Teacher occasionally asks questions of students. Teacher infrequently uses appropriate amount of wait time for students to respond to questions. For the most part, the meeting is conducted in a disorderly manner. Often, students respond to questions without being called upon. Often, multiple students speak at the same time.

Not Variation observed (e)

Teacher asks few questions of the students. Teacher does not use wait time for students to respond to questions.

The meeting is conducted in a disorderly manner. All or nearly all of the time students speak without being called upon by the teacher. All or nearly all of the time ©2006 by Edvantia, Inc. All rights reserved


28

Appendix D

(a)

(b)

at the same time. Most students are on task.

(c)

(d)

Many students are not paying attention and are off task.

multiple students speak at the same time.



29

Appendix D

B.

Lesson Indicate the length of time (in minutes) devoted to the Lesson. Check if the Lesson was not included as a part of the videotaped classroom observation.

If the Lesson was a part of the videotaped classroom observation, complete the following section. If the Lesson was not a part of the videotaped classroom observation, skip to Lesson Practice.

6

7

8

(a)

(b)

(c)

(d)

At the beginning of the Lesson, the teacher clearly states the objective of the lesson to the students (e.g., states what students will be learning in the Lesson). Teacher frequently models the concepts being taught in the lesson (e.g., writes numbers on the board or shows students how to do something). Saxon Math manipulatives are used as a part of the Lesson.

At the beginning of the Lesson, the teacher makes an indirect reference to a Lesson objective but does not specifically describe the objective.

Somewhere in the Lesson (other than the beginning) the teacher directly or indirectly refers to the Lesson objective.

Teacher does not make any reference to an objective. Students are not told what they will be learning in the Lesson.

Teacher sometimes models the concepts being taught in the Lesson.

Teacher infrequently models the concepts being taught in the Lesson.

Teacher does not do any modeling of the concepts being taught. All teaching is done verbally.

The appropriate manipulatives are used as a part of the Lesson, although they may not be the specific ones recommended by Saxon Math.

Manipulatives are used as a part of the Lesson, although they are not appropriate for the concepts being taught.

Manipulatives are not used as a part of the Lesson.



30

Appendix D

9

Teacher models the use of manipulatives prior to having students use the manipulatives. The teacher clearly explains to students how they are to perform the task with the manipulatives.

The teacher verbally explains how to use the manipulatives prior to having students use the manipulatives.

The teacher does not demonstrate the use of manipulatives prior to having students use the manipulatives.

10


Teacher asks a moderate number of questions of students.

Teacher occasionally asks questions of students.

11

Teacher frequently uses appropriate amount of wait time for students to respond to questions. Teacher allots 2–3 minutes at the end of the Lesson for students to share their observations about what they learned in the Lesson.

Teacher sometimes uses appropriate amount of wait time for students to respond to questions. Teacher allots 2–3 minutes at the end of the Lesson for students to share; however, sharing not necessarily related to observations about what was learned in the Lesson.

The Lesson is conducted in an orderly manner. Only one student speaks at a time unless the teacher asks for a choral response. All students are focused and on

For the most part, the Lesson is conducted in an orderly manner. Occasionally students respond to questions without being called upon. Occasionally multiple students speak

Teacher infrequently uses appropriate amount of wait time for students to respond to questions. Teacher allots less than 2–3 minutes at the end of the Lesson for students to share. Sharing may or may not be related to their observations about what they learned in the Lesson. For the most part, the Lesson is conducted in a disorderly manner. Often students respond to questions without being called upon. Often multiple students speak at the same time.

12

13

The teacher does not demonstrate the use of manipulatives prior to having students use the manipulatives. The teacher provides confusing directions to students in how they are to perform the task with the manipulatives. Teacher asks few questions of the students. Teacher does not use wait time for students to respond to questions. Teacher does not allow any time at the end of the lesson for students to share their observations about what they learned in the Lesson.

The Lesson is conducted in a disorderly manner. All or nearly all of the time students speak without being called upon by the teacher. All or nearly all of the time ©2006 by Edvantia, Inc. All rights reserved


31

Appendix D

task.





32

Appendix D

C.

Lesson Practice Indicate the length of time (in minutes) devoted to the Lesson Practice sheet. Check if the Lesson Practice sheet was not included as a part of the videotaped classroom observation.

Check if the Lesson Practice sheet was to be a part of the videotaped classroom observation. Refer to the print version of the Saxon Math lesson you are observing to make this determination. If Lesson Practice was a part of the videotaped classroom observation, complete the following section. If Lesson Practice was not a part of the videotaped classroom observation, skip to Handwriting Practice. (a)

(b)

(c)

(d)

Teacher does not have students complete Lesson Practice sheet during class (e.g., either not completed at all or all of Lesson Practice sheet given as homework). Teacher does not have students complete Lesson Practice sheet during class (e.g., either not completed at all or all of Lesson Practice sheet given as homework).

14

Teacher reads the directions to the students. Students complete Lesson Practice sheet independently.

Teacher and students complete most of the Lesson Practice sheet as a class activity (e.g., jointly solving problems).

Teacher and students complete all of the Lesson Practice sheet as a class activity (e.g., jointly solving problems).

15

Teacher circulates and checks students’ papers continuously as they work on Lesson Practice sheet.

Teacher circulates and checks students’ papers for the majority of the time they work on Lesson Practice sheet.

Teacher sits at desk or stands in front of room and responds to questions students may have about the Lesson Practice sheet.



33

Appendix D

D.

Handwriting Practice Indicate the length of time (in minutes) devoted to Handwriting Practice. Check if Handwriting Practice was not included as a part of the videotaped classroom observation.

Check if Handwriting Practice was to be a part of the videotaped classroom observation. Refer to the print version of the Saxon Math lesson you are observing to make this determination. If Handwriting Practice was a part of the videotaped classroom observation, complete the following section. If Handwriting Practice was not a part of the videotaped classroom observation, skip to Counting Practice. (a)

(b)

(c)

16

Teacher reads the directions to the students and gives class guidance on initial completion of Handwriting Practice sheet.

Teacher and students complete most of the Handwriting Practice sheet as a class activity.

Teacher and students complete all of the Handwriting Practice sheet as a class activity.

17

Teacher circulates and checks students’ papers continuously as they work on Handwriting Practice sheet.

Teacher circulates and checks students’ papers for the majority of the time they work on Handwriting Practice sheet.

Teacher sits at desk or stands in front of room and responds to questions students may have about the Handwriting Practice sheet.

(d)


Teacher does not have students complete Handwriting Practice sheet during class (e.g., either not completed at all or all of Handwriting Practice sheet given as homework). Teacher does not have students complete Handwriting Practice sheet during class (e.g., either not completed at all or all of Handwriting Practice sheet given as homework).


34

Appendix D

E.

Counting Practice Indicate the length of time (in minutes) devoted to Counting Practice. Check if Counting Practice was not included as a part of the videotaped classroom observation.

Check if Counting Practice was to be a part of the videotaped classroom observation. Refer to the print version of the Saxon Math lesson you are observing to make this determination. If Counting Practice was a part of the videotaped classroom observation, complete the following section. If Counting Practice was not a part of the videotaped classroom observation, skip to the Use of Additional Math Materials and Resources section. (a)

(b)

(c)

18

Teacher reads the directions to the students and gives class guidance on initial completion of Counting Practice sheet.

Teacher and students complete most of the Counting Practice sheet as a class activity.

Teacher and students complete all of the Counting Practice sheet as a class activity.

19

Teacher circulates and checks students’ papers continuously as they work on Counting Practice sheet.

Teacher circulates and checks students’ papers for the majority of the time they work on Counting Practice sheet.

Teacher sits at desk or stands in front of room and responds to questions students may have about the Counting Practice sheet.

(d)


Teacher does not have students complete Counting Practice sheet during class (e.g., either not completed at all or all of Counting Practice sheet given as homework). Teacher does not have students complete Counting Practice sheet during class (e.g., either not completed at all or all of Counting Practice sheet given as homework).


35

Appendix D

F.

Classroom Arrangement and Resources

For each of the following lists, indicate the classroom resources which were used or present and the type of classroom arrangement. If you did not have the opportunity to observe the classroom arrangement, check “did not observe.” The Meeting Classroom Arrangement (check one): Students seated on floor in semicircle or circle in front of the board Students seated in chairs in semicircle or circle in front of the board Students seated in desks in rows Students seated in desks in clusters or other arrangement other than rows Other (describe): Did not observe The Lesson Classroom Arrangement (check one): Students seated on floor in semicircle or circle in front of the board Students seated in chairs in semicircle or circle in front of the board Students seated in desks in rows Students seated in desks in clusters or other arrangement other than rows Other (describe): Did not observe Use of Resources and Materials (check all that apply): The Meeting Board Manipulatives (The Meeting) Poster-size Lesson Charts Teacher Handwriting Charts Manipulatives (The lesson) Student Lesson Practice sheet Student Handwriting Practice sheet Student Counting Practice sheet Student Number Cards Student Number Lines Math Center Materials and Activities G. Use of Additional Math Materials and Resources: Make note of any additional math materials and resources that were used throughout the videotaped observation that were not a part of the Saxon Math program. Include a complete description of how and what purpose they were used. ©2006 by Edvantia, Inc. All rights reserved Saxon Elementary Math Program Effectiveness Study

36

Appendix D

First-Grade Innovation Configuration Matrix (ICM) Edvantia Reviewer Date Directions: Refer to the print version of the Saxon Math Meeting and lesson you are observing. Completion of the ICM requires familiarity with the skills that are intended to be included in The Meeting, the manipulatives to be used in both The Meeting and the Lesson, as well as a general understanding of the script to be followed in teaching The Meeting and Lesson. For each component of the Saxon Math Meeting and Lesson shown in the matrices below, indicate the descriptor best illustrating your analysis of the videotaped classroom observation. For each component, indicate the variation in teaching that you observe (a, b, c, d) in the column titled “Variation.” If a particular component was not observed, indicate “e” in the Variation column. Teacher Name School Name Lesson Number

A.


If The Meeting was a part of the videotaped classroom observation, complete the following section. If The Meeting was not a part of the videotaped classroom observation, skip to Number Fact Practice.


37

Appendix D

(a)

(b)

(c)

(d)

The Meeting focus is on reviewing previous lesson concepts. The Meeting focus is on only one to two skills to be taught in The Meeting, or there is no practice of meeting skills. Manipulatives and resources are not used as a part of The Meeting.

1




2



3


The appropriate manipulatives and resources are used as a part of The Meeting, although they may not be the specific ones recommended by Saxon Math. Teacher asks a moderate number of questions of students.


Teacher asks few questions of the students.

4

Teacher frequently uses appropriate amount of wait time for students to respond to questions. The meeting is conducted in an orderly manner. Only one student speaks at a time unless the teacher asks for a choral

Teacher sometimes uses appropriate amount of wait time for students to respond to questions. For the most part, the meeting is conducted in an orderly manner. Occasionally students respond to questions without being called

Teacher infrequently uses appropriate amount of wait time for students to respond to questions. For the most part, the meeting is conducted in a disorderly manner. Often, students respond to questions without being called

Teacher does not use wait time for students to respond to questions.

5


The meeting is conducted in a disorderly manner. All or nearly all of the time students speak without being called upon by ©2006 by Edvantia, Inc. All rights reserved


38

Appendix D

(a)

response. All students are focused and on task.

(b)

(c)

(d)

upon. Occasionally, multiple students speak at the same time. Most students are on task.

upon. Often, multiple students speak at the same time. Many students are not paying attention and are off task.

the teacher. All or nearly all of the time multiple students speak at the same time.



39

Appendix D

B.

Number Fact Practice Indicate the length of time (in minutes) devoted to Number Fact Practice. Check if Number Fact Practice was not included as a part of the videotaped classroom observation.

Check if Number Fact Practice was to be a part of the videotaped classroom observation. Refer to the print version of the Saxon Math lesson you are observing to make this determination. If Number Fact Practice was a part of the videotaped classroom observation, complete the following section. If Number Fact Practice was not a part of the videotaped classroom observation, skip to the Lesson. (a)

(b)

(c)

(d)

6


7

Saxon Math manipulatives are used as a part of Fact Practice (e.g., Fact Cards, Wrap-Ups).

Teacher occasionally asks questions of students. Manipulatives are used as a part of Fact Practice, although they are not appropriate for the concepts being taught.

Teacher asks few questions of the students. Manipulatives are not used as a part of Fact Practice.

8

Teacher allows time for students to independently complete Class Fact Practice sheet (Side A).

Teacher asks a moderate number of questions of students. The appropriate manipulatives are used as a part of Fact Practice, although they may not be the specific ones recommended by Saxon Math. Teacher allows time for students to complete the majority of Class Fact Practice sheet (Side A) independently. Part of Side A is completed together as a class.

Teacher and students complete all of the Class Fact Practice sheet Side A as a class activity (e.g., jointly solving problems).

9

Teacher reads all the problems and answers

Teacher reads the majority of the

Teacher reads a few of the problems and

Teacher does not have students complete Class Fact Practice sheet (Side A) during class (e.g., either not completed at all or all of Class Fact Practice sheet, Side A and B, given as homework). Teacher does not have students complete Class



40

Appendix D

(a)

for the Class Fact Practice sheet (Side A) after students complete the sheet.

(b)

problems and answers for the Class Fact Practice sheet (Side A) after students complete the sheet.

(c)

answers for the Class Fact Practice sheet (Side A) after students complete the sheet.

(d)


Fact Practice sheet during class (Side A; e.g., either not completed at all or all of Class Fact Practice sheet given as homework).


41

Appendix D

C.

Lesson Indicate the length of time (in minutes) devoted to the Lesson Check if the Lesson was not included as a part of the videotaped classroom observation.

If the Lesson was a part of the videotaped classroom observation, complete the following section. If the Lesson was not a part of the videotaped classroom observation, skip to Guided Practice.

10

11

12

13

(a)

(b)

(c)

(d)

At the beginning of the lesson, the teacher clearly states the objective of the lesson to the students (e.g., states what students will be learning in the lesson). Teacher frequently models the concepts being taught in the lesson (e.g., writes numbers on the board or shows students how to do something). Saxon Math manipulatives are used as a part of the lesson.

At the beginning of the lesson, the teacher makes an indirect reference to a lesson objective but does not specifically describe the objective.

Somewhere in the lesson (other than the beginning), the teacher directly or indirectly refers to the lesson objective.



Teacher infrequently models the concepts being taught in the Lesson.


The appropriate manipulatives are used as a part of the Lesson, although they may not be the specific ones recommended by Saxon Math. The teacher verbally explains how to use the


Manipulatives are not used as a part of the lesson.

The teacher does not demonstrate the use of


Teacher models the use of manipulatives prior



42

Appendix D

(a)

(b)

(c)

(d)

to having students use the manipulatives. The teacher clearly explains to students how they are to perform the task with the manipulatives.

manipulatives prior to having students use the manipulatives.


14


Teacher asks a moderate number of questions of students.


manipulatives prior to having students use the manipulatives. The teacher provides confusing directions to students in how they are to perform the task with the manipulatives. Teacher asks few questions of the students.

15

Teacher frequently uses appropriate amount of wait time for students to respond to questions. Teacher allots 2–3 minutes at the end of lesson for students to share their observations about what they learned in the Lesson.

Teacher sometimes uses appropriate amount of wait time for students to respond to questions. Teacher allots 2–3 minutes at the end of lesson for students to share; however, sharing not necessarily related to observations about what was learned in the lesson.

The Lesson is conducted in an orderly manner. Only one student speaks at a time unless the teacher asks for a choral response. All students

For the most part, the lesson is conducted in an orderly manner. Occasionally, students respond to questions without being called upon. Occasionally,

Teacher infrequently uses appropriate amount of wait time for students to respond to questions. Teacher allots less than 2–3 minutes at the end of the Lesson for students to share. Sharing may or may not be related to their observations about what they learned in the Lesson. For the most part, the Lesson is conducted in a disorderly manner. Often, students respond to questions without being called upon. Often, multiple

16

17


Teacher does not use wait time for students to respond to questions. Teacher does not allow any time at the end of the lesson for students to share their observations about what they learned in the lesson.

The lesson is conducted in a disorderly manner. All or nearly all of the time students speak without being called upon by the teacher. All or ©2006 by Edvantia, Inc. All rights reserved


43

Appendix D

(a)

are focused and on task.

(b)

(c)

multiple students speak students speak at the at the same time. Most same time. Many students are on task. students are not paying attention and are off task.

(d)


nearly all of the time multiple students speak at the same time.


44

Appendix D

D.

Guided Practice Indicate the length of time (in minutes) devoted to the Guided Practice sheet. Check if the Guided Practice sheet was not included as a part of the videotaped classroom observation.

Check if the Guided Practice sheet was to be a part of the videotaped classroom observation. Refer to the print version of the Saxon Math lesson you are observing to make this determination. If Guided Practice was a part of the videotaped classroom observation, complete the following section. If Guided Practice was not a part of the videotaped classroom observation, skip to the Use of Additional Math Materials and Resources section. (a)

(b)

(c)

(d)

18

Teacher reads the directions separately for completion of each problem on the Guided Class Practice sheet (Side A).

19

Teacher allows time for students to complete each problem on the Guided Class Practice sheet (Side A) and then writes the answer to each problem on a chart.

Teacher has the students read the directions for completion of each problem on the Guided Class Practice sheet (Side A; either out loud to the class or silently to themselves). Teacher allows time for students to complete all problems on the Guided Class Practice (Side A). Teacher writes answers to the problems after all problems are completed.

Teacher does not have students complete the Guided Class Practice sheet (Side A) during class (e.g., either not completed at all or all of Guided Class Practice sheet, Side A and B, given as homework). Teacher does not have students complete the Guided Class Practice sheet (Side A) during class (e.g., either not completed at all or all of Guided Class Practice sheet, Side A and B, given as homework).

20

Teacher circulates and checks students’ papers entire time as they

Teacher does not read directions for any of the problems on the Guided Class Practice sheet (Side A) or has the students read the directions, but instead proceeds directly to solving the problems. Teacher allows time for students to complete all problems on the Guided Class Practice sheet (Side A). Teacher does not write answers to the problems after the problems are completed. Teacher sits at desk or stands in front of room and responds to

Teacher circulates and checks students’ papers for the majority of the



Teacher does not have students complete the Guided Class Practice ©2006 by Edvantia, Inc. All rights reserved 45

Appendix D

(a)

(b)

(c)

(d)

work on each problem of the Guided Class Practice sheet (Side A).

time they work on each problem of the Guided Class Practice sheet (Side A).

questions students may have on the Guided Class Practice sheet (Side A).

sheet during class (Side A; e.g., either not completed at all or all of Guided Class Practice sheet given as homework).



46

Appendix D

E.

Classroom Arrangement and Resources

For each of the following lists indicate the classroom resources which were used or present and the type of classroom arrangement. If you did not have the opportunity to observe the classroom arrangement, check “did not observe.” The Meeting Classroom Arrangement (check one): Students seated on floor in semicircle or circle in front of the board Students seated in chairs in semicircle or circle in front of the board Students seated in desks in rows Students seated in desks in clusters or other arrangement other than rows Other (describe): Did not observe The Lesson classroom Arrangement (check one): Students seated on floor in semicircle or circle in front of the board Students seated in chairs in semicircle or circle in front of the board Students seated in desks in rows Students seated in desks in clusters or other arrangement other than rows Other (describe): Did not observe Use of Resources and Materials (check all that apply): The Meeting Board Manipulatives (The Meeting) Poster-size Lesson Charts Manipulatives (The lesson) Lesson Masters (for students) Student Guided Practice sheet Student Class Practice sheet Student Written Assessments Student Fact Cards Learning Wrap-UpsTM Teacher Fact Cards Student math offices F. Use of Additional Math Materials and Resources: Make note of any additional math materials and resources that were used throughout the videotaped observation that were not a part of the Saxon Math program. Include a complete description of how and what purpose they were used. ©2006 by Edvantia, Inc. All rights reserved Saxon Elementary Math Program Effectiveness Study

47

Appendix D

Second- and Third-Grade Innovation Configuration Matrix (ICM) Edvantia Reviewer Date Directions: Refer to the print version of the Saxon Math Meeting and lesson you are observing. Completion of the ICM requires familiarity with the skills that are intended to be included in The Meeting, the manipulatives to be used in both The Meeting and the Lesson, as well as a general understanding of the script to be followed in teaching The Meeting and Lesson. For each component of the Saxon Math Meeting and Lesson shown in the matrices below indicate the descriptor best illustrating your analysis of the videotaped classroom observation. For each component, indicate the variation in teaching that you observe (a, b, c, d) in the column titled “Variation.” If a particular component was not observed, indicate “e” in the Variation column. Teacher Name School Name Lesson Number

A.


If The Meeting was a part of the videotaped classroom observation, complete the following section. If The Meeting was not a part of the videotaped classroom observation, skip to Number Fact Practice.


48

Appendix D

(a)

(b)

(c)

(d)

The Meeting focus is on reviewing previous lesson concepts. The Meeting focus is on only one to two skills to be taught in The Meeting or there is no practice of meeting skills. Manipulatives and resources are not used as a part of The Meeting.

1




2



3


4


5

The meeting is conducted in an orderly manner. Only one student speaks at a time unless the teacher asks for a choral response. All students are

The appropriate manipulatives and resources are used as a part of The Meeting, although they may not be the specific ones recommended by Saxon Math. Teacher asks a moderate number of questions of students. Teacher sometimes uses appropriate amount of wait time for students to respond to questions. For the most part, the meeting is conducted in an orderly manner. Occasionally students respond to questions without being called upon. Occasionally

Teacher occasionally asks questions of students. Teacher infrequently uses appropriate amount of wait time for students to respond to questions. For the most part, the meeting is conducted in a disorderly manner. Often students respond to questions without being called upon. Often multiple students


Teacher asks few questions of the students. Teacher does not use wait time for students to respond to questions.

The meeting is conducted in a disorderly manner. All or nearly all of the time students speak without being called upon by the teacher. All or ©2006 by Edvantia, Inc. All rights reserved


49

Appendix D

(a)

focused and on task.

(b)

(c)

(d)

multiple students speak at the same time. Most students are on task.

speak at the same time. Many students are not paying attention and are off task.

nearly all of the time multiple students speak at the same time.



50

Appendix D

B.

Number Fact Practice Indicate the length of time (in minutes) devoted to Number Fact Practice. Check if Number Fact Practice was not included as a part of the videotaped classroom observation.

Check if Number Fact Practice was to be a part of the videotaped classroom observation. Refer to the print version of the Saxon Math lesson you are observing to make this determination. If Number Fact Practice was a part of the videotaped classroom observation, complete the following section. If Number Fact Practice was not a part of the videotaped classroom observation, skip to the Lesson. (a)

(b)

(c)

(d)

6


7


Teacher occasionally asks questions of students. Manipulatives are used as a part of Fact Practice, although they are not appropriate for the concepts being taught.

Teacher asks few questions of the students. Manipulatives are not used as a part of Fact Practice.

8

Teacher has students complete Fact Practice sheet (Side A) while timing them for a specified period of time.

Teacher asks a moderate number of questions of students. The appropriate manipulatives are used as a part of Fact Practice, although they may not be the specific ones recommended by Saxon Math. Teacher has students complete Fact Practice sheet (Side A) untimed.

Teacher works with students to complete Fact Practice sheet (Side A).

9

Teacher has the students read the answers to the Fact

Teacher reads the answers for the Fact Practice sheet (Side A)

Teacher or students read part of the answers to the Fact Practice

Teacher does not have students complete Fact Practice sheet (Side A) during class (e.g., either not completed at all or all of Fact sheet, Side A and B, given as homework). Teacher does not have students complete Fact Practice sheet (Side A)



51

Appendix D

(a)

Practice sheet (Side A).

(b)

after students complete the sheet.

(c)

sheet (Side A).

(d)


during class (e.g., either not completed at all or all of Fact Practice sheet given as homework).


52

Appendix D

C.

Lesson Indicate the length of time (in minutes) devoted to the Lesson Check if the Lesson was not included as a part of the videotaped classroom observation.

If the Lesson was a part of the videotaped classroom observation, complete the following section. If the Lesson was not a part of the videotaped classroom observation, skip to Guided Practice. (a)

10

11

12

13

At the beginning of the Lesson, the teacher clearly states the objective of the Lesson to the students (e.g., states what students will be learning in the Lesson). Teacher frequently models the concepts being taught in the Lesson (e.g., writes numbers on the board or shows students how to do something). Saxon Math manipulatives are used as a part of the lesson.

Teacher models the use of manipulatives prior

(b)

(c)

(d)

At the beginning of the Lesson, the teacher makes an indirect reference to a Lesson objective but does not specifically describe the objective.

Somewhere in the Lesson (other than the beginning), the teacher directly or indirectly refers to the Lesson objective.



Teacher infrequently models the concepts being taught in the lesson.


The appropriate manipulatives are used as a part of the Lesson, although they may not be the specific ones recommended by Saxon Math. The teacher verbally explains how to use the


Manipulatives are not used as a part of the Lesson.





53

Appendix D

(a)

(b)

(c)

to having students use the manipulatives. The teacher clearly explains to students how they are to perform the task with the manipulatives.



14


15


16

Teacher allots 2–3 minutes at the end of the Lesson for students to share their observations about what they learned in the Lesson.

Teacher asks a moderate number of questions of students. Teacher sometimes uses appropriate amount of wait time for students to respond to questions. Teacher allots 2–3 minutes at the end of the Lesson for students to share, however, sharing not necessarily related to observations about what was learned in the Lesson.

17

The meeting is conducted in an orderly manner. Only one student speaks at a time unless the teacher asks for a choral response. All students are focused and on task.

Teacher occasionally asks questions of students. Teacher infrequently uses appropriate amount of wait time for students to respond to questions. Teacher allots less than 2–3 minutes at the end of the Lesson for students to share. Sharing may or may not be related to their observations about what they learned in the Lesson. For the most part, the meeting is conducted in a disorderly manner. Often, students respond to questions without being called upon. Often, multiple students speak at the same time.

For the most part, the meeting is conducted in an orderly manner. Occasionally, students respond to questions without being called upon. Occasionally, multiple students speak

(d)


manipulatives prior to having students use the manipulatives. The teacher provides confusing directions to students in how they are to perform the task with the manipulatives. Teacher asks few questions of the students. Teacher does not use wait time for students to respond to questions.

Teacher does not allow any time at the end of the lesson for students to share their observations about what they learned in the Lesson.

The meeting is conducted in a disorderly manner. All or nearly all of the time students speak without being called upon by the teacher. All or nearly all of the time ©2006 by Edvantia, Inc. All rights reserved


54

Appendix D

(a)

(b)


(c)

(d)





55

Appendix D

D.

Guided Practice Indicate the length of time (in minutes) devoted to the Guided Practice sheet. Check if the Guided Practice sheet was not included as a part of the videotaped classroom observation.

Check if the Guided Practice sheet was to be a part of the videotaped classroom observation. Refer to the print version of the Saxon Math lesson you are observing to make this determination. If Guided Practice was a part of the videotaped classroom observation, complete the following section. If Guided Practice was not a part of the videotaped classroom observation, skip to the Use of Additional Math Materials and Resources section. (a)

(b)

(c)

(d)

18

Teacher reads the directions separately for completion of each problem on the Guided Class Practice sheet (Side A).

19

Teacher does each problem on the Guided Class Practice sheet (Side A) with the students one problem at a time, walking the students through each problem. Students then do each problem on their own one at a time. As a class the teacher and students then look at each problem one at

Teacher has the students read the directions for completion of each problem on the Guided Class Practice sheet (Side A) either out loud to the class or silently to themselves. Teacher models all the problems on the Guided Class Practice sheet (Side A) at once while the students listen. Students then complete all the problems on the sheet. Following completion of the entire sheet, the problems are corrected together as a class.

Teacher does not read directions for any of the problems on the Guided Class Practice sheet (Side A) or have the students read the directions, but instead proceeds directly to solving the problems. Teacher has students independently complete Guided Class Practice sheet (Side A) during class time without any modeling from the teacher. Following completion of the entire sheet the problems are corrected together as a class.

Teacher does not have students complete the Guided Class Practice sheet (Side A) during class (e.g., either not completed at all or all of Guided Class Practice sheet, Side A and B, given as homework). Teacher does not have students complete the Guided Practice sheet (Side A) during class (e.g., either not completed at all or all of Guided Class Practice sheet, Side A and B, given as homework).



56

Appendix D

(a)

20

a time to verify correctness. Teacher has students explain how to find each answer of the Guided Class Practice sheet (Side A) one problem at a time.

(b)

Teacher has students explain how to find some answers of the Guided Class Practice sheet (Side A) one problem at a time.

(c)

(d)

Teacher has students give answers to the Guided Class Practice sheet (Side A), but do not explain how they arrived at their answers.

Teacher does not have students complete Guided Class Practice sheet (Side A) during class (e.g., either not completed at all or all of Guided Class Practice sheet given as homework).



57

Appendix D

E. Classroom Arrangement and Resources For each of the following lists, indicate the classroom resources which were used or present and the type of classroom arrangement. If you did not have the opportunity to observe the classroom arrangement, check “did not observe.” The Meeting Classroom Arrangement (check one): Students seated on floor in semicircle or circle in front of the board Students seated in chairs in semicircle or circle in front of the board Students seated in desks in rows Students seated in desks in clusters or other arrangement other than rows Other (describe): Did not observe The Lesson Classroom Arrangement (check one): Students seated on floor in semicircle or circle in front of the board Students seated in chairs in semicircle or circle in front of the board Students seated in desks in rows Students seated in desks in clusters or other arrangement other than rows Other (describe): Did not observe Use of Resources and Materials (check all that apply): The Meeting Board Manipulatives (The Meeting) Meeting Masters (for students) Poster-size Lesson Charts Manipulatives (the lesson) Lesson Masters (for students) Student Guided Practice sheet Student Class Practice sheet Student Written Assessments Student Fact Cards (second grade only) Learning Wrap-UpsTM Teacher Fact Cards Student Math Folders (for storing Meeting Masters and various lesson masters) F. Use of Additional Math Materials and Resources: Make note of any additional math materials and resources that were used throughout the videotaped observation that were not a part of the Saxon Math program. Include a complete description of how and what purpose they were used. ©2006 by Edvantia, Inc. All rights reserved Saxon Elementary Math Program Effectiveness Study

58

Appendix D

Introductory Protocol for LoU Telephone Interviews Saxon Mathematics Program Study Levels of Use Telephone Interview 2005–06 Teacher: Grade Level: School: Date: Interviewer: Time Interview Began: Time Interview Concluded:

A.

Hello, this is ________. I’m with Edvantia in Charleston, West Virginia. Thank you for taking the time to talk with me today.

B.

The purpose of this interview is to help us understand how the Saxon Math program is being implemented in your classroom.

C.

Your responses will be kept confidential and they will be summarized with the responses from other teachers participating in the national study on the effectiveness of the Saxon Math program and included in our final report to Harcourt Achieve.

D.

It is anticipated the interview will take approximately 30 minutes. I will be tape recording the interview so that I can more closely focus on what you are saying. Is this all right with you? [NOTE: Only include the previous 2 sentences if you are tape recording the interview.] You have the right to withdraw from the study, including participation in this interview, at any time, should you choose. You will not be penalized in any way if you choose to do so. Do you have any questions before we begin?

E.

Following the interview, if you have any other questions about this interview or the Saxon Math Program Study, you may contact myself, Kim Good, at [email protected] or 1-800-624-9120 ext. 5449. For questions about participants’ rights, you may contact Merrill Meehan, AEL’s Institutional Review Board Chair, 1-800624-9120 ext. 5432 or [email protected].


59

Appendix D

LoU Telephone Interview Protocol LoU Interview Questions Are you currently using the Saxon Mathematics Program? (if yes, go to YES RESPONSE questions. If no, go to NO RESPONSE questions. NO RESPONSE: Have you ever used it in the past? If so, when? Why did you stop? If yes, have used in the past… Can you describe for me how you organized your use of the Saxon Math program, what problems you found, what its effects appeared to be on students? When you assess the Saxon Math program at this point in time, what do you see as the strengths and weaknesses? If no … 0/I–II: Have you made a decision to use the Saxon Math program in the future? I/II:

If so, when will you begin use?

Knowledge: Can you describe the Saxon Math program for me as you see it? Acquiring Information: Are you currently looking for any information about the Saxon Math program? What kind? For what purposes? Knowledge: What do you see as the strengths and weaknesses of the Saxon Math program in your situation? Assessing: At this point in time, what kinds of questions are you asking about the Saxon Math program? [Interviewer: Add examples] Sharing: Do you ever talk with others and share information about the Saxon Math program? What do you share? Planning: What are you planning with respect to the Saxon Math program? Can you tell me about any preparation or plans you have been making for the use of the Saxon Math program? Final Question (optional): Can you summarize for me where you see yourself right now in relation to the use of the Saxon Math program?


60

Appendix D

YES RESPONSE: Open-ended: Please describe for me how you use the Saxon Math program. Probe on The Meeting, the Lesson, in-class Guided Practice (applicable Gr 1–3), independent homework written practice (applicable Gr 1–3), Lesson practice (applicable K), Handwriting Practice/Counting Practice (applicable K), manipulatives, math center activities and assessments) Assessing/Knowledge: What do you see as the strengths and weaknesses of the Saxon Math program in your situation? (Have you made any attempt to do anything about weaknesses? Probe those mentioned by teachers specifically.) Acquiring Information: Are you currently looking for any information about the Saxon Math program? What kind? For what purposes? LoUV: Do you work with others in your use of the Saxon Math program? Do you meet on a regular basis? Have you made any changes in your use of the Saxon Math program based on this coordination? If yes, go to LoU V Probes. If no, continue with Sharing Question. Sharing Question: Do you ever talk with others [in your school] about the Saxon Math program? What do you tell them? Assessing Question: Have you considered any alternatives or different ways of doing things with the program? Are you doing any evaluating, either formally or informally, that would affect your use of the Saxon Math program? Have you received any feedback from students that would affect the way you’re using the Saxon Math program? What have you done with the information you got? Question III/IVA/IVB: Have you made any changes recently in how you use the Saxon Math program? What? Why? How recently? Are you considering making any changes? Planning/Status Reporting Question: As you look ahead to later this year, what plans do you have in relation to your use of the Saxon Math program? Question III–V/VI: Are you considering or planning to make major modifications or replace the Saxon Math program at this time?


61

Appendix D

LoU V Probes 1. Please describe for me how you work together. What things do you share with each other. 2. What do you see as the effects of this collaboration? 3. Are you looking for any particular kind of information in relation to this collaboration? 4. Do you talk with others about your collaboration? If so, what do you share with them? 5. Have you done any formal or informal evaluation of how your collaboration is working? 6. What plans do you have for this effort in the future? If you have enough evidence to place the person at an LoU V, go to Question III–V/VI. If you do not think the person is an LoU V, go to the Sharing Question.


62

Appendix D

Experimental-Site Coordinators’ Posttest Administration Instructions


To:

Saxon Math Study Site Coordinators (experimental schools)

From: Kimberly Good, PhD, Research Associate Date: Spring 2006 Re:

Site Coordinator Stanford Achievement Test, Ninth Edition Posttest Teacher Surveys Saxon Math Student-Assessment Data

This mailing contains the following testing materials as well as teacher surveys and Saxon Math student assessment data: 1. Teacher Posttest Information Memo Distribute one Teacher Posttest Guidelines from Edvantia to each participating teacher. 2. Participating Students Listing The participating student listing identifies students who completed the pretests in November 2005. If for some reason a student is no longer participating in the study (e.g., moved or withdrew participation from the study), strike the student’s name from the roster. Distribute one Participating Student Listing to each teacher. 3. Directions for Administering booklets Distribute the appropriate grade level Directions for Administering booklet to the participating teachers. Booklets are labeled by grade level, and there is one booklet for each teacher. 4. Test Booklets Distribute the appropriate test booklets to participating teachers. The test booklets are labeled by grade level (Kdg = SESAT 2, first grade = PRIMARY 1/Abbr, second grade = PRIMARY 2/Abbr, third grade = PRIMARY 3/Abbr). Each teacher should receive eight test booklets corresponding to the grade level they teach. Note: Teachers will only be administering the math test. 5. Rulers Distribute the rulers to the participating teachers. Each first-, second-, and third-grade teacher should receive eight rulers.


63

Appendix D

6. Teacher Surveys Distribute the appropriate grade-level survey and one envelope to each participating teacher. The grade level is indicated at the top of the survey. Each teacher should complete and return it to the site coordinator in the sealed envelope. 7. Saxon Math Student-Assessment Data Distribute to each teacher one memo that includes the instructions for recording the Saxon Math student-assessment data and the appropriate grade-level spreadsheet for recording that information. General Information Please administer the test to the students as close to the end of the school year as possible. Students will mark all responses in their test booklets. The site coordinator should collect all test booklets (both completed and extra test booklets, if applicable) and the Directions for Administering booklet from each teacher. The test booklets should be returned to Edvantia immediately upon conclusion of testing. Please rubber band each classroom set of completed test booklets together. If applicable, separately rubber band each set of unused grade level test booklets. It is critical that all test booklets and the Directions for Administering booklets be returned to Edvantia. Please return all test booklets and related documents, the completed surveys and Saxon Math student assessment data within a week after testing. After you have assembled all data and information mentioned above, please return them to Edvantia using the UPS envelope and label provided. Enter your name and school’s address in the return envelope. Call the UPS number on the label provided to schedule a pick-up at your school. All shipping charges are being handled by Edvantia. We prefer that all materials (posttests, teacher surveys and Saxon Math student-assessment data) be submitted at one time. Upon receipt of the posttests, teacher surveys and Saxon Math student-assessment data we will process the second half of stipend checks for the site coordinators and the participating teachers. The checks will be mailed to your home address. If you have any questions about the contents of this memo or the study, please feel free to contact me at 1.800.624.9120 (ext. 5449) or [email protected].


64

Appendix D

Experimental-Site Teachers’ Posttest Administration Instructions


To:

Saxon Math Study Participating Teachers (experimental schools)


Site Coordinator Stanford Achievement Test, Ninth Edition Posttest Teacher Surveys Saxon Math Student-Assessment Data

You should have received the following materials for administering the Stanford Achievement Test, Ninth Edition (SAT 9). Although you have the full battery of SAT 9 tests, you will only be administering the math subtest. This packet also contains a teacher survey and information on collecting Saxon Math student-assessment data. 1. Participating Students Listing The participating student listing identifies students who completed the pre-tests in November 2005. If for some reason a student is no longer participating in the study (e.g., moved or withdrew participation from the study), strike the student’s name from the roster. 2. Directions for Administering Booklet Please review the information contained in the Directions for Administering booklet. It is particularly important to familiarize yourself with the section titled “General Directions for Administering.” 3. Test Booklets Only the eight students randomly selected to participate in the study will be tested. You will only be administering the math test. The test booklets are labeled by grade level (Kdg = SESAT 2, first grade = PRIMARY 1/Abbr, second grade = PRIMARY 2/Abbr, third grade = PRIMARY 3/Abbr). The total testing time for kindergarten is 30 minutes, and total testing time for Grades 1–3 is 80 minutes. The testing time does not account for the preparation time. For Grades 1–3, there are two math subtests: Problem Solving and Procedures. It is advised that you administer the Problem Solving subtest one day and the Procedures subtest a second day. We encourage you to administer the test as close to the end of the school year as possible in order to have the maximum time between the pre- and posttest. Students will mark their responses to the math subtest in the test booklet. Students will need a number 2 pencil to complete the test. Saxon Elementary Math Program Effectiveness Study

65

Appendix D

***IMPORTANT*** You will also need to complete the information on the back side of the test booklet for each of the eight students (student name, school, gender, grade, teacher, district and date of birth) Please bubble in as appropriate. You do not need to complete the student number or other information boxes. 4. Rulers You will receive a ruler for each of the eight students being tested (applicable for first, second and third grades). Specific instructions on when and how to use these are found in the Directions for Administering booklets. 5. Teacher Survey Complete and return your grade-level survey, and return it to the site coordinator in the sealed envelope. 6. Saxon Math Student-Assessment Data Review the memo that includes the instructions for recording the Saxon Math student-assessment data and complete the student assessment information in the corresponding spreadsheet. Return the completed spreadsheet to the site coordinator. General Information Return (1) the completed test booklets (extra test booklets, if applicable); (2) Directions for Administering booklet; (3) completed teacher survey; and (4) completed Saxon Math student-assessment data spreadsheet to your site coordinator immediately after testing. It is critical that all test booklets (both completed and unused, if applicable) and the Directions for Administering booklet be returned to your site coordinator. Return these items to your site coordinator immediately following testing. Upon receipt of the posttests, teacher surveys, and Saxon Math student-assessment data, we will process the second half of stipend checks for the site coordinators and the participating teachers. The checks will be mailed to your home address. If you have any questions about the contents of this memo or the study, please feel free to contact me at 1.800.624.9120 (ext. 5449) or [email protected].


66

Appendix D

Comparison-Site Coordinator’ Posttest Administration Instructions


To:

Saxon Math Study Site Coordinators (comparison schools)


Site Coordinator Stanford Achievement Test, Ninth Edition Posttest Teacher Surveys

This mailing contains the following testing materials and teacher surveys: 1. Teacher Posttest Information Memo Distribute one Teacher Posttest Guidelines from Edvantia to each participating teacher. 2. Participating Students Listing The participating student listing identifies students who completed the pre-tests in November 2005. If for some reason a student is no longer participating in the study (e.g., moved or withdrew participation from the study), strike the student name from the roster. Distribute one Participating Student Listing to each teacher. 3. Directions for Administering Booklets Distribute the appropriate grade-level Directions for Administering booklet to participating teachers. Booklets are labeled by grade level, and there is one booklet for each teacher. 4. Test booklets Distribute the appropriate test booklets to participating teachers. The test booklets are labeled by grade level (Kdg = SESAT 2, first grade = PRIMARY 1/Abbr, second grade = PRIMARY 2/Abbr, third grade = PRIMARY 3/Abbr). Each teacher should receive eight test booklets corresponding to the grade level they teach. Note: Teachers will only be administering the math test. 5. Rulers Distribute the rulers to the participating teachers. Each first-, second-, and third-grade teacher should receive eight rulers.


67

Appendix D

6. Teacher Surveys Distribute the appropriate grade-level survey and one envelope to each participating teacher. The grade level is indicated at the top of the survey. Each teacher should complete and return it to the site coordinator in the sealed envelope. General Information Please administer the test to the students as close to the end of the school year as possible. Students will mark all responses in their test booklets. The site coordinator should collect all test booklets (both completed and extra test booklets, if applicable) and the Directions for Administering booklet from each teacher. The test booklets should be returned to Edvantia immediately upon conclusion of testing. Please rubber band each classroom set of completed test booklets together. If applicable, separately rubber band each set of unused grade-level test booklets. It is critical that all test booklets and the Directions for Administering booklets be returned to Edvantia. Please return all test booklets and related documents and the completed surveys within a week after testing. After you have assembled all data and information mentioned above, please return them to Edvantia using the UPS envelope and label provided. Enter your name and school’s address in the return envelope. Call the UPS number on the label provided to schedule a pick-up at your school. All shipping charges are being handled by Edvantia. We prefer that all materials (posttests and teacher surveys) be submitted at one time. Upon receipt of the post-tests and teacher surveys we will process the second half of stipend checks for the site coordinators and the participating teachers. The checks will be mailed to your home address. If you have any questions about the contents of this memo or the study, please feel free to contact me at 1.800.624.9120 (ext. 5449) or [email protected].


68

Appendix D

Comparison Teachers’ Posttest Administration Instructions


To:

Saxon Math Study Participating Teachers (comparison schools)


Site Coordinator Stanford Achievement Test, Ninth Edition Posttest Teacher Surveys

You should have received the following materials for administering the Stanford Achievement Test, Ninth Edition (SAT 9). Although you have the full battery of SAT 9 tests, you will only be administering the math subtest. This packet also contains a teacher survey. 1. Participating Students Listing The participating student listing identifies students who completed the pretests in November 2005. If for some reason a student is no longer participating in the study (e.g., moved or withdrew participation from the study), strike the student’s name from the roster. 2. Directions for Administering Booklet Please review the information contained in the Directions for Administering booklet. It is particularly important to familiarize yourself with the section titled “General Directions for Administering.” 3. Test Booklets Only the eight students randomly selected to participate in the study will be tested. You will only be administering the math test. The test booklets are labeled by grade level (Kdg=SESAT 2, first grade = PRIMARY 1/Abbr, second grade = PRIMARY 2/Abbr, third grade = PRIMARY 3/Abbr). The total testing time for kindergarten is 30 minutes, and total testing time for Grades 1–3 is 80 minutes. The testing time does not account for the preparation time. For Grades 1–3 there are two math subtests: Problem Solving and Procedures. It is advised that you administer the Problem Solving subtest one day and the Procedures subtest a second day. We encourage you to administer the test as close to the end of the school year as possible in order to have the maximum time between the pre- and posttest. Students will mark their responses to the math subtest in the test booklet. Students will need a number 2 pencil to complete the test.


69

Appendix D

***IMPORTANT*** You will also need to complete the information on the back side of the test booklet for each of the eight students (student name, school, gender, grade, teacher, district, and date of birth). Please bubble in as appropriate. You do not need to complete the student number or other information boxes. 4. Rulers You will receive a ruler for each of the eight students being tested (applicable for first, second and third grades). Specific instructions on when and how to use these are found in the Directions for Administering booklets. 5. Teacher Survey Complete and return your grade-level survey, and return it to the site coordinator in the sealed envelope. General Information Return (1) the completed test booklets (extra test booklets, if applicable); (2) Directions for Administering booklet; and (3) completed teacher survey to your site coordinator immediately after testing. It is critical that all test booklets (both completed and unused, if applicable) and the Directions for Administering booklet be returned to your site coordinator. Return these items to your site coordinator immediately following testing. Upon receipt of the posttests and teacher surveys we will process the second half of stipend checks for the site coordinators and the participating teachers. The checks will be mailed to your home address. If you have any questions about the contents of this memo or the study, please feel free to contact me at 1.800.624.9120 (ext. 5449) or [email protected].


70

Appendix D

Instructions for Recording Saxon Math Student-Assessment Data


To:

Saxon Math Study Site Coordinators and Teachers

From: Kimberly Good, PhD, Research and Evaluation Specialist I Date: Spring 2006 Re:

Instructions for Recording Saxon Math Student-Assessment Data

As a part of the Saxon Math Study, we are collecting two types of student-achievement data. Participating students are completing a pre- and posttest of the SAT 9 math assessment. The second type of student achievement we are requesting is a sample of Saxon Math studentassessment data. We are asking for scores from a beginning, mid-year, and end of the year Saxon assessment. Specific assessments are listed below for each grade. Please complete the spreadsheet with the requested information for each student participating in the study. The following are specific instructions for each grade level about which assessments we are requesting and how we would like the information recorded. We realize you may use a different scoring method than we are requesting. If necessary, please transform your students’ scores as described below. This will enable us to aggregate and analyze the student-assessment data across all participating schools in this study. Kindergarten Beginning of Year use Oral Assessment 2 (for use with Lesson 20): Enter the number each student reached on the final (fourth) assessment date. For example, on the final or fourth assessment date, if the child counted to 50, record 50 in the spreadsheet. Middle of Year: Determine the midpoint of your school year and record the assessment that is administered as close to that point as possible (e.g., may be the first assessment administered in the second semester). Enter the number each participating student answered correctly. Points are only given for a right answer. No points are given for a wrong or partially right response. Enter the # of that assessment in the top of Column D of the spreadsheet and the total number possible a student could get correct (e.g., Oral Assessment #5 = 7 points, Oral Assessment #6 = 4 points, Oral Assessment #7 = 6 points, and Oral Assessment #8 = 4 points possible).


71

Appendix D

End of Year: At the conclusion of the school year, record the student-assessment data for the last assessment administered. Enter the number each participating student answered correctly. Points are only given for a right answer. No points are given for a wrong or partially right response. Enter the # of that assessment in the top of Column E of the spreadsheet and the total number possible a student could get correct (e.g., Oral Assessment #9 = 5 points, Oral Assessment #10 = 3 points, Oral Assessment #11 = 5 points, Oral Assessment #12 = 4 points, and Oral Assessment #13 = 6 points possible). First, Second, and Third Grades Each assessment is worth a maximum total of 100 points. See the Assessment Scoring Guide accompanying the Teacher’s Resource Materials. Beginning of Year use Written Assessment 5 (for use with Lesson 30, part 2): Enter the total number of points out of a possible 100 each participating student received. Middle of Year: Determine the midpoint of your school year, and record the assessment that is administered as close to that point as possible (e.g., may be the first assessment administered in the second semester). Enter the total number of points out of a possible 100 each participating student received. Enter the # of that assessment in the top of Column D of the spreadsheet (e.g., Written Assessment 13). End of Year: At the conclusion of the school year record the student-assessment data for the last assessment administered. Enter the total number of points out of a possible 100 each participating student earned. Enter the # of that assessment in the top of Column E of the spreadsheet (e.g., Written Assessment 26).


72

Appendix D

Experimental Teacher Surveys (Kindergarten)

Kindergarten Teacher Survey Experimental School Saxon Mathematics Effectiveness Study Instructions: We are asking that you complete this survey as a part of the Saxon Mathematics Effectiveness Study in which your school is participating. The purpose of this survey is to gather information about your mathematics classroom curriculum and instruction. Please be assured that we are not evaluating your teaching. Because your name will never be associated with your answers, please feel free to answer as candidly as possible. Specific instructions appear throughout to guide you through the survey. You will be able to complete the survey in about 20–25 minutes. Once you have completed the survey, please place it in the manila envelope and return the sealed envelope to your site coordinator. The site coordinator will mail the envelopes to Edvantia for data entry. All responses will be kept confidential, and data will be reported in aggregate format only. You have the right to withdraw from the study, including completion of this survey, at any time. You will not be penalized in any way if you choose to do so. If you have any questions about this survey or about the Saxon Math Program Study, you may contact Kim Good at [email protected] or 800.624.9120 ext. 5449. If you have questions about subjects’ rights, contact Merrill Meehan, Edvantia’s Institutional Review Board Chair, at 800.624.9120 ext. 5432 or [email protected]. Implementation of Saxon Mathematics Program Components The statements in Parts A–E of this survey address several of the primary components of the Saxon Mathematics Program (The Meeting, Lesson, Lesson Practice, Handwriting Practice, and Counting Practice). Please indicate the extent to which you use the components in the ways described. Part A. The Meeting Check one response for each statement. Never

Rarely

Sometimes

Often

Always







73

Appendix D Part B. Lesson Check one response for each statement. Never

Rarely

Sometimes

Often

Always








I allot 2–3 minutes at the end of the Lesson for students to share their observations about what they learned in the Lesson.


Never

Rarely

Sometimes

Often

Always

I read the directions of the Lesson Practice sheet to the students.

Students complete the Lesson Practice sheet independently.

I circulate and check students’ papers continuously as they work on the Lesson Practice sheet.

Part C. Lesson Practice Check one response for each statement.


74

Appendix D Part D. Handwriting Practice Check one response for each statement. Never

Rarely

Sometimes

Often

Always

I read the directions for completion of the Handwriting Practice sheet to the students.

I give the class guidance on initial completion of the Handwriting Practice sheet.

I circulate and check students’ papers continuously as they work on the Handwriting Practice sheet.

Never

Rarely

Sometimes

Often

Always

I read the directions for completion of the Counting Practice sheet to the students.

I give the class guidance on initial completion of the Counting Practice sheet.

I circulate and check students’ papers continuously as they work on the Counting Practice sheet.

Part E. Counting Practice Check one response for each statement.

Part F. Implementation Logistics 1.

On average, how much time do you devote to each of the following Saxon Math components when they are taught? (Check one response for each component.) None

10 min.

11–15 min.

16–20 min.

21–25 min.

26–30 min.

31–35 min.

36 min. or more

The Meeting

Lesson

Lesson Practice

Handwriting Practice

Counting Practice


75

Appendix D 2.

During what part of the day do you usually conduct each of the following Saxon Math components? (Check one response for each component.) Don’t use


Middle of the morning

End of the morning (before lunch)



End of the day

The Meeting

The Lesson

Lesson Practice

Handwriting Practice

Counting Practice

3.

By the conclusion of this school year, what is the last lesson # you anticipate completing? ________________________

4.

By the conclusion of this school year, what is the last oral assessment # you anticipate completing? ________________

Part G. Use of Saxon Math Assessments Yes

No

Did you use the Saxon Math Assessments?

Did you use the assessment reporting forms?

Did you use the assessment scoring guides?

If you indicate “no” for any of the questions below, please provide detail on the assessments used, reporting format, and/or the scoring method used.

Part H. Use of Saxon Resources

Yes

No

Did you use the Teacher’s Resource CD?

Did you use the Lesson Planner CD?


76

Appendix D Part I. Additional Information 1.

Did you use other materials to supplement your mathematics curriculum? If yes, what materials did you use and for what purposes (e.g., what curriculum areas did the materials address)? If not, why not?

2.

What aspects of the Saxon Math program made you or your school decide to adopt the program?

3.

What do you like best about the Saxon Math program?

4.

What changes, if any, should be made to the Saxon Math program?


77

Appendix D Part J. Classroom and Teacher Information 1.

How is your kindergarten day structured? (Check one)

2.

Do you teach in a self-contained classroom (i.e., are you responsible for teaching all or most academic subjects to a single class)?

3.

Half day Whole day Other (please specify): ______________________________________________________________________

Yes No

Do you teach in a multiage classroom (i.e., is more than one grade level taught by you in your classroom)?

Yes No

4.

How many total students are enrolled in your class (or in each of your classes if you teach more than 1 section)?_______

5.

How would you describe your class in terms of variation in students’ mathematics ability? (Check one)

6.

Fairly homogeneous and low in ability Fairly homogeneous and average in ability Fairly homogeneous and high in ability Heterogeneous with a mixture of two or more ability levels

How many students in your class are formally classified as each of the following? (Estimate if necessary.) English language learner/LEP Special education Gifted and talented

7.

Are you:

8.

Which of these age categories includes you?

9.

Male Female

20–30 31–40 41–50 51–60 61 or over

Including this year, how many years have you taught full time in a regular teaching position? Total: In this district: In this school: At this grade level:

10. Including this year, how many years have you been teaching the Saxon Mathematics Program? ____________________


78

Appendix D

Experimental Teacher Survey (First Grade) First-Grade Teacher Survey Experimental School Saxon Mathematics Effectiveness Study Instructions: We are asking that you complete this survey as a part of the Saxon Mathematics Effectiveness Study in which your school is participating. The purpose of this survey is to gather information about your mathematics classroom curriculum and instruction. Please be assured that we are not evaluating your teaching. Because your name will never be associated with your answers, please feel free to answer as candidly as possible. Specific instructions appear throughout to guide you through the survey. You will be able to complete the survey in about 20-25 minutes. Once you have completed the survey, please place it in the manila envelope and return the sealed envelope to your site coordinator. The site coordinator will mail the envelopes to Edvantia for data entry. All responses will be kept confidential, and data will be reported in aggregate format only. You have the right to withdraw from the study, including completion of this survey, at any time. You will not be penalized in any way if you choose to do so. If you have any other questions about this survey or about the Saxon Math Program Study, you may contact Kim Good at [email protected] or 800.624.9120 ext. 5449. For questions about subjects’ rights, contact Merrill Meehan, Edvantia’s Institutional Review Board Chair, 800.624.9120 ext. 5432 or [email protected]. Implementation of Saxon Mathematics Program Components The following are statements about several of the primary components of the Saxon Mathematics Program (The Meeting, Number Fact Practice, Lesson, and Guided Practice). Please indicate the extent to which you use the components in the ways described. Part A. The Meeting Check one response for each statement. Never

Rarely

Sometimes

Often

Always







79

Appendix D Part B. Number Fact Practice Check one response for each statement. Never

Rarely

Sometimes

Often

Always



I allow time for students to independently complete the Class Fact Practice sheet (Side A).

I read all the problems and answers for the Class Fact Practice sheet (Side A) after students complete the sheet.

Never

Rarely

Sometimes

Often

Always


I model the concepts being taught in the lesson (e.g., by writing numbers on the board or showing students how to do something).






I allot 2–3 minutes at the end of the lesson for students to share their observations about what they learned in the Lesson.

The lesson is conducted in an orderly manner. (Only one student speaks at a time unless I ask for a choral response. All students are focused and on task.)

Part C. Lesson Check one response for each statement.


80

Appendix D Part D. Guided Practice Check one response for each statement. Never

Rarely

Sometimes

Often

Always


I allow time for students to complete each problem on the Guided Class Practice sheet (Side A) and then write the answer to each problem on a chart.

I circulate and checks students’ papers the entire time as they work on each problem of the Guided Class Practice sheet (Side A).

Part E. Implementation Logistics 1.


10 min.

11–15 min.

16–20 min.

21–25 min.

26–30 min.

31–35 min.

36 min. or more

The Meeting

Number Fact Practice

Lesson

Guided Practice

2.







End of the day

The Meeting


Lesson

Guided Practice


81

Appendix D 3.

By the conclusion of this school year, what is the last lesson # you anticipate completing? _________________

4.

By the conclusion of this school year, what is the last assessment # you anticipate completing? _____________

Part F. Use of Saxon Math Assessments Yes

No


Did you use the assessment reporting form?


If you indicate “no” for any of the questions below, please provide detail on the assessments used, reporting format, and/or the scoring method used.

Part G. Use of Saxon Resources

Yes

No




82

Appendix D Part H. Additional Information 1.


2.


3.


4.



83

Appendix D Part I. Classroom and Teacher Information 1.


2.

Yes No


Yes No

3.

How many total students are enrolled in your class (or in each of your classes, if you teach more than 1 section)? ______________________________________________________________________________________

4.

How would you describe your class in terms of variation in mathematics ability? (Check one)

5.



6.

Are you:

7.


8.

Male Female

20–30 31–40 41–50 51–60 61 or over

Including this year, how many years have you taught full time in a regular teaching position? Total: In this district: In this school: At this grade level:

9.

Including this year, how many years have you been teaching the Saxon Mathematics Program? _____________


84

Appendix D

Experimental Teacher Survey (Second Grade) Second-Grade Teacher Survey Experimental School Saxon Mathematics Effectiveness Study Instructions: We are asking that you complete this survey as a part of the Saxon Mathematics Effectiveness Study in which your school is participating. The purpose of this survey is to gather information about your mathematics classroom curriculum and instruction. Please be assured that we are not evaluating your teaching. Because your name will never be associated with your answers, please feel free to answer as candidly as possible. Specific instructions appear throughout to guide you through the survey. You will be able to complete the survey in about 20–25 minutes. Once you have completed the survey, please place it in the manila envelope and return the sealed envelope to your site coordinator. The site coordinator will mail the envelopes to Edvantia for data entry. All responses will be kept confidential, and data will be reported in aggregate format only. You have the right to withdraw from the study, including completion of this survey, at any time. You will not be penalized in any way if you choose to do so. If you have any questions about this survey or about the Saxon Math Program Study, you may contact Kim Good at [email protected] or 800.624.9120 ext. 5449. For questions about subjects’ rights, contact Merrill Meehan, Edvantia’s Institutional Review Board Chair, at 800.624.9120 ext. 5432 or [email protected]. Implementation of Saxon Mathematics Program Components The statements in Parts A–D of this survey address several of the primary components of the Saxon Mathematics Program (The Meeting, Number Fact Practice, Lesson, and Guided Practice). Please indicate the extent to which you use the components in the ways described. Part A. The Meeting Check one response for each statement. Never

Rarely

Sometimes

Often

Always







85


Rarely

Sometimes

Often

Always



I have students complete Fact Practice sheet (Side A) while timing them for a specified period of time.

I have students read the answers to the Fact Practice sheet (Side A).

Never

Rarely

Sometimes

Often

Always

At the beginning of the Lesson, I clearly state the objective(s) of the Lesson to the students (e.g., what students will be learning in the Lesson).







I allot 2–3 minutes at the end of the Lesson for students to share their observations about what they learned in the Lesson.




86


Rarely

Sometimes

Often

Always


I do each problem on the Guided Class Practice sheet (Side A) with the students one problem at a time, walking the students through each problem. Students then do each problem on their own, one at a time.

As a class, after the students have completed a problem, the students and I look at each problem one at a time to verify correctness.

I have students explain how to find the answer to each problem on the Guided Class Practice sheet (Side A), one problem at a time.


2.


10 min.

11–15 min.

16–20 min.

21–25 min.

26–30 min.

31–35 min.

36 min. or more

The Meeting


Lesson

Guided Practice







End of the day

The Meeting


Lesson

Guided Practice

3.



87

Appendix D

4.



No




If you indicate “no” for any of the questions below, please provide detail on the assessments used, reporting format and/or the scoring method used.


Yes

No




88



2.


3.


4.



89

Appendix D Part I. Classroom and Teacher Information 1.


2.

Yes No


Yes No

3.

How many total students are enrolled in your class (or each of your classes if you teach more than 1 section)? _______

4.

How would you describe your class in terms of variation in mathematics ability? (Check one)

5.



6.

Are you:

7.


8.

Male Female

20–30 31–40 41–50 51–60 61 or over

Including this year, how many years have you taught full-time in a regular teaching position? Total: In this district: In this school: At this grade level:

9.

Including this year, how many years have you been teaching the Saxon Mathematics Program? _____________


90

Appendix D

Experimental Teacher Survey (Third Grade) Third-Grade Teacher Survey Experimental School Saxon Mathematics Effectiveness Study Instructions: We are asking that you complete this survey as a part of the Saxon Mathematics Effectiveness Study in which your school is participating. The purpose of this survey is to gather information about your mathematics classroom curriculum and instruction. Please be assured that we are not evaluating your teaching. Because your name will never be associated with your answers, please feel free to answer as candidly as possible. Specific instructions appear throughout to guide you through the survey. You will be able to complete the survey in about 20–25 minutes. Once you have completed the survey, please place it in the manila envelope and return the sealed envelope to your site coordinator. The site coordinator will mail the envelopes to Edvantia for data entry. All responses will be kept confidential, and data will be reported in aggregate format only. You have the right to withdraw from the study, including completion of this survey, at any time. You will not be penalized in any way if you choose to do so. If you have any questions about this survey or about the Saxon Math Program Study, you may contact Kim Good at [email protected] or 800.624.9120 ext. 5449. For questions about subjects’ rights, contact Merrill Meehan, Edvantia’s Institutional Review Board Chair, at 800.624.9120 ext. 5432 or [email protected]. Implementation of Saxon Mathematics Program Components The statements in Parts A–D of this survey address several of the primary components of the Saxon Mathematics Program (The Meeting, Number Fact Practice, Lesson, and Guided Practice). Please indicate the extent to which you use the components in the ways described. Part A. The Meeting Check one response for each statement. Never

Rarely

Sometimes

Often

Always







91


Rarely

Sometimes

Often

Always



I have students complete Fact Practice sheet (Side A) while timing them for a specified period of time.

I have students read the answers to the Fact Practice sheet (Side A).

Never

Rarely

Sometimes

Often

Always

At the beginning of the lesson I clearly state the objective(s) of the lesson to the students (e.g., what students will be learning in the lesson).

I model the concepts being taught in the lesson (e.g., by writing numbers on the board or showing students how to do something).

I use Saxon Math manipulatives as a part of the lesson.





I allot 2–3 minutes at the end of the lesson for students to share their observations about what they learned in the lesson.

The lesson is conducted in an orderly manner. (Only one student speaks at a time unless I ask for a choral response. All students are focused and on task.)



92


Rarely

Sometimes

Often

Always


I do each problem on the Guided Class Practice sheet (Side A) with the students one problem at a time, walking the students through each problem. Students then do each problem on their own one at a time.

As a class, after the students have completed a problem, the students and I look at each problem one at a time to verify correctness.

I have students explain how to find the answer to each problem on the Guided Class Practice sheet (Side A), one problem at a time.


6.


10 min.

11–15 min.

16–20 min.

21–25 min.

26–30 min.

31–35 min.

36 min. or more

The Meeting


Lesson

Guided Practice







End of the day

The Meeting


Lesson

Guided Practice

7.



93

Appendix D 8.



No




If you indicate “no” for any of the questions below, please provide detail on the assessments used, reporting format and/or the scoring method used.


Yes

No




94



6.


7.


8.



95

Appendix D Part I. Classroom and Teacher Information 10. Do you teach in a self-contained classroom (i.e., are you responsible for teaching all or most academic subjects to a single class)?

Yes No

11. Do you teach in a multiage classroom (i.e., is more than one grade level taught by you in your classroom)?

Yes No

12. How many total students are enrolled in your class (or each of your classes, if you teach more than 1 section)? ______ 13. How would you describe your class in terms of variation in mathematics ability? (Check one)


14. How many students in your class are formally classified as each of the following? (Estimate if necessary.) English language learner/LEP Special education Gifted and talented 15. Are you:

Male Female

16. Which of these age categories includes you?

20–30 31–40 41–50 51–60 61 or over

17. Including this year, how many years have you taught full-time in a regular teaching position? Total: In this district: In this school: At this grade level: 18. Including this year, how many years have you been teaching the Saxon Mathematics Program? _____________


96

Appendix D

Comparison Teacher Survey (Kindergarten) Kindergarten Teacher Survey Comparison School Saxon Mathematics Effectiveness Study Instructions: Your school is participating as a comparison site in the Saxon Mathematics Effectiveness Study. We are asking you to complete this survey to help us gather information on mathematics classroom curriculum and instruction in your school. Please be assured that we are not evaluating your teaching. Because your name will never be associated with your answers, please feel free to answer as candidly as possible. Specific instructions appear throughout to guide you through the survey. You will be able to complete the survey in about 20–25 minutes. Once you have completed the survey, please place it in the manila envelope and return the sealed envelope to your site coordinator. The site coordinator will mail the envelopes to Edvantia for data entry. All responses will be kept confidential, and data will be reported in aggregate format only. You have the right to withdraw from the study, including completion of this survey, at any time. You will not be penalized in any way if you choose to do so. If you have any questions about this survey or about the Saxon Math Program Study, you may contact Kim Good at [email protected] or 800.624.9120 ext. 5449. If you have questions about subjects’ rights, contact Merrill Meehan, Edvantia’s Institutional Review Board Chair, at 800.624.9120 ext. 5432 or [email protected]. Part A. Scope and Sequence Please place an “X” in the box by each one of the mathematical concepts listed below that is taught in your classroom. Mathematics Concepts

Number Sense and Numeration Concepts of Whole-Number Computation Number Recognition Counting Fractions (e.g., one half and one fourth) Money Geometry and Spatial Relationships Time Calendar Temperature Linear Measurement Estimation Weight (mass) Capacity (volume)

Data Analysis and Statistics Probability Patterns, Algebra, and Functions (patterning) Other (describe): ______________ Other (describe): ______________


97

Appendix D

Part B. Mathematics Instruction 1. How much time do you devote on a daily basis to teaching mathematics? (Check one)

10 minutes or less 11–15 minutes 16–20 minutes 21–25 minutes 26–30 minutes 31–35 minutes 36 minutes or more

2. During what part of the day do you usually teach mathematics? (Check one)

Beginning of the day Middle of the morning End of the morning (before lunch) Beginning of the afternoon (following lunch) Middle of the afternoon End of the day

3. Instructional practices: Respond to each of the following statements based on your mathematics classroom instruction. (Check one response for each statement.) Instructional Practices

Never

Rarely

Sometimes

Often

Always

Not Applicable



I introduce new concepts with handson activities.

Students practice handwriting in my math class.

Students practice counting in my math class.


Students participate in math center activities.


98

Appendix D


Never

Rarely

Sometimes

Often

Always

Not Applicable




















99

Appendix D


Never

Rarely

Sometimes

Often

Always

Not Applicable



During the course of the lesson, I am able to provide extra assistance to students who need help.


My students use computers in the classroom.

Part C. Assessments 1. What methods of assessment do you use to assess students’ performance? (Select all that apply.)

Oral Written Rubrics Other (describe): ____________________________________________________________

2. On average, how often do you assess students’ mathematics performance? (Check one)

Once a week or more Bimonthly Monthly Quarterly Once a semester or less


100

Appendix D

Part D. Student Engagement and Classroom Management Check one response for each statement. Instructional Practices

Never

Rarely

Sometimes

Often

Always

Not Applicable














101

Appendix D

Part E. Mathematics Resources 1. How much of your mathematics textbook do you anticipate completing by the end of the school year? (Check one)

Less than 50% 51%–75% 76%–99% 100%

2. What is the name of your kindergarten mathematics textbook?_______________________________ 3. Who is the publisher of your kindergarten mathematics textbook? ____________________________ 4. Do you use any other supplemental mathematics materials? If so, what materials do you use? ___________________________________________________________________________________ ___________________________________________________________________________________ Part F. Classroom and Teacher Information 1. How is your kindergarten day structured? (Check one)

Half day Whole day Other (please specify): ______________________________________________________________

2. Do you teach in a self-contained classroom (i.e., are you responsible for teaching all or most academic subjects to a single class)?

Yes No


Yes No

4. How many total students are enrolled in your class (or in each of your classes, if you teach more than 1 section)?________________________________________________________________________


102

Appendix D

5. How would you describe your class in terms of variation in students’ mathematics ability? (Check one)



Male Female


20-30 31-40 41-50 51-60 61 or over

9. Including this year, how many years have you taught full-time in a regular teaching position? Total: In this district: In this school: At this grade level:


103

Appendix D

Comparison Teacher Survey (First Grade) First-Grade Teacher Survey Comparison School Saxon Mathematics Effectiveness Study Instructions: Your school is participating as a comparison site in the Saxon Mathematics Effectiveness Study. We are asking you to complete this survey to help us gather information on mathematics classroom curriculum and instruction in your school. Please be assured that we are not evaluating your teaching. Because your name will never be associated with your answers, please feel free to answer as candidly as possible. Specific instructions appear throughout to guide you through the survey. You will be able to complete the survey in about 20–25 minutes. Once you have completed the survey, please place it in the manila envelope and return the sealed envelope to your site coordinator. The site coordinator will mail the envelopes to Edvantia for data entry. All responses will be kept confidential, and data will only be reported in aggregate format only. You have the right to withdraw from the study, including completion of this survey, at any time. You will not be penalized in any way if you choose to do so. If you have any questions about this survey or about the Saxon Math Program Study, you may contact Kim Good at [email protected] or 800.624.9120 ext. 5449. If you have questions about subjects’ rights, contact Merrill Meehan, Edvantia’s Institutional Review Board Chair, at 800.624.9120 ext. 5432 or [email protected]. Part A. Scope and Sequence Please place an “X” in the box by each one of the mathematical concepts listed below that is taught in your classroom. Mathematics Concepts

Number Sense and Numeration Concepts of Whole-Number Operations Whole-Number Computation Counting Fractions and Decimals Money Geometry and Spatial Relationships Time Calendar Temperature Weather Graphing Linear Measurement Weight (mass) Capacity (volume)

Data Analysis and Statistics Probability Algebra and Functions Patterning Mental Computation Place Value Tallying Problem Solving Other (describe): ________________ Other (describe): ________________


104

Appendix D





3. Instructional Practices: Respond to each of the following statements based on your mathematics classroom instruction. (Check one response for each statement.) Instructional Practices

Never

Rarely

Sometimes

Often

Always

Not Applicable












105

Appendix D


Never

Rarely

Sometimes

Often

Always

Not Applicable





















106

Appendix D

Instructional Practices My students use computers in the classroom.

Never

Rarely

Sometimes

Often

Always

Not Applicable

Part C. Assessments 1. What methods of assessment do you use to assess students’ performance? (Select all that apply.)

Oral Written Rubrics Other (describe): ________________________________________________________________




Never

Rarely

Sometimes

Often

Always

Not Applicable










107

Appendix D


Never

Rarely

Sometimes

Often

Always

Not Applicable






Less than 50% 51%–75% 76%–99% 100%

2. What is the name of your first-grade mathematics textbook?_____________________________________ 3. Who is the publisher of your first-grade mathematics textbook? __________________________________ 4. Do you use any other supplemental mathematics materials? If so, what materials do you use? ________________________________________________________________________________________ ________________________________________________________________________________________ ________________________________________________________________________________________ Part F. Classroom and Teacher Information 1. Do you teach in a self-contained classroom (i.e., are you responsible for teaching all or most academic subjects to a single class)?

Yes No


Yes No

3. How many total students are enrolled in your class (or in each of your classes if you teach more than 1 section)? _____________________________________________________________________________


108

Appendix D

4. How would you describe your class in terms of variation in students’ mathematics ability? (Check one)



Male Female


20–30 31–40 41–50 51–60 61 or over



109

Appendix D

Comparison Teacher Survey (Second Grade) Second-Grade Teacher Survey Comparison School Saxon Mathematics Effectiveness Study Instructions: Your school is participating as a comparison site in the Saxon Mathematics Effectiveness Study. We are asking you to complete this survey to help us gather information on mathematics classroom curriculum and instruction in your school. Please be assured that we are not evaluating your teaching. Because your name will never be associated with your answers, please feel free to answer as candidly as possible. Specific instructions appear throughout to guide you through the survey. You will be able to complete the survey in about 20–25 minutes. Once you have completed the survey, please place it in the manila envelope and return the sealed envelope to your site coordinator. The site coordinator will mail the envelopes to Edvantia for data entry. All responses will be kept confidential, and data will be reported in aggregate format only. You have the right to withdraw from the study, including completion of this survey, at any time. You will not be penalized in any way if you choose to do so. If you have any questions about this survey or about the Saxon Math Program Study, you may contact Kim Good at [email protected] or 800.624.9120 ext. 5449. For questions about subjects’ rights, contact Merrill Meehan, Edvantia’s Institutional Review Board Chair, at 800.624.9120 ext. 5432 or [email protected]. Part A. Scope and Sequence Please place an “X” in the box by each one of the mathematical concepts listed below that is taught in your classroom. Mathematics Concepts

Number Sense and Numeration Concepts of Whole-Number Operations Whole-Number Computation Counting Fractions and Decimals Money Geometry and Spatial Relationships Time Calendar Temperature Graphing Linear Measurement Weight (mass) Capacity (volume) Perimeter, Area, and Volume

Data Analysis and Statistics Probability Algebra and Functions Patterning Mental Computation Place Value Problem Solving Writing of Numbers Fact Families Other (describe): ________________ Other (describe): ________________


110

Appendix D






Never

Rarely

Sometimes

Often

Always

Not Applicable












111

Appendix D


Never

Rarely

Sometimes

Often

Always

Not Applicable





















112

Appendix D


Never

Rarely

Sometimes

Often

Always

Not Applicable

Part C. Assessments 1. What methods of assessments do you use to assess students’ performance? (Select all that apply.)





Never

Rarely

Sometimes

Often

Always

Not Applicable










113

Appendix D


Never

Rarely

Sometimes

Often

Always

Not Applicable






Less than 50% 51%–75% 76%–99% 100%

2. What is the name of your second-grade mathematics textbook? __________________________________ 3. Who is the publisher of your second-grade mathematics textbook?________________________________ 4. Do you use any other supplemental mathematics materials? If so, what materials do you use? ________________________________________________________________________________________ ________________________________________________________________________________________ ________________________________________________________________________________________ Part F. Classroom and Teacher Information 1. Do you teach in a self-contained classroom (i.e., are you responsible for teaching all or most academic subjects to a single class)?

Yes No


Yes No

3. How many total students are enrolled in your class (or in each of your classes, if you teach more than 1 section)? _____________________________________________________________________________ 4. How would you describe your class in terms of variation in students’ mathematics ability? (check one)


114

Appendix D



Male Female


20–30 31–40 41–50 51–60 61 or over



115

Appendix D

Comparison Teacher Survey (Third Grade) Third-Grade Teacher Survey Comparison School Saxon Mathematics Effectiveness Study Instructions: Your school is participating as a comparison site in the Saxon Mathematics Effectiveness Study. We are asking you to complete this survey to help us gather information on mathematics classroom curriculum and instruction in your school. Please be assured that we are not evaluating your teaching. Because your name will never be associated with your answers, please feel free to answer as candidly as possible. Specific instructions appear throughout to guide you through the survey. You will be able to complete the survey in about 20–25 minutes. Once you have completed the survey, please place it in the manila envelope and return the sealed envelope to your site coordinator. The site coordinator will mail the envelopes to Edvantia for data entry. All responses will be kept confidential, and data will be reported in aggregate format only. You have the right to withdraw from the study, including completion of this survey, at any time. You will not be penalized in any way if you choose to do so. If you have any questions about this survey or about the Saxon Math Program Study, you may contact Kim Good at [email protected] or 800.624.9120 ext. 5449. For questions about subjects’ rights, contact Merrill Meehan, Edvantia’s Institutional Review Board Chair, at 800.624.9120 ext. 5432 or [email protected]. Part A. Scope and Sequence Please place an “X” in the box by each one of the mathematical concepts listed below that is taught in your classroom. Mathematics Concepts

Number Sense and Numeration Concepts of Whole-Number Operations Whole-Number Computation Counting Fractions and Decimals Money Geometry and Spatial Relationships Time Calendar Temperature Graphing Linear Measurement Weight (mass) Capacity (volume) Perimeter, Area, and Volume

Data Analysis and Statistics Probability Algebra and Functions Patterning Mental Computation Problem Solving Writing about Math Other (describe): ________________ Other (describe): ________________


116

Appendix D






Never

Rarely

Sometimes

Often

Always

Not Applicable












117

Appendix D


Never

Rarely

Sometimes

Often

Always

Not Applicable





















118

Appendix D


Never

Rarely

Sometimes

Often

Always

Not Applicable

Part C. Assessments 1. What methods of assessments do you use to assess students’ performance? (Select all that apply.)





Never

Rarely

Sometimes

Often

Always

Not Applicable










119

Appendix D


Never

Rarely

Sometimes

Often

Always

Not Applicable






Less than 50% 51%–75% 76%–99% 100%

2. What is the name of your third-grade mathematics textbook? ____________________________________ 3. Who is the publisher of your third-grade mathematics textbook? _________________________________ 4. Do you use any other supplemental mathematics materials? If so, what materials do you use? ________________________________________________________________________________________ ________________________________________________________________________________________ ________________________________________________________________________________________ Part F. Classroom and Teacher Information 1. Do you teach in a self-contained classroom (i.e., are you responsible for teaching all or most academic subjects to a single class)?

Yes No


Yes No

3. How many total students are enrolled in your class (or in each of your classes, if you teach more than 1 section)? _____________________________________________________________________________ 4. How would you describe your class in terms of variation in students’ mathematics ability? (Check one)


120

Appendix D



121

Appendix D


Male Female


20–30 31–40 41–50 51–60 61 or over



122


Appendix E Lessons Learned

Appendix E

Lessons Learned Several lessons were learned from the conduct of this study. Institutionalization of these lessons in future studies should strengthen project processes, data-collection efforts, and ultimately, findings and conclusions. Highlights of these lessons are bulleted below.

Recruitment of schools should be completed prior to the conclusion of the previous school year. Attempts were made to recruit participating schools in May, which was prior to the end of the following school year, at which time the study was to begin. However, due to lack of response by potential participating respondents, it was necessary to repeatedly random sample from the population and contact schools, inviting them to participate in the study. Because of this, a great deal of recruitment took place throughout the summer. With vacation schedules and summer school closures, this was not an ideal time to contact principals and invite their school’s participation in a study.

Initial recruitment efforts involved sending a letter to the principals of schools that were randomly selected to participate in the study. An alternative approach might be to make an initial call to the principal, followed immediately with faxed letter of invitation detailing purpose and requirements.

The principals of participating schools signed a memorandum of understanding that explained the purposes and requirements of the study and what they agreed to by having their schools commit to being a part of the study. This commitment helped to ensure that schools followed through with all study requirements. Likewise, the site coordinators and teachers signed a consent form. Not only did the consent form indicate their rights as a participant in the study, but it also detailed all expectations of the study.

Monetary incentives (e.g., stipends) given to the site coordinators and teachers following fulfillment of study requirements guaranteed a high response rate and that all data requested was sent to Edvantia.

Having a site coordinator at each school was beneficial for logistical reasons. The site coordinator was the communication liaison between Edvantia and the participating teachers. This individual served as central point of contact (e.g., for scheduling interviews, distributing, gathering, and submitting data).

There is a delicate balance between waiting as long as possible to have the posttests administered in each of the schools and have the Saxon Math studentassessment data collected and needing a report that Harcourt Achieve wanted to have by August for decision-making purposes. Approximately one half of the schools did not conclude school until June, and two schools were not finished until later. In order to obtain valid posttest assessment results, it was desired that schools administer tests as close to the conclusion of the school year as possible. This created a tension in getting data back to Edvantia, scored and analyzed and


2

Appendix E

included in a final report. As a compromise, the 30 schools that had submitted their data to Edvantia by June 2 were included in the interim report.

The Saxon Math curriculum is implemented at a different pace in each classroom and school. Participating teachers selected the Saxon assessment they were on at the mid- and end-of-year and submitted results for participating students for those two points. Consequently, when aggregating the data from assessments within a grade level for the mid- and end-of-year, results are based on a variety of different assessments. This may call into question the comparability of the data. On the other hand, assessment results are converted to percentages based on a total of 100%.

The participating teachers selected the lesson number they had videotaped and submitted to Saxon. A suggestion for improving this process would be to have all teachers in the same grade level video the same lesson so the reviewer rates the same lesson, which may enhance the reliability of reviewer ratings.


3

©2007 Harcourt Education. All rights reserved. 2919/10M/NY/SPECTRA/10-07 9994059653

www.saxonpublishers.com 1.800.531.5015

Saxon Elementary Math Program Effectiveness Study

Saxon Elementary Math Program Effectiveness Study

Suggest Documents

Saxon Math

Saxon Math Paper

Download Saxon Math, Intermediate 4 - Google Sites

MATH 119: ELEMENTARY STATISTICS

Extension Projects - Elementary Math

Saxon Math - Eric - U.S. Department of Education

[PDF] Saxon Math, Intermediate 4 - Google Sites

Saxon Math K-4 Kit Components - Nelson Math

Advanced Math Advanced Math, 2nd Edition, Saxon Solutions ...

Saxon Math - Eric - U.S. Department of Education

Saxon Math Order Form (2011–2012)

PDF Saxon Math K - Google Sites

Saxon Math Wk 15 7_6

Saxon Math Intermediate 3 - Barren County Schools

Saxon Math 6/5 Third Edition

Evaluating Program Effectiveness for an Online Elementary Education ...

Evaluating Program Effectiveness for an Online Elementary Education ...

Math 115: Elementary Algebra Syllabus

Elementary Math Presentation.pdf - Google Drive

[PDF] Saxon Math 7/6 Homeschool: Saxon Teacher CD ... - Google Sites

TenMarks Summer Math Program

Great Books for the Elementary Math - Sullivan's Island Elementary ...

Saxon Math Price List (2013-2014) Oxford University Press (Prices ...

PDF Saxon Math 1: An Incremental Development, Part ... - Google Sites