2010 SREE Conference Abstract Template - ERIC - US Department of ...

Implementing Student-Level Random Assignment During Summer School: Lessons Learned from an Efficacy Study of Online Algebra I for Credit Recovery

Author(s): Jessica Heppen, Elaine Allensworth, Kirk Walters, Amber Stitziel Pareja, Anja Kurki, Takako Nomi and Nicholas Sorensen.

SREE Fall 2011 Conference Abstract Template

Background / Context: The consequences of failing core academic courses during the first year of high school are dire. Research by the Consortium on Chicago School Research (CCSR) demonstrates that students who fall ―off track‖ during the first year of high school have a substantially lower probability of graduating than students who stay ―on track.‖ In the Chicago Public Schools (CPS), only about one-fifth of off-track freshmen—students who fail more than one semester of a core academic course and/or fail to earn enough credits to be promoted to 10th grade—graduate high school, compared with over 80% of on-track freshmen (Allensworth & Easton, 2005, 2007). Failure of Algebra I is particularly problematic. In CPS, only 13% of students who fail both semesters of Algebra I in 9th grade graduate in 4 years, and the largest share of 9th grade algebra failures occur in the second semester of the course. Elucidating the ways that students can get back on track is of the utmost policy importance. Credit recovery is one strategy to deal with high failure rates. The primary goal of credit recovery programs is to give students an opportunity to retake classes that they failed in an effort to get them back on track and keep them in school (Watson & Gemin, 2008). Most recently, as schools across the nation struggle to keep students on track and re-engage students who are off track, online learning has emerged as a promising and increasingly popular strategy for credit recovery: more than half of respondents from a national survey of administrators from 2,500 school districts reported using online learning in their schools for credit recovery, with just over a fifth (22%) reporting ―wide use‖ of online learning for this purpose (Greaves & Hayes, 2008). Despite the growing use of online courses for credit recovery, the evidence base is thin. This paper describes the design and initial implementation of a randomized control trial that was designed to strengthen the evidence base surrounding online courses used for credit recovery. They study is testing the (1) the impact of online Algebra I for credit recovery against the standard face-to-face (f2f) version of the course and (2) the effects of offering expanded credit recovery options with online algebra, relative to business as usual (i.e., the summer programming that schools would offer in the absence of efforts to expand credit recovery). Purpose / Objective / Research Question / Focus of Study: The primary goals and research questions of the study are: 1) To test the efficacy of online Algebra I for credit recovery, compared with standard f2f Algebra I for credit recovery. Through a within-school, randomized trial in 20 schools with freshman Algebra I failure rates above 20%, we will directly compare online and f2f Algebra I in a head-to-head trial. This comparison will allow us to answer these specific research questions: 1. What is the impact of taking online Algebra I for credit recovery on at-risk 9th graders’ short-term academic outcomes (score on end-of-Algebra I course assessment, credit attainment, mathematics test scores) and long-term outcomes (geometry and chemistry grades; ACT mathematics score; total mathematics and science credits; likelihood of dropout in the 2nd, 3rd, or 4th year of high school; 4-year graduation), compared with retaking Algebra I as a standard f2f summer course? 2. How does the instructional experience (engagement, classroom personalism, academic demand, self-efficacy in mathematics) compare for students taking online and f2f summer credit recovery courses?


1

3. Is online credit recovery differentially beneficial for students with different levels of risk for failure (e.g., different failure rates across all courses, different 9th grade GPAs, different incoming skills as measured on prior tests)? 2) To determine the supporting conditions in the classroom under which online Algebra I for credit recovery yields higher efficacy. This study will identify specific contributing factors to program effectiveness (e.g., mentor practices, online teacher interactions): 4. Under what supporting conditions in the classroom does the implementation of online Algebra I for summer credit recovery produce the strongest treatment effects, relative to f2f Algebra I for summer credit recovery? 3) To gauge the extent to which credit recovery can help at-risk students get back on track, relative to students who passed Algebra I in 9th grade. This question will provide critical information about whether summer credit recovery options can help close the gap between students who failed Algebra I in 9th grade and students who passed: 5. To what extent do students who take online Algebra I and f2f Algebra I for credit recovery “close the gap” with students who passed Algebra I in 9th grade, in terms of short-term outcomes (number of credits, algebra skills) and long-term outcomes (dropout rates over the next 3 years of high school, grades in geometry and chemistry)? 4) To gauge the effects of expanding summer credit recovery through online courses. This aim is to assess the effects of expanding credit recovery through expanded online opportunities, relative to typical summer credit recovery opportunities: 6. To what extent does expanded summer credit recovery through online classes lead to improved academic outcomes (progress in the mathematics sequence, staying in school, improvements in mathematics scores), compared with the outcomes of students in schools with limited access to summer credit recovery? 7. Are the benefits of expanded summer credit recovery options different for students with different levels of risk for failure (e.g., different failure rates across all courses, different 9th grade GPAs, different incoming skills as measured on prior tests)? Setting: The setting will be CPS high schools with freshman Algebra I failure rates of 20% or higher. CPS is the third-largest U.S. district, serving more than 435,000 students in 666 schools, of which 116 are public high schools and 27 are public charter high schools. School reform and improvement have been high priorities in Chicago for a number of years, as high schools in CPS continue to struggle with low student performance and low graduation rates (Kahne, Sporte, de la Torre, & Easton, 2006). The overall graduation rate in the district is just 54%. The average composite ACT score for CPS juniors is 17, lower than the 20.5 for juniors in the state of Illinois and well below the score required by most colleges (Allensworth, Correa, & Ponisciak, 2008). Population / Participants / Subjects: The target students for this study are first-time freshmen who failed Algebra IB but passed the first semester. In CPS schools with freshman failure rates above 20%, the average number of 9th graders is 275, and 35% of them fail Algebra IB. We estimate that approximately 25–30% of first-time freshmen fail Algebra IB after passing IA (70–83 students per school). We will target SREE Fall 2011 Conference Abstract Template

2

these students for recruitment. Our goal is to enroll, on average, 40 students per school in summer credit recovery for Algebra IB. Intervention / Program / Practice: A number of providers offer online courses for credit recovery, including commercial providers and courses offered as part of district or state virtual schools programs (Archambault et al., 2010). The Department of Graduation Pathways at CPS began to pilot online credit recovery options in summer 2008. In the pilot, the district compared two vendors that offered online courses for the express purpose of credit recovery. Twenty-one schools participated in the pilot. More than 700 students initially registered for an online course in a variety of subjects, and 365 students attended the full session. Based on review of the rates of successful course completion and surveys of mentors and students, CPS discerned that one of the two course providers appeared to better serve at-risk students. Specifically, CPS observed that the success (i.e., completion) rate for Aventa online summer courses was 72% (vs. 43% for the other provider, Apex Learning) and that the Aventa teachers appeared to have been better trained to work with at-risk students. The district, mentors, and students noted that the Aventa teachers were more communicative, responded more quickly, and gave more support than the teachers in the other online course. Since the 2008 pilot, Aventa has been providing credit recovery courses to an increasing number of schools and students in CPS. Research Design: The proposed Goal 3 trial of online Algebra I for summer credit recovery will rely on withinschool randomization of struggling 9th graders to condition: online Algebra I (treatment) and standard f2f Algebra I (control). We have conducted detailed power calculations to justify the size and scope of this student-level random assignment design. The within-school basic design will involve 20 public high schools in CPS. We will select the schools from a sampling pool of high schools that have freshman Algebra I failure rates above 20%, based on the previous year’s district records, and that do not have existing expanded summer credit recovery programs in place (i.e., schools that do not have external funding to increase the number of Algebra I course offerings to increase enrollments of students in need of credits). We will focus on these schools because they are in the greatest need of improved options for credit recovery and because the high numbers of freshmen who fail the spring semester of Algebra I will ensure a sufficient number of students in the two conditions to test our central hypotheses. Within each school, we will randomly assign students who (a) failed second semester Algebra I, (b) are willing to enroll in summer credit recovery, and (c) are willing to take an online course if available. Our focus is on students who failed the second semester only and are positioned to get back on track in mathematics if they recover the ½ credit in the summer after 9th grade. Our estimates of the number of students per class are based on typical enrollment patterns in CPS during the 2008 and 2009 summer sessions, with an assumed additional 20% enrollment from usual due to a concerted push to motivate students to attempt summer credit recovery in the schools participating in the study. For each of the two summers of implementation (2011 and 2012), we anticipate that we will randomly assign an average of 40 students per school into online and f2f classes of 20 students each, for a total of 800 students per cohort. Minimum detectable effect sizes (MDESs) for impacts on student achievement and other outcomes range from 0.14 to 0.19, for analyses conducted separately by cohort. We believe these


3

MDESs are reasonable given our theory of action that proposes that the online course is more engaging for at-risk students, who are thus more likely to persist and complete than students who retake the traditional course in which they were previously unsuccessful. These MDESs are also reasonable based on previous research that finds the average effect of online learning relative to f2f is about 0.24 standard deviations (U.S. Department of Education, 2009). This within-school, student-level randomized design establishes the core comparison that allows us to answer Research Questions 1–4, delineated under Aim 1 and Aim 2. Data Collection and Analysis: To fully capture the extent to which participation in credit recovery courses affects the future performance of students in mathematics, we will collect both achievement and course-taking data for both cohorts of high school students. We will also collect annual data on students’ dropout/persistence status and, for Cohort 1, graduation status. Of great benefit to this study, nearly all of the student outcomes are available for all CPS students; therefore, we will have fully intact ITT samples with little missing data. We will also be able to readily obtain, without going directly to non-participating schools, all the data required for the secondary comparisons (i.e., to on-track and other off-track students with limited access to summer credit recovery). Given the nested data structure (i.e., students nested within schools), the primary analytic method for this study will be hierarchical linear modeling (HLM; Raudenbush & Bryk, 2002). By explicitly taking into account the dependence or clustering among individuals nested within the same higher-level units, the HLM method generates more-accurate standard errors for parameter estimates and enables researchers to address questions that cannot be addressed with the traditional regression model (e.g., heterogeneity of regression slopes). For all analyses, we will include the full ITT sample of online and f2f students. We will also conduct exploratory TOT analyses to supplement the ITT findings. Further, we will verify the findings from the HLM models through fixed-effects models that remove all school-specific variation. Findings / Results: The study is first being implemented in summer 2011, and we will describe the study design and report on the implementation of the first summer cohort, including challenges and lesson learned from expanding access to credit recovery courses for at-risk students and conducting random assignment ―on the spot‖ as students show up to take summer classes. The paper will also describe methods for measuring student participation and engagement in online courses, including interactions with online teachers, online students and in-class mentors. Conclusions: Most of the conclusions will be based on the implementation of the first cohort of students in summer 2011. However, the study team has already identified and addressed a number of design challenges associated with student-level random assignment that are likely to be highly relevant to researchers attending the SREE conference. For example, if a study design calls for on-site random assignment that involves stratification and occurs in a large number of schools, what are the best processes to ensure that the random assignment will be done correctly and efficiently? This issue specifically and other issues surrounding student-level random assignment will be presented.


4

Appendix A. References Allensworth, E., & Easton, J. (2005). The on-track indicator as a predictor of high school graduation. Chicago: Consortium on Chicago School Research. Allensworth, E., & Easton, J. Q. (2007). What matters for staying on-track and graduating in Chicago Public High Schools: A close look at course grades, failures and attendance in the freshman year. Chicago: Consortium on Chicago School Research. Allensworth, E., Correa, M., & Ponisciak, S. (2008). From high school to the future: ACT Preparation–Too much, too late. Why ACT scores are low in Chicago and what it means for schools. Chicago: Consortium on Chicago School Research. Archambault, L., Diamond, D., Coffey, M., Foures-Aalbu, D., Richardson, J., Zygouris-Coe, V., Brown, R., & Cavanaugh, C. (2010). INACOL Research Committee Issues Brief: An exploration of at-risk learners and online education. Vienna, VA: International Association for K–12 Online Learning. Greaves, T. W., & Hayes, J. (2008). America’s digital schools 2008: Six trends to watch. Encinitas, CA: The Greaves Group. Kahne, J. E., Sporte, S. E., de la Torre, M., & Easton, J. Q. (2006). Small high schools on a larger scale: The first three years of the Chicago High School Redesign Initiative. Chicago: Consortium on Chicago School Research. Raudenbush, S. W., & Bryk, A. S. (2002). Hierarchical linear models: Applications and data analysis methods (2nd ed.). Thousand Oaks, CA: Sage. U.S. Department of Education, Office of Planning, Evaluation, and Policy Development. (2009). Evaluation of evidence-based practices in online learning: A meta-analysis and review of online learning studies, Washington, DC: Author. Watson, J., & Gemin, B. (2008, June). Promising practices in online learning: Using online learning for at-risk students and credit recovery. Vienna, VA: North American Council For Online Learning. Retrieved September 14, 2010, from http://www.inacol.org/research/promisingpractices/NACOL_CreditRecovery_Promising Practices.pdf.


B-1