The Clearing House: A Journal of Educational Strategies, Issues and Ideas
ISSN: 0009-8655 (Print) 1939-912X (Online) Journal homepage: http://www.tandfonline.com/loi/vtch20
Identifying Evidence-Based Practices for Behavior: Analysis of Studies Reviewed by the What Works Clearinghouse Adam Scott LeRoy To cite this article: Adam Scott LeRoy (2017) Identifying Evidence-Based Practices for Behavior: Analysis of Studies Reviewed by the What Works Clearinghouse, The Clearing House: A Journal of Educational Strategies, Issues and Ideas, 90:4, 125-128, DOI: 10.1080/00098655.2017.1318026 To link to this article: https://doi.org/10.1080/00098655.2017.1318026
Published online: 16 May 2017.
Submit your article to this journal
Article views: 171
View related articles
View Crossmark data
Full Terms & Conditions of access and use can be found at http://www.tandfonline.com/action/journalInformation?journalCode=vtch20
THE CLEARING HOUSE , VOL. , NO. , – https://doi.org/./..
Identifying Evidence-Based Practices for Behavior: Analysis of Studies Reviewed by the What Works Clearinghouse Adam Scott LeRoy School of Education and Human Services, Oakland University, Rochester, MI, USA
ABSTRACT
KEYWORDS
Prior concerns have been raised about the ability of schools to access evidence-based practices, however, these practices are instrumental for addressing behavior concerns. This is particularly true at the secondary level, where students are more likely to be disproportionately identified for school removal. This review investigates studies of behavior intervention programs reviewed by the What Works Clearinghouse (WWC), founded by the US Department of Education’s Institute of Education Sciences. Three studies meet the WWC evidence standards for middle or high school students. The limited number of studies meeting evidence standards and the demographic samples of those studies meeting evidence standards lowers access to robust practice.
Response to intervention; student behavior; achievement gap; discipline; disproportionality; evidence-based practice; research-based practice; what works clearinghouse
Introduction Implementing evidence-based practices (those practices that through experiment consistently result in positive effects) has a profound impact on behavior. However, a disconnect between classroom practice and evidence-based practices is an oft-remarked concern— particularly in methodologies for behavior and discipline (Lewis et al. 2015; Webster-Stratton, Reid, and Hammond 2004). Teachers and administrators may implement practices wholly unsupported by research or implement practices in a way that does not align with the intended scope (Cook, Cook, and Landrum 2013; Englehart 2011). In both cases, the potential for positive effect is blunted. Stakeholder skepticism toward the reliability of research and, thereby, execution is one barrier of implementation for evidence-based practices (Cook and Odom 2013; Torres, Farley, and Cook 2014). Potentially, implementation increases if teachers and administrators can access a reliable screening of research. The What Works Clearinghouse (WWC) serves as a resource for screening evidence-based practices that include intensive intervention for a student with persistent behavior problems. Established in 2002 by the US Department of Education’s Institute of Education Sciences, the WWC reviews research on programs, CONTACT Adam Scott LeRoy Rochester, MI –, USA. © Taylor & Francis Group, LLC
[email protected]
products, and practices in education. After review, the WWC assigns one of three ratings: meets evidence standards without reservations, meets evidence standards with reservations, or does not meet evidence standards (US Department of Education, IES, WWC n.d.). By analyzing these reports, school-based decision makers can select appropriate research-based interventions. For example, a curriculum director can review an intervention report to ensure the accuracy of claims about a specific program or analyze valid practices. Since 2006, the WWC has reviewed 56 interventions designed to have a positive effect on behavior. Twentyeight programs did not meet evidence standards and they were not reviewed. Five program-based interventions (Building Decision Skills, Connect with Kids, Facing History and Ourselves, Skills for Action, and Skills for Adolescence) were reviewed for effects on behavior, knowledge, attitudes, and values in middle or high school students (US Department of Education, IES, WWC, 2006a–e). Building Decision Skills had potentially positive effects (d = 0.84) for knowledge, attitudes and values. Connect with Kids had potentially positive effects (d = 0.61, d = 0.48) for behavior. Facing History and Ourselves and Skills for Action did not have a discernable effect (d = 0.11, d = 0.04). Skills for Adolescence had a potentially positive but non-significant
School of Education and Human Services, Oakland University, G Pawley Hall, Pioneer Drive,
126
A. S. LEROY
effect (d = 0.13). Daily report cards—a non-program based intervention—had a positive effect for external behaviors (d = 0.35) but positive effects for math (d = 0.08) or reading (d = 0.02) achievement were not found. These programs can be implemented as targeted behavior interventions; however, the limited number of studies meeting evidence standards indicates both a lack of access to behavior intervention for secondary students and the need for relevant and reliable research. Overview of Selected Programs Behavior
Connect with Kids is designed to affect attitudes and encourage positive behavior. The middle and high school program contains 36 lessons. Each lesson introduces a trait followed by three video segments for the particular trait. A discussion follows the video and a follow-up activity on the trait occurs one to two weeks later (US Department of Education, IES, WWC 2006b). Only one study met WWC standards. Page and D’Agostino (2005) conducted a quasi-experimental study involving elementary, middle, and high school students (n = 800) in Kansas and Missouri. Students and teachers completed surveys designed to measure perceptions of positive and negative classroom behavior (honesty, kindness, etc.). Though intended to be a school-wide program, the study only implemented Connect with Kids within individual classrooms. For students in middle and high school, a statistically significant effect (d = 0.61, d = 0.48) was found for behavior (Page and D’Agostino 2005; US Department of Education, IES, WWC 2006b). Implementation Connect With Kids’ reliance on technology provides positive effects for engagement, parent education, and a reduction in barriers to technology access (Reynolds and Chui 2016). However, the availability of appropriate technology and the time commitment required by a program comprised of 36 lessons and multiple video segments limits implementation in all environments. It is also important to note that the study reviewed by WWC measured perceptions of behavior and not actual incidences of behavior that resulted in referral for discipline (US Department of Education, IES, WWC 2006b).
Knowledge, Attitudes, and Values
Building Decision Skills consists of 10-lessons designed to increase the ethical awareness and development of core values for middle and high school students. Lessons occur over two weeks and involve readings, small-group activities, discussions, and homework assignments. For example, one passage on ethical dilemmas concludes with a discussion on resolving ethical dilemmas. WWC found one study eligible for review. Leming (2001) conducted a quasi-experimental study using the Building Decision Skills curriculum in an elective course at a suburban high school in St Louis (n = 283). Seventy-five percent of participants were female. Participants also completed community service activities (e.g. retirement home outreach or setting up a recycling center) four days a week during the semester. The WWC confirmed a statistically significant effect (d = 0.84) for ethical perspective. Statistically significant effects were not reported for six other domains: ethical awareness, ethical responsibility, self-esteem, social responsibility (school), social responsibility (general), and anticipated community participation (US Department of Education, IES, WWC 2006a). Implementation As there are few lessons, Building Decision Skills is ideal as a short-term intervention. An instructor utilizing the program can support a concept from a specific unit for a small group of students identified as at risk. One consideration rests with ensuring the fidelity of implementation. For example, the community service component of the program could create a substantial burden for a student—particularly at-risk students. The likelihood of a positive effect for behavior domains lessens if a program is not implemented as conducted (Cook 2014).
External Behavior
Daily Report Cards (DRCs) track student behavior throughout the day. At regular intervals a student meets with a teacher and receives a behavior rating for a particular time period. For example, one interval may be “lunch/recess” and a student receives one to five points for meeting certain behavior targets. Staff alter the rating system and targets to ensure individualization (Fabiano et al. 2010; US Department of Education, IES, WWC 2012).
THE CLEARING HOUSE
WWC reviewed one study by Fabiano et al. (2010) that assessed DRCs for students in Grades 1 through 6 (n = 63) with both an ADHD diagnosis and an Individualized Education Program. Participants were predominantly White (79%) and male (86%). A school psychology graduate student met with each family and teacher to review performance and goal setting. DRCs were sent home and parents were asked to revoke or provide privileges based on performance. Ratings scales for on-task behavior, ADHD/ODD symptoms, and rule violations indicated a statistically significant effect (d = 0.35) (US Department of Education, IES, WWC 2012). Implementation DRCs present a targeted intervention for specific students with frequent behavior referrals. Unfortunately, the study reviewed by the WWC only peripherally includes grades associated with middle school and this is the only DRC study reviewed by WWC. DRC research not reviewed by WWC indicates some positive students beyond Grade 6 (Chafouleas, Riley-Tillman, and Sassu 2006; Moore, Whittaker, and Ford 2016). A more robust review of DRC research by WWC can provide greater support for implementation in secondary education. Although a low-monetary cost, DRCs require individualization and a staff member capable of completing several “check-ins” during the school day. Ideally, applying DRCs over a short period of time and prior to more formalized behavior plans minimizes this tax on resources.
Discussion Taken as a whole, this review serves as a barometer on current scholarship related to behavior intervention at the middle and high school level. Notably, very few studies were able to meet WWC evidence standards. In the review of Skills for Adolescence, of nine studies reviewed only one met WWC evidence standards—and it was not possible for the WWC to confirm a statistically significant effect (US Department of Education, IES, WWC, 2006e). Similarly, seven studies involving Facing History and Ourselves did not meet evidence standards. Although one study did meet standards, a statistically significant effect for behavior was not confirmed. However, according to intervention reports, both programs are often employed by schools to support character education (US Department of Education, IES, WWC 2006c).
127
One specific limitation of this review is the scope offered by only focusing on studies reviewed by the WWC. Although failure to meet WWC evidence standards does not directly impugn findings, the failure of multiple studies to meet standards is informative. For example, some comments within the WWC intervention reports indicate failure to use valid outcome measures or a comparison group. Such flaws in study design have ramifications for overall validity and appropriate analysis (Cook 2014). Educator access to robust, evidence-based practice is impaired. Finally, the limited number of studies that did meet evidence standards suggest fewer options for persistent negative behaviors at the middle and high school level. However, the need for effective, alternate discipline options are most paramount at the secondary level. This point underscores the contentions made by Cook and Odom (2013) when outlining the disconnect between classroom practice and practice advocated by research. For example, a national focus has been raised on the disproportionate discipline of African-American males (DeMatthews 2016; Monroe 2005). However, of those studies currently reviewed by the WWC none reflect that population. It also bears noting, that none of the studies presented above focused entirely on students that persistently display behavior problems leading to removal from school. This indicates both a need in scholarship or in the types of scholarship currently collected by the WWC for review. Still, these reviews offer some research support for staff seeking to implement interventions or analyze barriers such as time commitment, allocation of resources, and familiarity with intervention (Cook 2014).
Conclusion There is frequent agreement on the need for evidencebased practices in education (Cook and Odom 2013). However, implementation depends on access. Equally, it is imperative to consider the validity of research prior to selection (Cook 2014). This review finds both access and validity wanting. The WWC is one resource to consult when considering validity. Future research reviewed by WWC will benefit by incorporating participants more likely identified as at-risk (DeMatthews 2016; Monroe 2005).
128
A. S. LEROY
Acknowledgement The author thanks the staff at Oakland University, particularly those in the Office of Public School Academies, for their support and motivation.
References Chafouleas, S. M., T. C. Riley-Tillman, and K. A. Sassu. 2006. Acceptability and reported use of daily behavior report cards among teachers. Journal of Positive Behavior Interventions 8 (3): 174–82. doi:10.1177/10983007060080030601 Cook, B. G. 2014. A call for examining replication and bias in special education research. Remedial and Special Education 35 (4): 233–46. Cook, B., L. Cook, and T. Landrum. 2013. Moving research into practice: Can we make dissemination stick? Exceptional Children 79 (2): 163–80. Cook, B. G., and S. Odom. 2013. Evidence-based practices and implementation science in special education. Teaching Exceptional Children 79 (2): 135–44. doi:10.1177/001440291307900201 DeMatthews, D. 2016. Effective leadership is not enough: Critical approaches to closing the racial discipline gap. The Clearing House: A Journal of Educational Strategies, Issues and Ideas 89 (1): 7–13. doi:10.1080/00098655.2015.1121120 Englehart, J. M. 2011. Five half-truths about classroom management. The Clearing House: A Journal of Educational Strategies, Issues and Ideas 85 (2): 70–3. doi:10.1080/00098655.2011.616919 Fabiano, G. A., R. K. Vujnovic, W. E. Pelham, D. A. Waschbusch, G. M. Massetti, M. E. Pariseau, and J. Naylor. 2010. Enhancing the effectiveness of special education programming for children with attention deficit hyperactivity disorder using a daily report card. School Psychology Review 39 (2): 219–39. Leming, J. S. 2001. Integrating a structured ethical reflection curriculum into high school community service experiences: Impact on students’ sociomoral development. Adolescence, 36: 33–45. Lewis, T. J., B. S. Mitchell, D. T. Bruntmeyer, and G. Sugai. 2015. School-wide positive behavior support and response to intervention: System similarities, distinctions, and research to date at the universal level of support. Handbook of Response to Intervention, 703–17. doi:10.1007/978-1-48997568-3_40 Monroe, C. R. 2005. Why are “bad boys” always Black?: Causes of disproportionality in school discipline and recommendations for change. The Clearing House: A Journal of Educational Strategies, Issues and Ideas 79 (1): 45–50. doi:10.3200/tchs.79.1.45-50
Moore, D. A., S. Whittaker, and T. J. Ford. 2016. Daily report cards as a school-based intervention for children with attention-deficit/hyperactivity disorder. Support for Learning 31: 71–83. doi:10.1111/1467-9604.12115 Page, B. and D’Agostino A. 2005. Connect with Kids: 2004– 2005 Study Results for Kansas and Missouri. Available from: Compass Consulting Group, LLC. 5726 Fayetteville Road, Suite 203, Durham, NC 27713. Reynolds, R. and M. M. Chiu. 2016. Reducing digital divide effects through student engagement in coordinated game design, online resource use, and social computing activities in school. J Assn Inf Sci Tec 67: 1822–35. doi:10.1002/asi.23504 Torres, C., C. A. Farley, and B. G. Cook. 2014. A special educator’s guide to successfully implementing evidencebased practices. Teaching Exceptional Children 47 (2): 85– 93. doi:10.1177/0040059914553209 US Department of Education, Institute of Education Sciences, What Works Clearinghouse (IES, WWC). n.d.. https:// ies.ed.gov/ncee/wwc/ US Department of Education, Institute of Education Sciences, What Works Clearinghouse (IES, WWC). 2006a. Building Decision Skills. https://ies.ed.gov/ncee/wwc/Docs/InterventionReports/WWC_Building_Decision_090806.pdf US Department of Education, Institute of Education Sciences, What Works Clearinghouse (IES, WWC). 2006b. Connect with Kids. https://ies.ed.gov/ncee/wwc/Docs/InterventionReports/WWC_Connect_Kids_091406.pdf US Department of Education, Institute of Education Sciences, What Works Clearinghouse (IES, WWC). 2006c. Facing History and Ourselves. https://ies.ed.gov/ncee/wwc/Docs/ InterventionReports/WWC_Facing_History_090806.pdf US Department of Education, Institute of Education Sciences, What Works Clearinghouse (IES, WWC). 2006d. Skills for Action. https://ies.ed.gov/ncee/wwc/EvidenceSnapshot/283 US Department of Education, Institute of Education Sciences, What Works Clearinghouse (IES, WWC). 2006e. Skills for Adolescence. https://ies.ed.gov/ncee/wwc/Docs/InterventionReports/WWC_Skills_Adolescent_092806.pdf US Department of Education, Institute of Education Sciences, What Works Clearinghouse (IES, WWC). 2012. WWC Review of the Report “Enhancing the Effectiveness of Special Education Programming for Children with Attention Deficit Hyperactivity Disorder Using a Daily Report Card”. https:// ies.ed.gov/ncee/wwc/Docs/SingleStudyReviews/wwc_daily reportcards_061212.pdf Webster-Stratton, C., M. J. Reid, and M. Hammond. 2004. Treating children with early-onset conduct problems: Intervention outcomes for parent, child, and teacher training. Journal of Clinical Child and Adolescent Psychology 33 (1): 105–24.
