final report revised 081204

Telling the Story: The Results of the Evaluation of Richland School District One After-School Programs June 30, 2004 .

ACKNOWLEDGEMENTS This project was supported through a grant from the John S. and James L. Knight Foundation awarded to the University of South Carolina Research Foundation (Grant 2003-0242).

PRIMARY AUTHORS

CONTRIBUTING AUTHORS

Sara J. Corwin, M.P.H., Ph.D. Research Assistant Professor Office of Public Health Practice Arnold School of Public Health University of South Carolina Columbia, SC

Donna L. Richter, Ed.D., F.A.A.H.B. Interim Dean & Associate Dean for Public Health Practice Arnold School of Public Health University of South Carolina Columbia, SC

Willie H. Oglesby, M.S.P.H., Ph.D.(c) Research Associate & Associate Director for Technical Assistance & Program Evaluation Office of Public Health Practice Arnold School of Public Health University of South Carolina Columbia, SC Kara Montgomery, Dr.P.H. Research Associate Office of Public Health Practice Arnold School of Public Health University of South Carolina Columbia, SC

Ruth Saunders, Ph.D. Associate Professor Department of Health Promotion, Education, & Behavior Arnold School of Public Health University of South Carolina Columbia, SC Jason Coleman Graduate Student Department of Health Promotion, Education, & Behavior Arnold School of Public Health University of South Carolina Columbia, SC

Kim Nichols Dauner, M.P.H. Evaluation Manager Institute for HIV Prevention Leadership Office of Public Health Practice Arnold School of Public Health University of South Carolina Columbia, SC

The authors wish to acknowledge the support and input from all of the project partners and school personnel in the preparation of this evaluation report. We especially acknowledge and thank Alfredo Cruz for his insight, guidance, and support.

Telling the Story: The Results of the Evaluation of Richland School District One After-School Programs Page i

TABLE OF CONTENTS Executive Summary ...................................................................................................................... 1 Background ..................................................................................................................................... 2 Overview of Programs ................................................................................................................... 2 Demographics of Participating Schools Relevant Historical Events Proposed Program Goals Evaluation Approach...................................................................................................................... 6 Contextual Factors Evaluation Design Conceptual Framework & Logic Model Evaluation Measures & Methods Matrix Evidenced-Based Best Practices in After-School Programming Site Visits & Observations Key Stakeholder Interviews Data Collection Methods Data Analysis Evaluation Findings ........................................................................................................................ 24 Comparative Analysis to Evidenced-Based Best Practices Results of Interviews & Field Observations Program Effects Unanticipated Outcomes Recommendations Given By Interviewees Additional Evaluation Measures & Methods Discussion ........................................................................................................................................ 39 Methodological Scope & Limitations Summary of Key Findings USC Recommendations References ....................................................................................................................................... 45 Appendices Appendix A: Interview Discussion Guide for School Personnel Appendix B: Interview Discussion Guide for Community Advisory Committee (CAC) Appendix C: Interview Discussion Guide for Community Partners Contact Information

Telling the Story: The Results of the Evaluation of Richland School District One After-School Programs Page ii

EXECUTIVE SUMMARY In November 2003, the University of South Carolina Arnold School of Public Health (USC ASPH) was awarded a contract to evaluate the efforts of three middle school after-school programs in Richland School District One (Columbia, SC) that were funded by the John S. and James L. Knight Foundation. The evaluation occurred during January through June of 2004 and was conducted by an evaluation team of research faculty and staff through a variety of methods including direct observation of after-school programs, qualitative interviews conducted with all levels of key stakeholders involved in the program, and comparative analysis of the after-school program against evidenced-based best practices identified in the scientific literature. Evaluation findings revealed several strengths and weaknesses in the Richland School District One afterschool program. Weaknesses identified through the evaluation were a lack of fidelity to the program timeline; ambiguous District role in leadership and project management; inconsistent and varying program activities across the three after-school sites; ambiguous program activities and unreasonable outcome expectations stated in the grant proposal; and the high administrative burden placed on site coordinators. Identified strengths in the after-school program include success experienced with using the after-school curriculum; improved student/teacher relationships; improved student engagement; increased opportunities for students and parents to attend cultural events; inventive, motivated, and dedicated teachers; the ability of the site coordinators; and the support of the Foundation. Based on a review and synthesis of scientific literature on evidenced-based best practices, the recommendations provided by program stakeholders at all levels, and the direct observation by the evaluation team, several recommendations are proposed. These recommendations include incorporating the Office of After-School Programs into project management, formalizing the relationship between community partners and the after-school program, devoting more planning time to facilitate parental involvement and support, improving the program’s design to measure program effects, and conducting a formative evaluation before the next implementation of the program to assess critical gaps.

Telling the Story: The Results of the Evaluation of Richland School District One After-School Programs Page 1

BACKGROUND The children (in our program) speak positively about all of the enrichment activities…not just ours. They are so gracious for all we do. -- Stakeholder Interviewee, May 2004 In November 2003, the University of South Carolina Arnold School of Public Health (USC ASPH) was awarded a contract to evaluate the efforts of three middle school after-school programs in Richland School District One (Columbia, SC) that were funded by the John S. and James L. Knight Foundation. The evaluation occurred during January through June of 2004 and was conducted by an evaluation team of research faculty and staff through a variety of methods including direct observation of after-school programs, qualitative interviews conducted with all levels of key stakeholders involved in the program, and comparative analysis of the after-school program against evidenced-based best practices identified in the scientific literature. The purpose of this report is twofold: 1) to present the findings of USC ASPH’s evaluation of the John S. and James L. Knight Foundation’s Community Partners Program in the three middle schools and 2) to report on the achievement of USC ASPH’s proposed evaluation goals and activities. Generally, the latter is subsumed under the first purpose; however, in several instances the dynamic nature of the evaluation and “real life” occurrences necessitated the modification of our approach. These changes are described herein.

OVERVIEW OF PROGRAMS The John S. and James L. Knight Foundation funded after-school programs in three middle schools in Richland School District One during the 2003/2004 academic year. The three schools selected to receive funding for after-school programs were chosen based on demographic characteristics that reflected an increased need for academic remediation and the infusion of enrichment activities. Alcorn, Gibbes, and WA Perry middle schools were selected based on this criteria.


DEMOGRAPHICS OF PARTICIPATING SCHOOLS Alcorn, Gibbes, and WA Perry middle schools are located within urbanized areas of Richland County, South Carolina. School profile data maintained by the Richland School District One Office of Research & Evaluation demonstrate the need for targeted and effective interventions and enrichment programs:

Table 1. Demographics of Alcorn, Gibbes, and WA Perry Middle Schools* Demographic Characteristics

Alcorn Middle School (570 students)

Gibbes Middle School (476 students)

WA Perry Middle School (420 students)

60

23

46

Suspension rate 2

10.5%

4.5%

10.9%

Older than usual for grade 3

4.8%

4.1%

4.3%

Retention rate 4

26%

26%

21%

53% ELA 58 Math

48% ELA 67% Math

49% ELA 67% Math

Percent Poverty Level 6

91.9%

86.4%

97.84%

Public Housing Units 7

≈ 500

≈ 430

≈ 600

Children in foster care 8

≈ 31

≈ 40

≈ 30

Referrals to Department of Social Services 9

108

106

88

≈ 85%

≈ 78%

≈ 92%

Referrals to hearing office 1

Scoring below basic on state tests 5

Unemployment rate 10

* Data obtained from Richland School District One grant proposal submitted to the Knight Foundation. 1

Students referred to the hearing office in a non-duplicated count. A referral to the hearing office that results in automatic suspension. 3 Calculated by the SC State Department of Education based on students’ birth dates reported during state testing. 4 Percent of students who will fail unless they successfully complete summer school and are promoted. 5 Based on testing conducted in Spring 2002. 6 Data calculated by a formula used by the SC Department of Education using free and reduced lunch status. 7 Reported by the Central Midlands Council of Government and does not include families that are provided Section 8 vouchers. 8 A highly transient group. 9 Students are referred primarily by school nurses, school guidance counselors, school social workers, school resource officers, and school administrators and receive an array of services focused primarily on family assistance (food, transportation, medical services, and housing). 10 Unemployment rates in the schools’ attendance zones. 2


In all three schools, between one-fifth and one-forth of students are not promoted to the next grade level due to poor academic performance. More than half of the student body in each of the identified schools scored below basic in math; with nearly one half of the students not performing at a basic level in English and Language Arts (ELA). The vast majority of students (91.9% Alcorn, 86.4% Gibbes, & 97.84% WA Perry) at all three schools are on free or reduced lunch and live in areas with high unemployment rates (≈85% Alcorn, ≈78% Gibbes, & ≈92% WA Perry). The total student population is 99% African-American.

RELEVANT HISTORICAL EVENTS Two relevant historical events should be considered when reviewing the evaluation findings contained in this report: the physical relocation of Gibbes Middle School and the elimination of

More than half of the student body in each of the identified schools scored below basic in Math; with nearly one half of the students not performing at a basic level in English and Language Arts (ELA).

summer school. In December of the project period (2003-2004 academic year), the students, faculty, and staff of Gibbes Middle School were relocated to a new physical building while their school undergoes an essential renovation. During the relocation, all school operations (including the provision of the after-school programs) were interrupted and as a result, several after-school activities were postponed or were not offered. Additionally, effective during the 2003-2004 academic year, compulsory attendance at summer school for underperforming students was replaced with mandatory attendance in a comprehensive remediation program that took place after “day classes” during the school week. The shifting of remediation efforts to the afternoon (instead of during the summer months) increased the workload of teachers and administrators which impaired the activities of the funded after-school program. In spite of these challenges, however, project staff and community partners were successful in providing high quality after-school activities for their students.


PROPOSED PROGRAM GOALS Richland School District One proposed five program goals in the two grant proposals that were awarded by the John S. and James L. Knight Foundation. The goals of the after-school program were student–, family–, and community-based and focused on assisting students in meeting academic standards, enhancing social and cultural development, increasing parental participation, and the formation of meaningful community partnerships. Drawn from the original proposals, the stated program goals were:

Richland School District One After-School Program Goals: 1. Assist middle school students in meeting standards in core courses 2. Enhance the social and cultural development of students 3. Increase parental participation in school-related activities 4. Form meaningful community partnerships 5. Provide organized enrichment activities

In conjunction with the above goals, specific objectives, tasks, and outcomes were also stated.

The

objectives, tasks, and outcomes stated in the two grant proposals were to: " Hire site coordinators " Improve students’ performance on standardized achievement tests " Integrate enrichment programs into the after-school program " Increase student engagement " Provide professional development activities for site coordinators and teachers " Assess existing after-school programs’ strengths and weaknesses " Explore new after-school model programs " Design an after-school program that meets the needs of Alcorn, Gibbes, and WA Perry middle

schools " Identify alternative funding sources " Increase teacher collaboration " Decrease behavior problems " Implement at least one major enrichment event at each school away from campus " Complete secondary education and transition to work or further education. Telling the Story: The Results of the Evaluation of Richland School District One After-School Programs Page 5

EVALUATION APPROACH CONTEXTUAL FACTORS The philosophy and underlying framework guiding our evaluation approach is that program evaluation is collaborative, flexible, and meets the needs of multiple stakeholders – including the recipients of program activities. By adopting both an Empowerment Evaluation (Fetterman, Kaftarian, & Wandersman, 1995) and a Participatory Evaluation (SC DHEC, 2003; Durning, 1993; Zacharakis-Jutz & Gajenayake, 1994; Bhatnager & Williams, 1992) approach we hoped to assist the after-school sites by sharing our knowledge, experiences, and resources in a supportive role. Initially, we proposed to build our project partners’ evaluation skills and enhance their capacity to evaluate their own efforts.

However, several things

occurred that required us to modify our approach. First, all three Site Coordinators were not in place until late Fall 2003. The implementation of after-school enrichment activities, therefore, at each site was somewhat delayed. Second, because of the delay, the Site Coordinators were, justifiably so, focused on immediately planning and quickly implementing program activities. Including evaluation capacity-building strategies and professional development activities were simply not feasible. Third, an introductory project-wide meeting with all collaborating partners (including the School District personnel, partnering community organizations, and the evaluation team) was not held until mid-January 2004. Although at the mid-point of the funding year, expectations, role clarification and project timelines were established at this meeting. Finally, two District-level events influenced the amount of time after-school program staff had available: the implementation of the after-school Comprehensive Remediation Program in late October 2003 and the December 2003 relocation of Gibbes middle school to a temporary site while a new building is being constructed. Therefore, rather than providing technical assistance to build evaluation capacity we assumed more of a traditional, independent approach to our evaluation activities. Despite the modification of our approach, we were successful in achieving several of our originally proposed goals: " Establishing a productive working relationship with personnel at the three after-school

program sites and with key District personnel;


" Identifying process and outcome level measures and methods necessary for conducting

evaluation activities " Identifying mechanisms of support and resources for on-going evaluation activities with after-

school programs in the future; and, " Conducting qualitative evaluation activities to “tell the story” of how the programs have made

a difference from multiple perspectives: teachers, school administrators, program staff, and partnering agencies. Because of the reasons discussed above, we were less successful in: " Enhancing the capacity of the after-school programs to evaluate their own intervention efforts;

and, " Providing training, resources, and support for program evaluation activities to help the after-

school sites and Community Partners move towards their goals. It is our intent to include evaluation capacity-building and professional development activities for afterschool personnel in Year 2 as specific needs are identified.

EVALUATION DESIGN Since the Richland School District One after-school programs were in their first year of development and implementation, we conducted primarily a process-level evaluation. Doing so is appropriate for several reasons.

First, the after-school enrichment programs were implemented across the three sites with

varying levels of intensity over a relatively short time period. Realizing impacts in longer-term outcome measures (e.g., increases in standardized test scores, increases in student retention, enhancing the social development, etc.) will require intervention activities that are delivered consistently, reinforce clear messages, and last a sufficient length of time (e.g, five years) (Kirby, Lezin, Afriye & Gallucci, 2003). Second, process evaluation, frequently referred to as “implementation assessment,” “feasibility assessment,” or “quality assurance review (QAR),” is valuable in defining the structure, procedures, and content of intervention delivery, while simultaneously documenting program strengths and weaknesses (Windsor, Baranowski, Clark & Cutter, 1994). Third, process-level evaluation methods also address two important concerns in education programming: 1) the failure to implement (i.e., a Type III error) and 2) implementation with fidelity to the proposed plan (Steckler & Goodman, 1989; Basch, Sliepcevich, Gold, Telling the Story: The Results of the Evaluation of Richland School District One After-School Programs Page 7

Duncan & Kolbe, 1985). Fourth, process evaluation is essential for fine-tuning intervention elements and identifying contextual factors necessary for informing program replication in other settings. Lastly, process evaluation findings provide valuable insight into “how,” “why, or “why not” a program was successful or not (Windsor et al., 1994). All of these conditions are particularly important in the development, implementation and assessment of new, innovative programs like the Richland One after-school enrichment programs. A combination of procedures and methods were employed to conduct the process evaluation: " Development of a program logic model; " Creation of evaluation measures and methods matrix (“the grid”); " Literature review of evidence-based standards for successful after-school programs; " After-school site visits and observations; and, " Key stakeholder interviews.

CONCEPTUAL FRAMEWORK & LOGIC MODEL In order to gain clarity about the after-school program design, intent, and outcome expectations, the evaluation team utilized the District’s program proposals to construct an intervention map, or “program logic model.” Systematically analyzing the content of the funded proposal served several purposes. First, it identified the planned strategies and activities to be delivered to whom and by when (i.e., process-level goals).

Second, the underlying determinants (i.e., risk and protective factors) directly linked to the

intended behavior changes were clarified. Third, specific long- and short-term program outcomes were elucidated. Finally, and most importantly, the logic of the proposed model and its assumptions could be assessed. While there are numerous types of program logic models, the evaluation research literature is clear: regardless of format or type, they are crucial for the success of any program’s implementation and, ultimately, assessment of effectiveness (Philliber, 2004; Coffman, 2003; Steckler & Linnan, 2002; Kirby, 2001; W.K. Kellogg Foundation, 2001).

[See next page for the logic model.]


Figure 1. Logic Model for Richland School District One After-School Program

After-School Program/ Curricular Activities

Selected Individual Determinants (risk & protective factors)

Behaviors

Goal

Academic Activities - Select approved, establish curricula - Implement classroom based instruction in core subject areas (math, ELA) with effective teachers who are trained & receive a stipend - Provide individualized tutoring and homework assistance - Implement referral system for identifying & enrolling students into after-school program Enrichment (Arts/Creative Expression) Activities - Implement arts, music, drama programs for students through community organizations (partners) Enrichment (Jobs & Careers) Activities - Implement career, job, college exploration programs for students through community organizations (partners) Enrichment (Physical Activity, Recreation, Hobby) Activities - Implement sports, clubs, recreation programs for students through community organizations (partners) Enrichment (Mentoring) Activities - Implement mentoring programs for students through community organizations (partners) Enrichment (Service Learning) Activities - Implement service learning programs for students through community organizations (partners)

- Increase participation in academic remediation activities - Increase student “engagement”/ attachment to school - Foster positive attitudes toward school

↑ PACT Test Scores

- Reduce unstructured time

↑ Performance in core academic courses

- Cultivate students’ educational aspirations & plans for the future

↑ Student retention

- Increase ability to communicate effectively with teachers, peers, & parents

↓ Discipline referrals

Increase the academic success & social development of at-risk urban middle school students, ages 10-13 yrs.

↓ School suspensions

- Increase connectedness between students and parents - Increase attachment to adults in 1+ settings

Parental Program Activities - Implement programs to help parents understand their children and communicate with them - Implement programs through schools and community partners for parents and their children to do things together - Implement programs to encourage parents to become involved in monitoring their child’s academic performance


EVALUATION MEASURES & METHODS MATRIX The program logic model was used to identify specific measures and methods appropriate for evaluating the program’s proposed goals and objectives. For each of the three after-school sites, a “measures & methods” matrix, or “grid,” was created. For each proposed program objective, the following key process evaluation questions were posed: " “What did we say we would do?” (i.e., proposed goals & objectives) " “Who will do it?” (i.e., each of the specified objectives) " “How will we know if we did, or are doing, it?” (i.e., document each objective’s occurrence) " “How will we know how well we did, or are doing, it?” (i.e., determination of quality)

Initially, the evaluation team met with the after-school program Site Coordinators, school personnel, and District staff to collaboratively work through “The Grid.” However, due to time constraints, the Site Coordinators worked independently and with each other to complete the information relevant for their specific programs. The importance of this evaluation activity can not be understated. The matrix, in conjunction with the program logic model, produced valuable information.

First, it documented that after-school program

activities were occurring at all three school sites and were reaching the intended recipients (i.e., no Type III error). Second, it assisted the evaluation team in identifying internal system resources (i.e., assets and capacity) essential for conducting future outcome evaluations. Third, it helped build consensus among key stakeholders as to the purpose, direction, and types of evaluation data required. Lastly, although the iterative, participatory process was not fully implemented, it helped the evaluation team and the Site Coordinators establish an effective, working relationship.

[See next page for the evaluation measures & methods matrix.]


Table 2. Evaluation Measures & Methods Matrix What we said we would do:

1) Hire (3) Site Coordinators

Who will do it?

" District Interview " " "

2) Assess existing After- " School programs " strengths & weaknesses

" "

3) Cover transportation " costs "

" "

4) Site Coordinators work together to explore evidence-based After-School programs that work:

" Review at least 10

How will we know how well (i.e., the quality of) we

Comments/ Information Sources

Ms. Washington (Perry), The positions were Ms. Lackey (Alcorn), and successfully filled. Ms. Reese (Gibbes) were hired in October 2003.

A job description was secured from website.

USC After-School Programs Planning & Grants Site Coordinators

USC will perform an audit and conduct interviews with site coordinators.

A survey will be developed to collect data.

Knight Foundation Mr. Vanderpuji (principal at Perry) Mr. Adamson (principal at Alcorn) Mr. Coleman (principal at Gibbes)

Students will be safely transported the entire year.

Adequate funding will be The transportation available from the Knight budget will be assessed. Foundation and the school.

" Site Coordinators

Site Coordinators will complete an Evidence Based After-School Program Summary sheet for each of the (10) programs.

Team Mr. Vanderpuji (principal at Perry) Mr. Adamson (principal at Alcorn) Mr. Coleman (principal at Gibbes)

" Ms. Washington " " "

How will we know if we did/are doing it?

USC will analyze and evaluate data collected by each site coordinator.

The job description needs to be reviewed and updated.

A copy of the 20022003 evaluations needs to be secured.

(Perry) Ms. Lackey (Alcorn) Ms. Reese (Gibbes) Identified Team " Members

successful programs

"

will meet weekly to discuss effective programs. (Friday10 a.m.) Each coordinator will research at least (5) effective programs utilizing best practices. Site Coordinators will work together to analyze and review findings and identify key/ compatible components.

The Evidence Based After-School Program Summary sheets will be reviewed and used in the development of a model After-School program.

[Evaluation measures & methods matrix continued on next page.]


Table 2. Evaluation Measures & Methods Matrix (continued) What we said we would do:

5) Site Coordinators analyze social & other factors (homelessness, substandard living, transience, suspensions, etc.) interfering with student success

Who will do it?

" Ms. Washington " " " " " "

6) Site Coordinators examine partnerships, establish new relationships with potential partners

" Ms. Washington " " " " "

7) Identify alternative funding sources

(Perry) Ms. Lackey (Alcorn) Ms. Reese (Gibbes) Guidance Counselors School Social Worker Administrators Curriculum Resource Teacher (CRT)

" Dr. Buxton " Dr. Pastides (USC

" Teachers " Curriculum "

Monthly meetings will be held with the guidance counselors, the school social worker, and each administrator to collect and discuss data.

Resource Teacher (CRT) Tutors


An evaluation form will be created and utilized partnerships and to analyze each their services will partnership. be assessed. Site Coordinators will evaluate the effectiveness of existing partnerships and the services offered as they relate to the After-School Program.

" Continue to "

research alternative funding sources. Collect data on each source.

The teachers will implement expected practices: " 1 on 1 instruction " Small group learning " Address individual learning styles


The program A log sheet will be components will address created to document the needs of all students meetings. served.

" A list of

(Perry) Mr. Vanderpuji (principal at Perry) Ms. Lackey (Alcorn) " Mr. Adamson (principal at Alcorn) Ms. Reese (Gibbes) Mr. Coleman (principal at Gibbes)

VP for Research & Health Sciences)

8) Execute after school instruction in core academic areas


The evaluation form will be analyzed.

The After-School program will secure various grants and other alternative funding to ensure sustainability.

Annie E. Casey Foundation and Mott Foundation are possible funding sources.

" CRT will observe

" " " "

" "

all classes and provide feedback. USC will observe and provide feedback. District site review team members will observe and provide feedback.

" " " " " " "

PACT Coach Pre-test Post test Benchmark Indicators Larson Pre-Algebra Accelerated Reader STAR diagnostic Read 180 Teacher Created curriculum in all content areas Success Maker Ed Vision




Who will do it?



The CRT, site coordinators, and USC evaluation team will observe all classes and provide feedback.

A needs assessment will be developed.

The CRT, site coordinators, and USC evaluation team will observe all classes.

A checklist supplied by the Curriculum and Standards Office will be utilized to evaluate instruction.

A checklist will be utilized.

" Teachers " Ms. Washington

" Identify effective

" Students will able

" " "

"

The attendance of students and a schedule of activities will be utilized to track the percentage of participation.

9) Implement instruction " that addresses individual " learning styles

" " " " "

10) Implement 1-on-1 & group instruction for students

" Teachers " Curriculum " " " " "

11) Integrate enrichment programs into the After-School program: a) At least 5 activities b) 95% student participation rate for at least 3 activities

Teachers Curriculum Resource Teacher (CRT) Tutors USC evaluation team Ms. Washington (Perry) Ms. Lackey (Alcorn) Ms. Reese (Gibbes)

Resource Teacher (CRT) Tutors USC evaluation team Ms. Washington (Perry) Ms. Lackey (Alcorn) Ms. Reese (Gibbes)

(Perry) Ms. Lackey (Alcorn) Ms. Reese (Gibbes) Volunteers


" Teachers will

"

"

administer the learning styles inventory. The teachers will implement expected practices to address individual learning styles.

enrichment programs. Correlate enrichment programs to the curriculum and standards. Develop a schedule of activities that will be structured so that all students will have an opportunity to participate.

to select activities.

" Pre- and post-

" "

academic assignments are given to the students. Attendance Rosters will be reviewed. A log is maintained outlining which standards are met by the enrichment program.




Who will do it?

12) Design an After" School program to meet students’ needs: " a) Site Coordinators meet weekly " b) After-School advisory team meet monthly c) Model implemented in 2004-05 school year

" " " "

" " 13) Improve student performance on standardized tests: a) Participants’ PACT scores increase 5 pts.

Ms. Washington (Perry) Mr. Vanderpuji (principal at Perry) Ms. Lackey (Alcorn) Mr. Adamson (principal at Alcorn) Ms. Reese (Gibbes) Mr. Coleman (principal at Gibbes) Site-based AfterSchool Advisory team (including students) USC Grant Writing Team

" Teachers " Ms. Washington (Perry)

" Mr. Vanderpuji


" Site Coordinators "

"

"

will meet weekly. The site-based After-School advisory team will meet bi-weekly. The identified model program will be continuously assessed and revised during the 2004-05 school year. Draft copies of the grant will be created and edited.



The following documentation will be utilized: " Meeting agendas " Advisory Team Meeting Minutes " Log Sheet " Draft of model program recommendations " Draft copies of grant

The following information sources will be reviewed: " Evidence Based After-School program summary sheets " Site-based AfterSchool advisory minutes " Agendas " Log sheets " Draft copies of grant

Student performance " data will be analyzed and reviewed. "

(principal at Perry)

" Ms. Lackey (Alcorn) " Mr. Adamson

b) Participants’ PACT scores will have significant increase " over non-participants "

"

(principal at Alcorn) Ms. Reese (Gibbes) Mr. Coleman (principal at Gibbes) CRT

PACT data will be disaggregated. A comparison of After-School program participants and non-participants will be conducted (data analysis).

" PACT scores " Request that after–

"

school data be included in data from the Office of Research and Evaluation. Request that SASI/ IT create a system that will allow the calculation of grades in accordance with percentages agreed upon by principals.




14) Increase parental involvement: " 70% student parents participate in at least 2 activities

Who will do it?

" Ms. Washington " " "

" " "

15) Increase community involvement

" " "

" " " " " "



" The school will

Sign-in sheets, agendas, surveys, evaluation forms, and a calendar of activities will be utilized to track parental participation.

The following information sources will be reviewed: " Sign In sheets " Agendas " Calendar of activities " Parental Survey " Parental Evaluation

" The school will

Sign-in sheets, agendas, and a calendar of activities will be utilized to track community participation.

" Sign In sheets " Agendas " Calendar of

(Perry) Ms. Doucet (AfterSchool Parent Liaison at Perry) Ms. Lackey (Alcorn) " Ms. Osborne (After-School Parent Liaison at Alcorn) Ms. Reese (Gibbes) Ms. McBride (AfterSchool Parent " Liaison at Gibbes) Students

" Ms. Washington "


(Perry) Ms. Doucet (AfterSchool Parent Liaison at Perry) " Mr. Vanderpuji (principal at Perry) Ms. Lackey (Alcorn) Ms. Osborne (After-School Parent Liaison at Alcorn) Mr. Adamson (principal at Alcorn) Ms. Reese (Gibbes) Ms. McBride (AfterSchool Parent Liaison at Gibbes) Mr. Coleman (principal at Gibbes) Community Leaders Volunteers

sponsor at least four activities in order to encourage parental involvement. A survey will be administered to parents seeking their input on strategies for increasing parental involvement. An evaluation form will be administered to parents after each activity. sponsor activities to encourage community involvement. Identify all current and potential stakeholders.

activities




16) Increase student engagement: a) Student absences decrease 60% b) Student discipline referrals decrease 60%

Who will do it?


" School Staff " Tutors " Ms. Washington

" Attendance will be

" " " " "

17) Provide professional " development activities for site coordinators & " teachers: " a) All Site Coordinators attend 2 activities b) All teachers attend 2 activities

" " " "

(Perry) Ms. Lackey (Alcorn) Ms. Reese (Gibbes) " Parents Mentors Students

Ms. Washington " (Perry) Ms. Lackey (Alcorn) Ms. Reese (Gibbes) Office of AfterSchool Programs Office of Curriculum and Standards Office of " Professional Development Community based organizations

"

"



Attendance totals and discipline reports will be taken daily and analyzed. reviewed and analyzed on a weekly basis. Discipline referrals will be filed and the data will be entered into the SASI database. A discipline report will be generated and analyzed on a weekly basis.

The following information sources will be reviewed: " School Renewal Plan " Attendance Rosters " Discipline Records " Incident Reports

Teachers and Site " Coordinators will complete a professional development needs " assessment provided by the Office of Professional Development. Identify appropriate professional development activities for site coordinators and teachers. Professional development registration forms and agendas will be utilized. A professional development calendar will be created by July

" A Professional

The teachers and Site Coordinators will complete a self assessment. Skills obtained during professional development will be modeled in the After-School Program.

"

"

development list from the Office of Professional Development will be secured. A professional development calendar will be developed. A needs assessment form will be utilized.


EVIDENCE-BASED BEST PRACTICES IN AFTER-SCHOOL PROGRAMMING In order to confidently make judgments and draw conclusions about the quality and future effectiveness of the after-school programs, a core set of standards (i.e., criterion-referenced guidelines) were needed. Therefore, the evaluation team conducted a literature and resource review of the evidence-based best practices in the field of after-school programming. Online data bases such as ERIC, Social Sciences Index, PsycINFO, Social Work, and Medline were searched to locate current (within the past ten years) research and recommendations for practice. In addition, a web-based search using various Internet search engines (e.g., Google, Yahoo, AltaVista, etc.) was performed to identify best-practices endorsed by creditable governmental, professional, academic, and service organizations active in middle school after-school education. Numerous resources were synthesized and three most frequently cited sources were identified to produce the following list of 21 commonly accepted and proven characteristics of effective after-school programs. Table 2. Evidenced-Based Best Practices in After-School Programming 1. Safe, stable places 1 (Students have the opportunity to) develop a sense of belonging in an environment of physical and social safety 3 2. Basic care and services 1 3. Caring relationships 1 Staff who engage in frequent, positive interactions with participants 2 (Students have the opportunity to) build close relationships with peers and adults 3 (Students have the opportunity to) belong to a positive peer group, develop good relationships with caring adults 3 4. Relevant, challenging experiences 1 Authentic curricula that is age-appropriate and provides engaging, skill-building, hands-on activities geared toward the particular goals of the program 2 (Students have the opportunity to) be exposed to a wide range of challenging and interesting learning experiences that help them build understanding, skills, and competencies 3 (Students have the opportunity to) participate in meaningful and tangible projects that enhance self-esteem and offer opportunities for leadership 3 5. Networks and connections 1 (Students have the opportunity to) become involved in the larger community and make real contributions while broadening their knowledge and understanding the world around them 3 6. High expectations and standards 1 Staff with high expectations of all youth 2 Clear rules and expectations with consistent consequences 2 (Students have the opportunity to) take on meaningful roles that involve responsibility and opportunities for leadership 3

[Evidenced-based best practices continued on next page.] Telling the Story: The Results of the Evaluation of Richland School District One After-School Programs Page 17

Table 2. Evidenced-Based Best Practices in After-School Programming (continued) 7. Opportunities for voice, choice, and contribution 1 Flexibility that allows participants to choose activities that interest them or choose the approach they use to achieve goals 2 Youth valued as resources including having a voice in determining program content and connecting to larger community 2 A role for participants in creating, implementing, and reflecting on the program 2 (Students have the opportunity to) have input and make decisions regarding the activities they take part in, the rules they abide by, and the setting where they spend their time 3 8. Personalized, high-quality instruction 1 Low adult-to-youth ratios 2 High quality content 2 9. Sufficient well-trained and compensated staff who receive health and other benefits 2 10. Staff who understand the developmental tasks faced by young people and are able to help them accomplish these tasks 2 Staff who work in the program over a long period of time and are able to build relationships with the participants 2 Staff who understand the cultural, racial, ethnic, and class backgrounds of participants and are able to support healthy identity development in a diverse group of young people 2 11. Strong, sustainable administration 2 Adequate funding without constant threat of loss 2 12. Clarity of mission and goals 2 13. On-going self-assessment and evaluation 2 14. A full-time coordinator 2 Managers who are skilled administrators, savvy advocates, inspiring leaders and connected to community resources 2 15. Support of the school principal (especially if school-based) 2 Collaboration between after-school program staff and in-school staff 2 16. Involvement of parents 2 17. Access to appropriate space for program and storage 2 18. Connections to community partners and infrastructure for programming, training and technical assistance 2 19. If homework is an after-school program component, then teachers from the same school or within the same district should be a part of the after-school staff 2 Schoolteachers should not be the foundation of the after-school program workforce because of the long hours required to manage both positions 2 20. Literacy development, including reflection and discussion 2 21. Transportation 2 1

Hall, G., Yohalem, N., Tolman, J., Wilson, A. (2003) How Afterschool Programs Can Most Effectively Promote Positive Youth Development as a Support to Academic Achievement: A Report Commissioned by the Boston After-School for all Partnership. Miller, B. M. (2003). Critical Hours: Afterschool Programs and Educational Success. Quincy, MA: The Nellie Mae Foundation. 3 National Institute on Out-of-School Time. (2000). Making an Impact on Out-of-School Time: A guide for corporation for national service programs engaged in after school, summer, and weekend activities for young people. Wellesley, MA: The National Institute of Our-of-School Time. 2


Although a single set of characteristics or “best practices” has not been adopted within the field of afterschool practice, there are some common elements that have been identified as key components of effective after-school programs. Over the last decade, researchers from different organizations have developed their own research-based lists of characteristics of best practices in after-school/out-of-school time programs.

In Critical Hours: After-school Programs and Educational Success (2003), Miller cites evidence-based research that indicates that engagement in learning is a critical component of middle school or junior high after-school programs because of the lack of engagement many students experience during the traditional school day. This lack of engagement can be attributed to a number of factors including: larger schools, larger classroom sizes, and the distractions associated with adolescents’ physical and social changes. Afterschool programs can provide a medium for middle-school students to feel more connected to their school, more confident about themselves, and more likely to receive personal attention from caring adults.

In addition, Miller reviewed research focusing on middle school programs. One study reviewed found that parents felt that the after-school program, which included enrichment programs in addition to both help with homework and tutoring, contributed to their children both having a more positive attitude about school and behaving better during school. A research study of publicly funded after-school programs for middle schoolers in North Carolina, whose goal was to improve academic performance, found that test scores for participants in both reading and mathematics improved over the two years the program was offered.

Miller also cites research that identifies the students who are most likely to benefit the most from afterschool programs. These students tend to be the most disadvantaged: those who are low income, those with low school attendance, those with limited proficiency in English, and those with poor test scores. In Telling the Story: The Results of the Evaluation of Richland School District One After-School Programs Page 19

addition, “dosage” makes a difference in the effects that programs have on students. Miller cites research that indicates that students who attend after-school programs for the most hours over the most years benefit more than the students who come less often.

Since the three middle schools selected for the grant have populations that are 99% African-American, it is important to review the literature relating to developmental assets among adolescents of color. The Search Institute (Sesma, 2003) created a Framework of Developmental Assets that groups factors that promote healthy development in youth ages 12 to 18. They are grouped into two categories: external assets and internal assets, each with four sub-categories. The external assets are: support, empowerment, boundaries and expectations, and constructive use of time. The internal assets are: a commitment to learning, positive values, social competencies, and positive identity.

Among six categories of youth

examined by race/ethnicity, including African American, young people with more assets have been shown to be less likely to engage in high-risk behaviors. The Search Institute also found that the internal asset of a commitment to learning is related to school success among African-American youth. This supports the premise that effective after-school programs involve students and engage students in learning. The more connected a young student is to his or her school, the more a student receives positive attention from adult at school, the more likely that the student will be successful (Sesma, 2003; Hall et al, 2003; Miller, 2003).


SITE VISITS & OBSERVATIONS Members of the evaluation team conducted two site visits at each of the three program sites in order to observe program activities. Each evaluator manually recorded field notes of their classroom and site observations.

These notes included observations of teacher/student interactions, the instructional

strategies utilized, the perceived receptivity of students, notation of observed behavioral problems (including not paying attention, disruption, etc.), the use of audio/visual instructional aids, and other important observations. A formal, structured observation checklist or guide was not developed or utilized during these observations due to the varying nature of each site’s after-school programs.

KEY STAKEHOLDER INTERVIEWS In order to provide a rich understanding and to “tell the story” of the program’s critical personal and environmental success factors, the evaluation team conducted a series of structured, personal interviews with key stakeholders (n=21). Stakeholders are persons or agencies who have a vested interest in the after-school program’s progress and impact. Moreover, stakeholders are often in a position to facilitate change regarding the success or failure of a program. Intended recipients of program activities are valuable stakeholders as well. However, given that the program participants are minor children and that USC’s Institutional Review Board (IRB) human subject approval process would take 2-3 months, students could not be included as part of the key stakeholder interviews but will be included in future evaluations.

DATA COLLECTION METHODS From April – May 2004, eight persons (e.g., site coordinators, principals, and teachers) from the three middle schools, seven members of the Community Advisory Board, three persons from the District office, and five community partners from three organizations participated in the interviews. A standardized interview guide containing nine open-ended questions with probes, developed by the evaluation team, was used to elicit stakeholders’ perceptions of and experiences with the after-school programs. The wording of three questions was modified to reflect variations in experiences and level of involvement across stakeholder type (i.e., school personnel, District personnel, Community Advisory Board member, and community partner). (See Appendices for interview guides).


Participants were asked to describe their perceptions and opinions in the following areas: " General, overall perceptions of the after-school program & partnerships; " Specific program strengths and weaknesses; " Student and/or volunteer response to program activities; " Program effects on student participants; " Unanticipated outcomes; " Curricula and materials used; and, " Barriers and potential solutions for improvement.

In accordance with USC’s Institutional Review Board (IRB) policies, the evaluation procedures and stakeholder interviews met the requirements of exempt status and the requirement for IRB review was waived. Specifically, the interviews were conducted in commonly accepted educational settings and were for the purpose of determining the effectiveness of instructional strategies, curricula, or classroom management methods. Before the start of the interview, stakeholders were clearly informed of the purpose of the interview (i.e., for capturing their opinions and experiences).

Passive informed consent was obtained from each

participant; and, assurance was given that the transcribed data would be stripped of identifying information. Participants were also informed that they could withdraw from the interview at any time without repercussions. Three members of the evaluation team trained in qualitative data collection methods conducted the interviews in-person or via telephone. The interviews ranged in length from 70-90 minutes each. For the in-person interviews, a note-taker was present to record the data. The telephone interviewer recorded stakeholder responses him/herself. During the interviews, the evaluators used comprehension probes (Stewert & Shamdasni, 1994) to obtain and clarify responses from interviewees. For example, the interviewer may have asked “Would you give me an example of what you mean?” or “I’m not sure I understand, could you explain further?”


DATA ANALYSIS For confidentiality purposes, transcripts (both electronic and hard copy) of the interview notes were prepared with interviewees’ names and identifying information removed. In order to verify accuracy and completeness, the interviewer conducting the interview compared the electronic transcripts to the original handwritten notes and made corrections as necessary. NVivo® (QSR International, 2000) a qualitative data analysis program was used to code and analyze the interview and site observation transcripts. NVivo® allows for hierarchical, or “tree-like,” on-screen coding and analysis of large amounts of text across multiple themes, participant responses, and group types. The analytic strategy was to treat the data in a systematic manner by utilizing the interview questions as a framework and to adopt an ethnographic approach known as the “constant comparison method” (Glaser & Strauss, 1999; Strauss & Corbin, 1990). For example, to develop a codebook, 3 of the 21 transcripts were selected at random and the data was manually divided into as many “meaning categories” as possible. In this “open coding” (Strauss & Corbin, 1990) phase, an idea or theme was encountered NVivo®’s on-screen coding feature was utilized to mark the text and assign a semantic code. Using hard copy printouts, ideas and codes were used to validate thematic interpretations. Ultimately, over 80 codes (“meanings”) were created from the analysis of the 21 interviews and 6 field observations. Next, all of the coded transcripts were printed out by code type, reviewed for accuracy, and examined for links to other codes. In this step, “axial coding” (Strauss & Corbin, 1990) was performed in order to connect code categories and to look for relationships that generate themes. A theme was deemed to be any thought, idea, or experience noted by two or more interviewers or observers across all the stakeholder types. Themes were also analyzed for within and between stakeholder type similarities and differences.


EVALUATION FINDINGS COMPARATIVE ANALYSIS TO EVIDENCED-BASED BEST PRACTICES Detailed comparative analysis of the after-school program to evidenced-based best practices was not possible due to the lack of appropriate benchmarks in the program’s design. In can be inferred through the responses provided by program stakeholders and through direct observation by the evaluation team, however, that many of the identified evidenced-based best practices may be present in the after-school program. Table 3 (below) identifies which evidenced-based best practices may be present in the afterschool program as determined by stakeholders and the evaluation team. Additionally, the table includes the measure(s) needed in order to adequately assess the existence of the evidenced-based best practice.

Table 3. Likelihood of Existence of Evidenced-Based Best Practices in the After-School Program and Measures Needed to Confirm Existence Evidenced-Based Practice 1

1. Safe, stable places (Students have the opportunity to) develop a sense of belonging in an environment of physical and social safety 3 2. Basic care and services

1

Likelihood of Existence

Measure(s) Needed

Good

" Interview students " Documentation of reduced violence

Excellent

" Documentation

that demonstrates the provision of basic care and services

3. Caring relationships 1 Staff who engage in frequent, positive interactions with participants 2

Good

" Direct observation " Interview students " Interview teachers

Good

" Creation and

(Students have the opportunity to) build close relationships with peers and adults 3 (Students have the opportunity to) belong to a positive peer group, develop good relationships with caring adults 3 4. Relevant, challenging experiences 1 Authentic curricula that is age-appropriate and provides engaging, skillbuilding, hands-on activities geared toward the particular goals of the program 2 (Students have the opportunity to) be exposed to a wide range of challenging and interesting learning experiences that help them build understanding, skills, and competencies 3 (Students have the opportunity to) participate in meaningful and tangible projects that enhance self-esteem and offer opportunities for leadership 3

" " "

analysis of individualized learning plans Conduct a needs assessment with students Pre– and post-test assessment of students Interview students

[Table continued on next page.] Telling the Story: The Results of the Evaluation of Richland School District One After-School Programs Page 24

Table 3. Likelihood of Existence of Evidenced-Based Best Practices in the After-School Program and Measures Needed to Confirm Existence (Continued) Evidenced-Based Practice 1

5. Networks and connections (Students have the opportunity to) become involved in the larger community and make real contributions while broadening their knowledge and understanding the world around them 3

Likelihood of Existence Good

Measure(s) Needed

" Curriculum " "

1

6. High expectations and standards Staff with high expectations of all youth 2

Good

Clear rules and expectations with consistent consequences 2 (Students have the opportunity to) take on meaningful roles that involve responsibility and opportunities for leadership 3

" Interview students " Interview teachers " Review of " "

7. Opportunities for voice, choice, and contribution 1 Flexibility that allows participants to choose activities that interest them or choose the approach they use to achieve goals 2

Good

assessment for community service Pre– and Post-test assessment of students Interview students

program-wide policies Review of disciplinary actions Curriculum assessment for youth leadership

" Interview students " Conduct a needs

Youth valued as resources including having a voice in determining program content and connecting to larger community 2

"

A role for participants in creating, implementing, and reflecting on the program 2

"

assessment with students Review of meeting minutes to assess youth involvement Student-developed ground rules

(Students have the opportunity to) have input and make decisions regarding the activities they take part in, the rules they abide by, and the setting where they spend their time 3 8. Personalized, high-quality instruction 1 Low adult-to-youth ratios 2

Good

High quality content 2 9. Sufficient well-trained and compensated staff who receive health and other benefits 2

" Documentation of "

Fair

teacher/student ratios Curriculum review by expert panel

" Interview teachers " Documentation of teacher trainings.

" Salary comparison study

10. Staff who understand the developmental tasks faced by young people and are able to help them accomplish these tasks 2 Staff who work in the program over a long period of time and are able to build relationships with the participants 2

Good

" Documentation of teacher training

" Interview teachers

Staff who understand the cultural, racial, ethnic, and class backgrounds of participants and are able to support healthy identity development in a diverse group of young people 2

[Table continued on next page.]


Table 3. Likelihood of Existence of Evidenced-Based Best Practices in the After-School Program and Measures Needed to Confirm Existence (Continued) Evidenced-Based Practice 2

11. Strong, sustainable administration Adequate funding without constant threat of loss 2


Measure(s) Needed

Good

" Documentation of

" "

12. Clarity of mission and goals 2

Fair

" Documentation of

" " 13. On-going self-assessment and evaluation

2

Fair

Excellent

" " " 15. Support of the school principal (especially if school-based) Collaboration between after-school program staff and in-school staff 2

Good

regular selfassessments (program monitoring) Documentation of independent evaluation

" Clearly defined

" "

2

consistent and communicated program mission and goals Interview program managers (site coordinators) Interview teachers

" Documentation of

"

14. A full-time coordinator 2 Managers who are skilled administrators, savvy advocates, inspiring leaders and connected to community resources 2

efficiency in administrative activities Interview program managers (site coordinators) Documentation of a multi-year project

and realistic position descriptions Interview teachers Interview community partners Interview administrative personnel Documentation of employment reviews Direct observation

" Interview school principal

" Interview site coordinators

" Interview teachers

[Table continued on next page.]


Table 3. Likelihood of Existence of Evidenced-Based Best Practices in the After-School Program and Measures Needed to Confirm Existence (Continued) Evidenced-Based Practice 16. Involvement of parents

2


Measure(s) Needed

Fair

" Documentation of " " " "

17. Access to appropriate space for program and storage 2

Unknown

" Interview program "

18. Connections to community partners and infrastructure for programming, training and technical assistance 2

Fair

" "

" " " " "

20. Literacy development, including reflection and discussion 2

Good (this rating is tempered by the fact that most of the after-school teachers are also day-time teachers)

Unknown

managers (site coordinators) Direct observation

" Interview "

19. If homework is an after-school program component, then teachers from the same school or within the same district should be a part of the after-school staff 2 Schoolteachers should not be the only resource of the after-school program workforce because of the long hours required to manage both positions 2

parent participation Direct observation Interview parents Interview teachers Interview program managers (site coordinators)

community partners Interview program managers (site coordinators) Interview teachers Conduct a needs assessment with teachers and program mangers (site coordinators) Direct observation Documentation of trainings Formal assessment of trainings Documentation of technical assistance Formal assessment of technical assistance

" Curriculum review " "

for homework assignments Documentation of hours worked by teachers Documentation of a pool of qualified after-school teachers

" Curriculum review for literacy

" Pre– and post-test assessment

21. Transportation 2

Good

" Documentation of requests for transportation


RESULTS OF INTERVIEWS & FIELD OBSERVATIONS Fidelity to timeline Eight of the interviewees (36%) mentioned the lack of fidelity to the timeline as a weakness of the program. The main reasons why the timeline was not adhered to were delays in ordering the curriculum and hiring staff.

School and District personnel, as well as community partners and the community advisory

committee, mentioned the lack of fidelity to the timeline as an issue. Recruitment of Site Coordinators Two of the four interviewees who made comments related to the recruitment of site coordinators expressed that the expectations for the site coordinators were not clear. Interestingly, these comments were from persons outside of the school. On the other hand, school and District personnel made comments related to the retention of site coordinators; three interviewees felt that the site coordinators may be “overwhelmed” or “burned out”. Recruitment and retention (attendance) of teachers Very few comments were made regarding the recruitment and retention of teachers specifically. One site coordinator mentioned the difficulty of finding teachers to participate in the program and expressed frustrations over teachers not showing up for the program. Also, one community advisory committee member expressed the need to make sure all teachers were adequately trained to participate in the program. Two interviewees (both Site Coordinators) were surprised by the commitment and excitement of the teachers. Integration of enrichment into the after-school (AS) program Eight of the interviewees (36%) discussed the integration of enrichment activities into the curriculum. Overall, interviewees felt that the enrichment activities were tied into the academic activities. Some interviewees commented on specific partnerships where the integration of enrichment and academics was going especially well (the Art Museum and Trinity Cathedral).

Two interviewees expressed

recommendations for furthering the reach of the enrichment activities, including improving the link between “day learning and after-school learning” and making the enrichment-academics link more “holistic” and less “splintered.”


Assessment of strengths and weaknesses Strengths Interviewees from all stakeholder groups identified many strengths of the after-school program including: the site coordinators, the enrichment activities and the partners, the academic component, the teachers, the district, the schools, the volunteers, the Foundation, and the commitment of the administration.

The most frequently cited

strength, the enrichment activities and the partners providing them, was cited by 13 interviewees (59%). When talking about the strengths of the enrichment activities and the partners, interviewees mentioned the variety and quality of the enrichment activities, the commitment of the community partners, and how the enrichment piece exposes the students to new things. As one interviewee put it, when talking about the Columbia Art Museum’s involvement, “Being able to bring so many students to the museum who wouldn’t otherwise…come at all.” Another frequently mentioned strength of the program was the commitment of the administration. This strength was mentioned by seven interviewees (32%), including three community advisory

The most frequently cited strength of the afterschool program is the variety and quality of the enrichment activities provided by project partners.

committee members. The comments reflected the fact that the administration meant different things to different people; most often interviewees were talking about the commitment of the District, but some also talked about the commitment of other persons affiliated with the program such as the school principals, Site Coordinators, parent liaisons, and school resources personnel. Site Coordinators and teachers were each mentioned by six (27%) of the interviewees and the comments on them focused on their skills and expertise, commitment to the program, and ability to make personal connections with the students and other stakeholders (i.e. parents, the principal, partners). Weaknesses More often than not, interviewees discussed barriers to the implementation of the program rather than program weaknesses.

The next section discusses those barriers in more detail.

However, two

programmatic weaknesses were cited by more than one interviewee. The most frequently cited weakness was the lack of fidelity to the project’s timeline (see the section on fidelity to timeline above). The other weakness that was cited by two school-based interviewees was the process for selecting community Telling the Story: The Results of the Evaluation of Richland School District One After-School Programs Page 29

partners.

Both offered recommendations to make this process more effective, including a formal

application process and review, more involvement from the school (i.e. Site Coordinators and principals), and having the teachers develop the partnerships. Barriers to Implementation Eighteen interviewees (82%) discussed barriers to program implementation. The most frequently cited barrier was communication and it was mentioned by 13 interviewees (59% of all interviewees, and 72% of all those who mentioned any barriers). While the ways in which communication manifested itself as a barrier were multifaceted, several themes stood out.

First and foremost was the sentiment that

expectations had not been clear. Five (38%) interviewees, including 4 community advisory committee members and one District employee, talked about unclear expectations; however, their comments were different. community advisory committee members discussed how their expectations for the District should have been clearer. This is evidenced by the following quote: “We needed to be clearer on our end in regard to what elements needed to be in the District’s proposal so we’re all speaking the same language and on the same page so that we can start running the program.” Meanwhile, the District employee thought that expectations needed to be clearer between the District and the Site Coordinators and between the Foundation and the District. Another prominent theme was the general “disconnect” between the District and the other program stakeholders. This theme was most often mentioned by members of the community advisory committee, but was also mentioned by community partners and District personnel. One of the major ways in which this communication barrier manifested was in the lack of job descriptions for the Site Coordinators (mentioned by 4 interviewees).

Other comments

convey this sense of disconnect as well yet underscore the general tone that communication issues can be worked out.

Another prominent theme was the general “disconnect” between the district and the other program stakeholders.


Other barriers that were mentioned by at least three interviewees included staff burnout, difficulties communicating/coordinating with various community partners, and logistics (i.e. scheduling, transportation, persons not showing up). Many recommendations for addressing these barriers were also made. Those are discussed below in the “Recommendations Given By Interviewees” section.

“...the afterschool program is a safe haven for children and it addresses the children’s academic and cultural needs.”

After-school program meets needs of Alcorn, Gibbes, and WA Perry Schools This theme was just touched upon by two interviewees.

One

principal discussed that it was a goal of the school to put an afterschool program into place and how they have been able to achieve that goal.

In a similar vein, one community advisory committee

member discussed how the program is able to fulfill community needs, “We are addressing a critical need in the community…I am not sure what the district would do because of the lack of funding available for after-school programs in the District…the after-school program is a safe haven for children and it addresses the children’s academic and cultural needs.”

Design new enrichment activities that meets needs of Alcorn, Gibbes, and WA Perry Interviewees discussed this theme in terms of the student needs being met by the enrichment activities as evidenced in their engagement of the activities. See the “Increase student engagement” section under Program Effects. Teachers will collaborate more Two persons (one teacher and one principal) commented on increased teacher collaboration/ communication between day teachers, after-school teachers and/or the curriculum resource teachers. One of those persons commented on how that collaboration was a benefit to the school.


PROGRAM EFFECTS The qualitative data points toward a number of program effects that interviewees attributed to the after-school program including both academic and personal student benefits, increased student engagement, the provision of a safe environment leading to decreased behavioral problems, and increased parental involvement. However, community advisory committee members, in general, did not feel there is enough evidence to determine program outcomes. The USC evaluation team agrees that there is insufficient evidence to support outcome-based claims of program effectiveness until proper research design components are installed (described in “USC Recommendations”). In spite this limitation, however, several project stakeholders cited specific program effects that they have observed.

CAC members, in general, did not feel there is enough evidence to determine program outcomes.

Student Benefits Academic Eight interviewees (36%) discussed the academic benefits they have seen as a result of the after-school program. Of these, 75% of the comments were made by school personnel with the remaining 25% of the comments made by members of the community advisory committee. In general, the comments made by community advisory committee members focused on academic benefits in general terms, while the schoolbased interviewees mentioned more concrete benefits. Two interviewees explicitly mentioned grades going up and two explicitly mentioned how students in the after-school program can get points that can be used towards their day curriculum grades. Also, two comments were made on the benefit of seeing the same teachers in the day classes and in the after-school program. One teacher offered a more qualitative measure of students’ academic progress: “In PACT testing, they have to write complete sentences or it doesn’t get counted so I’m always telling the kids ‘Write in complete sentences!’ You can tell that the kids are learning [in the after-school program] because they’re writing in complete sentences more often.” Personal Other, personal, student benefits were mentioned by five interviewees (23%) from a broad representation of stakeholders including three community partners, one District, and one school-based person. A wide Telling the Story: The Results of the Evaluation of Richland School District One After-School Programs Page 32

variety of personal student benefits were identified, including increased confidence, socialization, motivation to learn, problem-solving ability, creativity, coordination, leadership, the ability to vocalize thoughts, and improved nutrition (through the snacks provided). One interviewee, discussing students’ increased confidence stated, “they are starting to believe that they can succeed.”

“[the kids] are starting to believe that they can succeed.”

Increase student engagement Fourteen interviewees (64%) commented on students' interest in and enjoyment of the after-school program. All stakeholder groups mentioned this theme; however, 43% (n=6) of these comments were made by school-based personnel. Indicators of student interest were students’ demonstrating interest in program activities and giving positive feedback on activities. It should be noted that, for the most part, these indicators of student interest were mentioned in reference to the enrichment activities. Observations of the classroom made by the USC evaluation team back up these indicators noting the “students visually are attentive and are encouraged to keep trying”. Decreased behavior problems This theme was manifested in the interviewees comments related to the program providing a “safe-haven” and “structured environment” for the students during the after-school hours. This is especially important given that research has demonstrated that youth more often engage in high-risk behaviors such as crime (Newman et. al., 2000) and sexual activity (Zill et. al, 1995) during the after-school hours.

Nine

interviewees (41%) from all of the project’s stakeholder groups (schools, the District, the community advisory committee, and community partners) mentioned this as a benefit of the program, though community advisory committee members mentioned this outcome more often than other stakeholders (44% of the nine who mentioned it). Only one interviewee specifically mentioned that the program directly decreased student behavioral problems. Increase parental involvement Nine interviewees (41%), almost half of whom (44%) were school personnel, discussed parental involvement.

The comments were broad and generally positive. A couple of comments focused on the

evening programs for parents and how they have been well received and/or attended. One comment mentioned how the evening programs “fill the need” for parental involvement and alluded to the fact that the evening scheduling helps them to be there. Several additional comments focused on the number and Telling the Story: The Results of the Evaluation of Richland School District One After-School Programs Page 33

strength of the parent liaisons. One interviewee commented on parental involvement being uneven across schools. The interviewee did not perceive this to be a lack of trying on the schools’ part, but more a factor of the number of parents needing to be reached.

UNANTICIPATED OUTCOMES Interviewees discussed several unanticipated outcomes as a result of the after-school program. These included the integration of after-school curricular components in to the day-to-day curriculum, improved teacher-student relationships, increased collaboration, and increased capacity of the community partner organizations. Interestingly, the various stakeholder groups identified different types of unanticipated outcomes. Comments related to the integration of the after-school curriculum into the day-to-day curriculum and improved teacher-student relations were made solely by school-based personnel (n=2). Examples of the types of comments made related to the integration of the after-school curriculum into the day-to-day curriculum were: " “After-school activities are totally integrated into day classes (Math/ELS core),” " “Daily teachers’ team meeting with after-school teachers integrated and coordinated,” " “Curriculum specialists work with after-school objectives related to core classes during day curriculum,” " “Day teachers use their planning periods for students who need after-school program aid,” and " “Teachers using after-school academic curriculum in day classes.”

Improved student/teacher relations were attributed to the “community forum” where students and teachers shared in a “spirit of community” their personal stories. According to the interviewee who mentioned this forum, there was open communication on all sides, and the students “responded to teachers talking from the heart, making emotional connections to their work and how it affects [the students].” As a result, the students were able to “see teachers as people and they respect teachers who give the kids respect.”

...the students were able to “see teachers as people and they respect teachers who give the kids respect.”


To the District, the most salient unanticipated outcome was increased collaboration. This theme was defined as improved collaboration between any two entities to capture the wide array of improved working relationships that the district identified. Sample comments illustrate these collaborative efforts: " “Tighter working relationships with after-school office and office of student support…joined at the hip

to monitor after-school programs; district wide effect felt because of this.” " “Arts-based curriculum for upcoming year have been requested by non-Knight funded schools; cross

pollination of arts to 6 other middle schools.” " “Brought people to table to write grant, “by force brought together” but collaborated to produce

application increased inter-departmental collaboration across District.” Meanwhile, a community advisory committee member and one of the community partners commented on the improved capacity of the partner organizations. These interviewees felt that the capacity of community partners had improved through growth, the opportunity to assess organizational needs and goals, as well as to strategically plan for those needs and goals.

RECOMMENDATIONS GIVEN BY INTERVIEWEES Interviewees made numerous and varied recommendations for improving the program. By and large the recommendations (n=14, 64%) were for overall improvements, most of which would take place at the administrative level, and came from all stakeholder groups. Some interviewees saw the need for more planning to take place, both across schools and within schools. Others felt that assessment and evaluation needed to be further developed with the end goal of making the program a model program that can be replicated.

Five

interviewees mentioned that they felt another person should be hired at the district level.

However, there was some

differences as to what that person’s role should be. Some felt that person should provide administrative support to the Site Coordinators, while others felt that person should be an overall coordinator.

To improve communication among

stakeholders, suggestions were made to have a meeting with all stakeholders present and to have monthly progress reports.

Most recommendations were for overall improvements, most of which would take place at the administrative level.


Other recommendations focused on improving the programming, especially the enrichment piece of it, increasing parental involvement, recruiting and retaining teachers and site coordinators, and improving communication with the Foundation. In terms of improving the program, interviewees wanted to see greater integration of activities with the curriculum (n=3) and offered suggestions for new enrichment activities and/or partnerships. Interviewees (n=3) also expressed the desire for more communication with and involvement from parents and recommended ways in which to do this including having more parent meetings, a handbook, a parent orientation and hosting an art show where the students’ work would be displayed like professional artists’ work. A couple of interviewees also expressed the desire to recruit more teachers while others felt that giving them one paid hour per week to complete paperwork, increasing their pay to $50 per teacher for after school services, or adding after-school work to their contract would help to retain teachers (n=3).

Likewise, suggestions were made to change the site

coordinators to an 11-month contract versus the 12-month contract currently in place (n=2). Lastly, five interviewees (including persons from the community advisory committee, the District, and community partners) expressed the need for more feedback from and communication with the Foundation. Additional Perspectives Provided by Community Advisory Committee Members Understanding of their role Three community advisory committee members were asked about their understanding of their role. All three were in agreement that their role in this project is to oversee the thoughtful distribution of the funds by providing review of proposals to establish after-school programs, to monitor the progress and outcomes of the after-school programs, and to make recommendations to the Foundation. Perceptions of and recommendations for the community advisory committee

CAC members expressed a desire for more interaction with each other.

Community advisory committee members' were asked about their perceptions of the committee including feelings on the breadth of community representation and feelings on frequency of meetings. Again, three community advisory committee members were asked about and responded to these questions. In general, community advisory committee members felt that the committee is comprised of a broad range of community representatives. Also, the interviewees expressed a desire for more interaction with each other, though not necessarily more meetings. One person suggested using email to keep in touch between meetings and another person suggested meeting for 2-3 hours every two months.


Sustainability Four interviewees, all community advisory committee members, discussed the need for the District to make sure the program is sustainable beyond the current funding received from the Foundation. They expressed the

desire

to

see

the

program,

if

successful,

If successful, the program should be institutionalized at the District level.

institutionalized at the District level.

ADDITIONAL EVALUATION MEASURES & METHODS Several additional evaluation measures and methods identified and placed on the Evaluation Measures & Methods Matrix (see Table 2) were not specifically incorporated into the interviews of program stakeholders or by direct observation but are still vital to the overall evaluation of the after-school program. The evaluation findings of the remaining measures are as follows: Student Participation in Enrichment Activities It is clear from the interviews conducted with site coordinators, community partners, and other program stakeholders and through direct observation by the evaluation team, that enrichment activities were planned for the students and that they attended these activities. However, analysis of attendance rosters could not be conducted in time for this report due to a delay in reporting from one of the school sites. PACT Score Increases Analysis of increases in PACT scores could not be conducted because the data were not available at the time this report was generated. It is anticipated that data will be available for analysis in August/September 2004. Increased Parental Involvement Analysis that could document increases in parental involvement in the after-school programs could not be adequately assessed due primarily for two reasons: 1) there is no baseline measure against which end-ofprogram results could be compared and 2) systematic recording of parental participation across the three school sites was not consistent or did not exist.


Decreased Student Absences and Discipline Referrals Analysis of attendance rosters could not be conducted in time for this report due to a delay in reporting from one of the school sites. Additionally, the District does not maintain databases of discipline referrals at the student-level. In preparation for the next funding year, school administrative officials will need to meet with evaluation staff to create a consistent tracking system of discipline referrals to detect any changes in the number of discipline referrals among after-school program participants. Provide Professional Development Activities for Site Coordinators and Teachers Site Coordinators participated in at least one external professional development activity during the project period (attendance at a conference hosted by the South Carolina After-School Alliance in March 2004). In addition, Site Coordinators participated in several District– and school-sponsored professional development programs. Documentation on the attendance of after-school teachers in professional development activities was not collected.


DISCUSSION METHODOLOGICAL SCOPE & LIMITATIONS The following methodological considerations have a direct bearing on the findings of this evaluation. Timeframe The after-school programs funded by the Foundation were not in place until late Fall 2003. In addition, we did not begin our evaluation efforts until November 2003. The late start of both implementation and evaluation activities restricted what could be reasonably accomplished before the close of the school year in May 2004.

Consequently, the focus of the evaluation was limited to process level measures and

procedures. Although essential and important in and of itself, process level evaluation does not allow for emphatic judgments about the programs’ efficacy or effectiveness in achieving the desired outcomes. Moreover, because of the nature and complexity of the outcomes themselves (i.e., increases in standard test scores, increases in core course grades, increases in student retention, and decreases in discipline referrals and suspensions) an intervention period of longer than part of one academic year will be necessary to document change. Qualitative Data The data collected in this evaluation was qualitative and has inherent limitations.

The sample of

interviewees was small (n=21) and the results, therefore, are only representative of the after-school program being evaluated.

Furthermore, although all key stakeholders were identified and invited to

participate, not all group types were equally represented in the interviews. For example, several community partner organizations did not respond to our requests to be interviewed. Despite the best efforts of trained, external interviewers, some stakeholders may not have felt comfortable in truthfully responding to the questions asked.

Although rigorous protocols were applied, the analysis and

interpretation of the interview data (i.e., text units) will always include a small degree of subjectivity. Evaluation Research Design The absence of a research design limits the extent to which any observed effects (e.g., changes in outcomes) can be attributed to the portion of the after-school program funded by the Foundation in the three middle schools. The implementation of cross-sectional qualitative measures and procedures (instead


of quasi-experimental or experimental research designs) yields rich insight about the structure and operations of the programs. In order, to make statements of the programs’ impact, a research-based experimental study design will need to be employed. Lack of Program Distinction All three of the programs had existing after-school programs in place prior to receipt of John S. and James L. Knight Foundation funding. The “mix” of existing and new Foundation-funded enrichment activities presented evaluation challenges. Often, it was difficult to parse-out the origination of activities and to determine if programming was being implemented in the manner outlined in the District’s proposal. The use of experimental controls or comparison schools matched to the intervention schools would provide a “cleaner” evaluation of enrichment activities.

SUMMARY OF KEY FINDINGS In summary, several key findings of the Richland School District One after-school program were identified through a combination of evaluation strategies: the development and analysis of the logic model and measures and methods matrix, stakeholder interviews, and direct observations by the evaluation team. The key findings were both positive and negative and address both programmatic and administrative issues. Lack of fidelity to the proposed project timeline. Obtained through both the interviews conducted with stakeholders at all levels and direct observation of program activities, there was a lack of adherence to the timeline stated in the original grant proposal and subsequent timelines developed after the project had begun. The hiring of the site coordinators after the beginning of the academic year impaired their ability to appropriately plan for activities and garner community support. Ambiguous District role regarding leadership and project management. It was unclear to the majority of interviewees and to the evaluation team during the course of direct observation as to the role of the District in providing leadership and overall project management. The Richland School District One Office of Grants & Planning was identified in the grant proposal as the chief administrative body of the program but given the nature of the activities (providing after-school programming), the more logical placement of responsibility for grant oversight seemed to both the interviewees and the evaluation team to be the Office of After-School Programs. Telling the Story: The Results of the Evaluation of Richland School District One After-School Programs Page 40

Inconsistent and varying program activities across the three after-school sites In part, due to the ambiguous role of the District in leadership in project management, there was no centralized mechanism that coordinated the efforts of the three after-school sites. Even though site coordinators, District staff, the evaluation team, and other stakeholders participated in regular meetings during the second half of the program, the opportunity for synergy across the three after-school sites was missed and instead, individual sites developed and implemented after-school activities on their own. The result of this was an inconsistent exposure to academic and enrichment programming across the three after-school sites which further impaired the ability to conduct outcome-based evaluation. Ambiguous program activities linked to unreasonable outcome expectations The two proposals that were funded by the Foundation contained ambiguous program activities and unreasonable outcome expectations. Significant challenges surfaced in designing an evaluation methodology that sought to measure the process, impact, and outcome of the program due to stated activities that were unclear or not linked to evidenced-based best practices, stated objectives that were not measurable, and program goals that were too distal and not adequately supported by the program. High administrative burden placed on Site Coordinators Site Coordinators devoted a significant amount of time to administrative activities and were not able to (but are qualified to) devote enough time toward developing and monitoring community partnerships, managing program activities, facilitating parental involvement and support, coordinating with other afterschool sites, locating alternative funding sources necessary for program sustainability, and other essential functions of a program manager. Integration of after-school curricular materials into the day-time curriculum Due to the success many after-school teachers experienced and communicated to other faculty, components and activities developed and used in the after-school program have also been used in day-time classes. This key finding is an unanticipated outcome and speaks directly to the quality of programming developed by the Site Coordinators, after-school teachers, District curriculum staff, community partners, and other program stakeholders. Improved student/teacher relationships An additional unanticipated outcome of the after-school program was the evolution in the relationship between students and teachers. Many stakeholders involved in the day-to-day operations of the afterTelling the Story: The Results of the Evaluation of Richland School District One After-School Programs Page 41

school programs observed and commented on how both the students’ perceptions of teachers and the teachers’ perceptions of the students changed during the course of the program. The additional time both the teachers and students spent with each other during the course of the program seemed to facilitate more understanding and respect. Improved student engagement During all site visits, it was recorded that students were highly engaged and responded with enthusiasm to the after-school academic and enrichment activities. It was clear that through the use of various interactive and experiential instructional methods, the students were more attentive and responsive to learning—both from the teacher (or community partner) and from each other. Despite the fact that students who are mandated to participate in programs “on their own time” are more challenging learners, no behavioral problems (including inattentiveness, sleeping, disruption, etc.) were observed during site visits and teachers, site coordinators, and school personnel commented on how engaged students were in the after-school program. Increased opportunities for students and parents to attend cultural events A commonly cited benefit of the after-school program was the increase in opportunities for students and parents to participate in cultural events that they would not otherwise have the opportunity in which to participate. Parents and students had the opportunity to participate in pop concerts, ballet performances, operas, museum shows and exhibits, and dance recitals which expanded their understanding and appreciation of the arts. Inventive, motivated, and dedicated teachers Obtained from both the data extracted from interviews with program stakeholders and from site visits conducted by the evaluation team, it was clear that the teachers in the after-school program were inventive, motivated, and dedicated to providing high quality activities to improve students’ overall academic performance.


Ability of Site Coordinators Despite numerous obstacles and barriers, the Site Coordinators were effective in quickly planning and immediately coordinating program activities that fused core academic components with enrichment activities. The support received from several key school personnel at each of the after-school sites (the principal, the curriculum resource teachers, after-school teachers, etc.) aided Site Coordinator’s ability to overcome challenges they faced in rapidly delivering the after-school program. Support of the Foundation Lastly, one major key finding identified by the evaluation team was the Foundation’s continued support and timely responsiveness to the needs of program stakeholders, as evidenced by frequent in-person meetings by the Community Liaison Program Officer, Mr. Alfredo Cruz. As mentioned in this evaluation report, significant barriers were faced by program stakeholders, but through on-going support provided by the Community Liaison Program Officer and the flexibility of the Foundation to adjust to changing conditions, personnel at Richland School District One and the staff implementing the after-school programs were better equipped to maintain project operations.

USC RECOMMENDATIONS Through analysis of recommendations made by all project stakeholders, the direct observation of the evaluation team, and the information obtained through a search of the scientific literature for evidencedbased best practices, the evaluation team proposes several recommendations: Incorporate the Office of After School Programs into project management Most of the findings involving the management and operations of the programs, can be addressed and corrected by implementing adequate planning procedures, process evaluation measures, and by dedicating appropriate resources (i.e., financial and personnel) to the oversight and coordination of the after-school programs through the Office of After School Programs. Formalize the relationship between community partners and the after-school program Implement policies and procedures for identifying, engaging, and monitoring activities of community partners to include: memoranda of understanding, regular reports of program activities (standardized for all sites and partners), and on-site observations of organizations activities and services provided.


Devote more planning to facilitate parental involvement and support Develop an organized plan and dedicate the resources necessary to increase parental involvement and support of after-school program activities. Improve program design Essential program evaluation design elements (i.e., quasi-experimental study designs with comparison groups, quantitative and qualitative data collection methods, standardized monitoring and report forms and mechanisms, etc.) should be developed and in-place at prior to the beginning of the program so that adequate measures can be taken at required intervals to provide valid and reliable evidence of program outcomes and to establish program effectiveness. Conduct formative evaluation Formative evaluation should be conducted prior to the full delivery of the after-school programs to assess the degree to which the programs incorporate evidence-based best practices identified in the scientific literature. Critical gaps could be identified and remedied which will increase the likelihood of achieving maximum impacts.


REFERENCES Basch, C. E., Sliepcevich, E. M., Gold, R. S., Duncan, D. and Kolbe, L. (1985). Avoiding Type III errors in health education program evaluation: A case study. Health Education Quarterly 12(4), 315-331. Bhatnagar, B., & Williams, A. C. (Eds.). (1992). Participatory development and the World Bank: Potential directions for change, World Bank discussion paper 183. Washington, DC: The World Bank. Coffman, J. (2003). Logic models in out-of-school time. The Evaluation Exchange. Vol. IX, No. 1. Harvard Family Research Project, Harvard Graduate School of Education, Cambridge, MA. Durning, D. (1993). Participatory policy analysis in a social service agency: A case study. Journal of Policy Analysis and Management, 12(2), 297-322. Fetterman, D., Kaftarian, S., & Wandersman, A. (Eds.). (1995). Empowerment evaluation: Knowledge and tools for self-assessment and accountability. Beverly Hills, CA: Sage Publications. Glaser, B.G. & Strauss, A.L. (1999). The discovery of grounded theory: Strategies for qualitative research. New York, NY: Aldine de Gruyter. Hall, G., Yohalem, N., Tolman, J., Wilson, A. (2003) How Afterschool Programs Can Most Effectively Promote Positive Youth Development as a Support to Academic Achievement: A Report Commissioned by the Boston AfterSchool for all Partnership. Kirby, D. (2001). BDI Logic models: A useful tool for designing, strengthening and evaluating programs to reduce teen pregnancy. Santa Cruz, CA: ETR Associates. Kirby, D., Lezin, N., Afriye, R., & Gallucci, G. (2003). Preventing teen pregnancy youth development and afterschool programs. Santa Cruz, CA: ETR Associates.


National Institute on Out-of-School Time. (2000). Making an Impact on Out-of-School Time: A guide for corporation for national service programs engaged in after school, summer, and weekend activities for young people. Wellesley, MA: The National Institute of Our-of-School Time. Newman, S.A., Fox, J.A., Flynn, E.A, and Christeson, W. (2000). America’s After-School Choice: The Prime Time for Juvenile Crime, or Youth Enrichment and Achievement. Washington, DC: Fight Crime: Invest in Kids. Philliber, W. (2004). Evaluating your adolescent pregnancy primary prevention program. Workshop presented at the 5th Summer Institute of the South Carolina Campaign to Prevent Teen Pregnancy, Charleston, SC, June 2004. QSR International Pty Ltd. (2000). NVivo® qualitative data analysis program Version 1.2. Melbourne, Austrialia:, QSR International Pty Ltd. South Carolina Department of Health and Environmental Control (SC DHEC). (2003). The centered evaluation guide. Columbia, SC. Miller, B. M. (2003). Critical Hours: Afterschool Programs and Educational Success. Quincy, MA: The Nellie Mae Foundation. Steckler A ,Goodman RM (1989). How to institutionalize health promotion programs. Am J Health Promot 3: 34-44. Steckler A & Linnan L. (2002). Process evaluation for public health interventions and research. San Francisco, CA: Jossey-Bass. Stewert, D.W. & Shamdasni, P.N. (1994). Focus groups: A practical guide for applied research. Thousand Oaks, CA: Sage Publications. Strauss, A.L. & Corbin, J. (1990). Basics of qualitative research. Thousand Oaks, CA: Sage Publications.


Windsor, R., Baranowski, T., Clark, N., & Cutter, G. (1994). Evaluation of health promotion, health education, and disease prevention programs. (2nd ed). Mountain View, CA: Mayfield Publishing Co., Inc. Weiss, C.H. (1998). Evaluation: Methods for studying programs and policies. (2nd ed.). Upper Saddle River, NJ: Prentice Hall. W.K. Kellogg Foundation. (2001). Using logic models to bring together planning, evaluation and action: Logic Model Development guide. Battle Creek, MI: W.K. Kellogg Foundation. Zacharakis-Jutz, J., & Gajenayake, S. (1994, July/August). Participatory evaluation's potential among nonprofit organizations: The Rockford, Illinois project. Adult Learning. Zill, N., Nord, C.W., and Loomis, L.S., (1995). Adolescent Time Use, Risky Behavior and Outcomes: An Analysis of National Data. A report by Westat, Inc. for the Office of the Assistant Secretary for Planning and Evaluation, U.S. Department of Health and Human Services. Washington, DC.


APPENDICES

APPENDIX A: INTERVIEW DISCUSSION GUIDE FOR SCHOOL PERSONNEL

Telling the Story: The Results of the Evaluation of Richland School District One After-School Programs

Page 1 of 1

SCHOOL PERSONNEL Knight Foundation/Richland School District One After-School Program “Personnel” Interview Guide

1. How do you think the Knight/Community Partnership after-school program is going so far?

2. What are the strengths of the current after-school program?

3. Weaknesses?

4. How do you feel the students/volunteers are responding to program activities?

5. What effects do you think the program has had upon the students?

6. What unanticipated outcomes have you noticed during the implementation of the after-school program?

7. How is the classroom curricula perceived by the students? Teachers? Parents?

8. What are the barriers you have encountered? Recommended solutions/ideas for improvement?

9. Anything else you want to talk about?

APPENDICES

APPENDIX B: INTERVIEW DISCUSSION GUIDE FOR COMMUNITY ADVISORY COMMITTEE (CAC)


Page 1 of 1

COMMUNITY ADVISORY COMMITTEE (CAC) Knight Foundation/Richland School District One After-School Program Interview Guide

Questions Related to the After-School Programs 1. How do you think the Knight/Community Partnership after-school program is going so far?


3. Weaknesses?

Questions Related to the Community Advisory Committee 4. What is your understanding of your role as a member of the Knight Foundation Community Partners’ Advisory Committee?

5. In what ways have you been able or unable to fulfill your role as a Committee member?

6. Do you feel the Community Partners’ Advisory Committee is comprised of a broad range of community representatives? Explain.

7. Do you feel the Community Partners’ Advisory Committee meets often enough? Suggestions for changes?

8. What other comments or suggestions do you have regarding the CAC in general, the after-school programs, and/or the Knight Foundation’s role.

APPENDICES

APPENDIX C: INTERVIEW DISCUSSION GUIDE FOR COMMUNITY PARTNERS


Page 1 of 1

COMMUNITY PARTNERS Knight Foundation/Richland School District One After-School Program Interview Guide 1. How do you think the Knight/Community Partnership after-school program is going so far?


3. Weaknesses?

4. How do you feel the students/volunteers are responding to program activities?

5. What effects do you think the program has had upon the students?

6. What unanticipated outcomes have you noticed during the implementation of your organization’s after-school program activities?

7. How do you think your program’s after-school activities are perceived (received) by the students? Parents? PROBE:

Give specific examples of several activities and how they are perceived by students and parents.

8. What are the barriers you have encountered? PROBE: Recommended solutions/ideas for improvement?

9. Anything else you want to talk about?

CONTACT INFORMATION

For more information about this report, contact: Office of Public Health Practice Arnold School of Public Health University of South Carolina Health Sciences Building, Room 102 800 Sumter Street Columbia, SC 29208 803/777-1410 voice 803/576-5950 fax

OFFICE OF PUBLIC HEALTH PRACTICE Putting Innovation to Work

final report revised 081204

final report revised 081204

Suggest Documents

Revised Final Report ESIF-2_20090116

C:\MrPTP\Final Report Phase I\Revised Final Draft ...

Final Revised Paper

Final Revised Paper - Biogeosciences

Final Revised Paper - Biogeosciences

Final Thesis Revised

Final Revised Paper

Final Revised Paper

Final Revised Paper

Rethinking hyperglycemia REVISED-FINAL

Final Revised Paper

PH_Manuscript - Revised final

Final Revised Paper

Revised Thesis FINAL

Final Report Final Report

final final report report - Data.unhcr.org

SPM Final Report-08 Dec-Revised-Layout Change21-Dec

Class VIII Report Revised Final 13th Sep.indd - Google Sites

SHADAC Revised Final SIM Minnesota Evaluation Report Executive ...

EAP 3J Final Report revised - Essex Autonomy Project - University of ...

Final Report Mongolia 28 Jan 2011 revised docx - UNIDO

21.02.2017-revised final report 2016-17 Palampur ...

Camera Ready Copy SRP Revised Final Report ... - University of Leeds

SHADAC Revised Final SIM Minnesota Evaluation Report Executive ...