considered in evaluating youth recreation programs are discussed. Determining whether ... problem behaviors youth display. Program goals also include ...
Evaluating youth recreation programs Witt, Peter A., John Crompton, and Dwayne Baker. "Evaluating youth recreation programs. " JOPERD--The Journal of Physical Education, Recreation & Dance. 66.n4 (April 1995): 27(4). Abstract: Park and recreation departments (PARDs) are primary community resource for promoting mental health and reducing the impact of negative social and economic issues on the youth. However, PARDS must develop effective strategies for evaluating their youth recreation programs to ensure that goals are being met. Program evaluations should focus on concrete positive outcomes and the effectiveness of program processes. Other key elements that should be considered in evaluating youth recreation programs are discussed. Determining whether program goals are achieved requires that well-thought-out evaluation methods are developed as programs are planned. Park and recreation departments (PARDs) are expected to provide myriad, high-quality youth services. Stakeholders - taxpayers, participants, policy makers, and parents of participants - want to be assured that the services and programs provided are worth the money invested. To secure future program funding, PARDs must demonstrate that their programs and services benefit users and meet the needs of the community. Thus, a clear strategy for program evaluation must be developed as programs are planned. Nowhere is the need for quality programs and effective evaluations more in demand than in the area of programs for youth, specifically, for youth considered at-risk. Along with other community agencies, PARDs are increasingly under pressure to provide services that will help decrease inappropriate behavior and increase positive attitudes and functional abilities of youth. After-school programs, youth sport programs, summer camps, and other types of activity programs are planned with the purpose of preventing or reducing the frequency and intensity of problem behaviors youth display. Program goals also include increasing participants' knowledge about appropriate recreation opportunities; changing attitudes toward the consequences of joining gangs; and increasing the repertoire of activity skills. Some programs are aimed at reducing criminal activity, drug and alcohol usage, school dropout rates, and teen pregnancy. Determining whether program goals are achieved requires well-thought-out evaluation methods. However, too often, formal evaluation of the prevention or intervention impact of program efforts is lacking. Many times, claims of program success are based on an increased number of participants each year; increased funds allocated to the program; or participants' and/or their parents' enjoyment of the program. While these reasons provide important program outcome information, evaluations must show changes in participant attitudes, knowledge, and/or behaviors as a result of program participation. Providing more substantial information about program outcomes requires several important changes in the way programs are planned, conducted, and evaluated. PARDs must focus on: (1)
establishing clear program goals based on an understanding of the problems and issues faced by the target population; (2) designing programs aimed at achieving the goals; and (3) formulating an evaluation plan to determine whether goals have been met. Understanding Participants' Needs Well-conceived program goals are necessary for successful program evaluation. In turn, goals must be based on a clear understanding of the issues and problems faced by the target population. For example, what are the needs of at-risk youth that can be addressed through participation in recreation programs? To answer this question, programmers need to select a working model that outlines the factors that influence inappropriate or negative attitudes and behaviors of youth in their communities. The model will guide program development so that programmers can prevent problems from arising or can intervene when problems arise. The discussions of "resilience" in the works of Jessor (1992; 1993), Luthar (1991), Masten et al., (1990), and Rutter (1990) may provide an example of a model that could help guide the development of objectives for programs targeting at-risk youth. Personal, family, economic, and/or community circumstances (risks) increase the likelihood of negative behaviors, such as dropping out of school or joining a gang. Some children exposed to these risks will exhibit problematic behaviors, while others will not. This is because protective factors "moderate, buffer, and insulate against" (Jessor, 1992, p. 30) the impact of risk on certain individuals even when they are exposed to the same risks. Risk factors such as the lack of attention and guidance inherent in a dysfunctional family may be mediated by the existence of an interested and caring mentor in the extended family or elsewhere in the community. Role models who reinforce inappropriate behavior can be replaced by positive role models and programs emphasizing the development of behavioral controls and personal responsibility. In addition, individuals who have low levels of self-esteem because they do not perform well in school, may be able to develop feelings of self-worth through involvement in community-based activities. Looking at individuals who are successful, despite the adversity that surrounds them, offers a basis for developing recreation-based programs which (1) provide caring adults as program leaders; (2) contain rule structures that reinforce appropriate modes of behavior; (3) promote involvement in appropriate activities; and (4) help individuals develop feelings of competence and self-efficacy, which may lead to improved feelings of self-worth. In addition, recreation programs may provide a substitute for the feeling of "belonging" afforded by gang membership by offering positive opportunities for group or individual involvement and accomplishment outside the gang structure. Evaluation Strategies Traditionally, recreation programs have couched objectives in terms of participation (e.g., a youth basketball program provides recreation opportunities for 1,500 adolescents). However, a formal evaluation specifying benefits is necessary; for example, as a result of participation in the youth basketball program, self-esteem will increase and specific physical skills will be improved.
Drawing on a strong conceptual foundation such as the resilience paradigm will help identify the attitudes, behaviors, abilities, and knowledge which programs should affect. Carefully crafted goals will also guide the program design process (i.e., program content, leadership style, rule structure, etc.). After a strong conceptual foundation, program objectives, and program structure have been developed, specific evaluation strategies must be designed. Past evaluation efforts have relied on: head counts of participants; anecdotal evidence or solicited testimonials from participants, parents, or program organizers; or findings based on correlational evidence; for example, statements by police personnel that crime in a given neighborhood has declined since a particular program was initiated (Pizor, 1992). Evaluations which use these sources are increasingly questioned, especially given the accountability concerns described. Thus, more systematic strategies must be used and should include: (1) outcome evaluations to determine changes in specific behaviors, abilities, or attitudes (i.e., polling parents, participants, policy makers, and program leaders to determine the match between their expectations for a given program and their perceptions of program outcomes); and (2) process evaluations of program goals, content, leadership, management, facilities, etc., to determine the match between desired program characteristics and actual program operation. Outcome evaluations might include collecting baseline data for the relevant variables before the program is implemented, and then collecting data again after the program ends to see if changes have occurred. However, to rule out a number of competing explanations for any changes from the pre- to the post-collection of data, tracking changes (if any) that may take place in a comparison group of individuals who are not involved in the program is also necessary. For example, changes could be caused by maturation of subjects and not program participation; or perhaps, subjects are involved in other programs which are the actual cause of any observed changes. At Texas A&M University, the following guidelines have been established for selecting programs as candidates for evaluation efforts. Good program candidates must have the potential for: * establishing a control group (i.e., establishment of a program waiting list or identification of non-participants from the same school classes as program participants); * separating out the program's impact, relative to all other types of programs and services in which clients are involved (i.e., special school programs or programs sponsored by other agencies); * appropriate pre- and post-testing of the target and control groups; and * collecting appropriate data (i.e., grades; results of standardized achievement tests; or juvenile justice incident, arrest, or probation records) from the files of other organizations (i.e., police department, school district, health department).
Specific evaluation strategies must be developed for measuring the effects of each targeted program. Evaluation strategies should encompass both quantitative and qualitative approaches. Outcomes can be measured using standardized, or if necessary, specifically developed instruments. These data should be augmented by information from participants, focus groups, parents, and police, juvenile justice system, or school district records. For example, parents can be asked, "As a result of your child's participation in a youth basketball program, did particular behaviors, abilities, or attitudes change?" To accurately interpret the results, programmers should collect and analyze information concerning background variables such as socioeconomic status, ethnicity, age, gender, and a "degree of at-risk" measure. Process evaluations address the program's goals and help identify key factors (i.e., program leadership, parental involvement, program structure) which lead to the outcomes. They also help identify factors which explain why some recreation intervention programs are more successful than others. Process evaluation is accomplished through questionnaires and interviews with key program personnel, advisory committee members, parents, and the participants themselves. In addition, actual program observations can be included to determine whether programs are being conducted in an optimal manner. Existing records regarding the program can also be reviewed. Measures of program quality might be a useful part of process evaluation. For example, using "quality service" models developed in the marketing literature (Zeithaml, Parasuraman, & Berry, 1990), questionnaires can be constructed which assess parent or participant evaluations of program tangibility (appearance of the program's physical facilities, equipment, personnel, and communication materials), reliability (ability to provide dependable and accurate service), responsiveness (willingness to help participants and provide prompt service), assurance (knowledge and courtesy of the employees and their ability to convey trust and confidence), and empathy (the caring and individualized attention the program provides to participants). Tackling the Issues While numerous books have been written on evaluation (e.g., Herman, 1987), it is an art as much as a science. Unlike laboratory investigations, where the subjects, environment, and data collection procedures can be carefully controlled, collecting data about behaviors, values, and attitudes in a community environment is more difficult to control. Difficult issues that must be tackled include (1) earning the confidence of stakeholders to make the evaluation meaningful (i.e., some stakeholders may agree to participate in the evaluation only if the process will politically help the program, but may be unwilling to supply any negative information that would hurt the program); (2) developing or identifying reliable and valid data collection instruments; and (3) securing the cooperation of persons from whom the data are to be collected. In addition, objectives which do not readily lend themselves to measurement make it difficult to determine whether program actually achieves behavioral, ability, or attitudinal changes. Finally, even when program objectives are clear, programs are often not designed to achieve the stated outcomes successfully. For example, leaders may not be focused on behavioral processes; too many children might be in the program to affect any one participant; or the program, even in the
hands of a skilled leader, might not provide an appropriate environment for the desired goals to be achieved. For programs to be successful, recreation personnel who are capable of translating community needs into desired objectives and creating programs that can lead to achieving the objectives need to be hired and trained. In addition, these individuals must work closely with professionals from other community agencies to create an overall, coordinated community approach to providing services. Future program and evaluation initiatives will probably cut across departmental lines, as well as involve the public, private, and voluntary sectors. Taking all the aforementioned factors into account, there is a strong need for careful planning of evaluation strategies, involvement of individuals who are knowledgeable about how to undertake the evaluation process, and "evaluation" of the evaluation process itself to make sure it is well conceived and appropriate for its intended purpose. While the need for systematic processes of program planning and evaluation are currently emphasized in many undergraduate recreation curricula, at conferences and workshops, and in professional publications, there is strong reason to suggest that these skills have not been fully implemented into professional practice. However, with the use of appropriate evaluation strategies, it should be possible to demonstrate the value of PARD programs in providing alternatives to unsupervised after-school and weekend experiences, which have the potential to draw youths into problem activities. Through these programs, PARDs can help youth develop the resources and resilience necessary to resist some of the pressures and influences that abound in their home and neighborhood interactions. Information produced through program evaluation processes will not only lead to program improvement, but will also help provide the necessary evidence to better position recreation and park services as vital elements in youth programming. References Herman, J. (Ed.) (1987). Program evaluation kit (2nd ed.). Newbury Park, CA: Sage Publications. Jessor, R. (1992). Risk behavior in adolescence: A psycho social framework for understanding and action. In Adolescents at Risk: Medical and Social Perspectives (Chapter 3, pp. 19-35). Boulder, CO: Westview Press. Jessor, R. (1993). Successful adolescent development among youth in high-risk settings. American Psychologist, 48(2), 117-126. Luthar, S. (1991). Vulnerability and resilience: A study of high-risk adolescents. Child Development, 62, 600-616. Masten, A., Morison, P., Pellegrini, D., & Tellegen, A. (1990). Competence under stress: Risk and protective factors. In J. Rolf, A.S. Masten, D. Cicchetti, K.H. Nuechterlein, & S. Weintraub
(Eds.), Risk and protective factors in the development of psycho pathology. Cambridge: Cambridge University Press. Pizor, A. (1992). Preventing delinquency: An assessment of park and recreation actions. Arlington, VA: National Recreation and Parks Association. Rutter, M. (1990). Psychosocial resilience and protective mechanisms. In J. Rolf, A.S. Masten, D. Cicchetti, K.H. Nuechterlein, & S. Weintraub (Eds.), Risk and protective factors in the development of psycho pathology. Cambridge: Cambridge University Press. Zeithaml, V., Parasuraman, A., & Berry, L.L. (1990). Delivering quality service: Balancing customer perceptions and expectations. New York: The Free Press. RELATED ARTICLE PARDs are a primary community resource for improving the mental health of program participants and reversing the negative consequences of some of the family, economic, and community issues facing today's youth. PARDs are in a unique position to provide services for at least three reasons: * Recreation centers and park areas (where many gangs and at-risk youth congregate) are distributed widely across communities and can be used as service centers for dealing with gang and youth-related problems. * PARD personnel are experienced in establishing empathetic relationships with their clients. While not well documented through evaluation studies, appropriately designed recreation programs (i.e., those involving good mentoring, the opportunity to improve physical and social skills, and the opportunity for participants to substitute positive/socially acceptable activities for activities that violate community standards) hove the potential to improve self-concept, increase participants' feelings of acceptance and belonging, and improve attitudes about the future. * Recreation activities are inherently appealing to large segments of the youth population, and thus offer a vehicle for positively influencing psychological well-being and subsequent behavior. Peter A. Witt is a professor and department head, John Crompton is a professor, and Dwayne Baker is a research assistant in the Department of Recreation, Park and Tourism Sciences at Texas A&M University, College Station, TX 77843-2261.
