evaluating the implementation of an olympic education program in ...

Review of Education (2005) 51:427–438 DOI 10.1007/s11159-005-8259-2

Ó Springer 2005

EVALUATING THE IMPLEMENTATION OF AN OLYMPIC EDUCATION PROGRAM IN GREECE VASILIOS GRAMMATIKOPOULOS, NIKOLAOS TSIGILIS, ATHANASIOS KOUSTELIOS and YANNIS THEODORAKIS

Abstract – The aim of this study was to develop an instrument for evaluating how an education program has been implemented. Such evaluation can provide insight into the effectiveness of a program. Examined here was the Olympic Education Program used in Greek schools since 2000. In it, students learn the history of the Olympic games and the importance of exercise for health along with the principles and values of sports and volunteerism. The evaluation instrument underlying this study addressed the following six factors: ‘facilities’, ‘administration’, ‘educational material’, ‘student–teacher relationships’, ‘educational procedures’, and ‘training’. Results indicate that the instrument, while adequate for assessing effectiveness, should be combined with advanced statistical methods. Zusammenfassung – EVALUATION DER EINFU¨HRUNG EINES OLYMPISCHEN BILDUNGSPROGRAMMES IN GRIECHENLAND – Das Ziel dieser Studie war, ein Instrumentarium zu entwickeln, um zu evaluieren, wie ein Bildungsprogramm eingefu¨hrt worden ist. Eine solche Evaluation kann Einsichten in die Wirksamkeit eines Programmes gewa¨hren. Gepru¨ft wurde hier das Olympische Bildungsprogramm, mit dem an griechischen Schulen seit dem Jahr 2000 gearbeitet wurde. Darin lernen die Schu¨ler die Geschichte der Olympischen Spiele und die wichtige Bedeutung der U¨bung fu¨r die Gesundheit im Verbund mit den Prinzipien und Werten von Sport und freiwilliger Arbeit kennen. Das Evaluationsinstrumentarium, welches dieser Studie zugrunde liegt, sprach die folgenden sechs Faktoren an: ,Einrichtungen‘, ,Verwaltung‘, ,Bildungsmaterial‘, ,Schu¨ler-Lehrer-Beziehungen‘, ,Bildungsverla¨ufe‘ und ,Ausbildung‘. Die Ergebnisse zeigen an, dass das Instrumentarium, wa¨hrend es fu¨r die Einscha¨tzung der Wirksamkeit angemessen ist, mit fortgeschrittenen statistischen Methoden kombiniert werden sollte. Re´sume´ – E´VALUER LA MISE EN ŒUVRE D’UN PROGRAMME D’E´DUCATION OLYMPIQUE EN GRE`CE – Le but de cette e´tude e´tait de de´velopper un instrument capable d’e´valuer la fac¸on dont un programme d’e´ducation avait e´te´ mis en œuvre. Une telle e´valuation peut fournir un aperc¸u de l’effectivite´ d’un programme. C’est le Programme d’E´ducation Olympique mis en place en Gre`ce depuis 2000 qui a fait ici l’objet d’un examen. Dans ce programme, les e´tudiants apprennent l’histoire des jeux Olympiques et l’importance de l’exercice pour la sante´, en meˆme temps que les principes et les valeurs du sport et du volontariat. L’instrument d’e´valuation sous-tendant cette e´tude a aborde´ les six facteurs suivants: ‘ame´nagements’, ‘administration’, ‘mate´riel e´ducatif’, ‘relation de l’e´tudiant a` l’enseignant’, ‘proce´de´s e´ducatifs’, et ‘entraıˆ nement’. Les re´sultats montrent que l’instrument, bien qu’adapte´ pour ce qui est de l’effectivite´ de l’estimation, devrait eˆtre combine´ a` des me´thodes statistiques avance´es.

428

Vasilios Grammatikopoulos et al.

Resumen – EVALUACIO´N DE LA IMPLEMENTACIO´N DE UN PROGRAMA DE EDUCACIO´N OLI´MPICA EN GRECIA – El objetivo de este trabajo ha sido desarrollar un instrumento para evaluar co´mo se ha implementado un programa educativo. Esta evaluaci o´n puede proporcionar una idea de la eficacia de un programa. En este caso, se ha examinado el Programa de Educacio´n Olı´ mpica que se esta´ usando en las escuelas griegas desde el an˜o 2000. En este programa, los estudiantes aprenden la historia de los Juegos Olı´ mpicos y la importancia que tienen los ejercicios fı´ sicos para la salud, junto con los principios y los valores del deporte y del voluntariado. El instrumento de evaluacio´n sobre el que basa este estudio se centra en los seis factores siguientes: ‘oferta’, ‘administracio´n’, ‘material educativo’, ‘relaciones estudiante–docente’, ‘procedimientos educativos’ y ‘capacitacio´n’. Los resultados indican que el instrumento, si bien es adecuado para evaluar la efectividad, deberı´ a combinarse con me´todos estadı´ sticos de avanzada.

Evaluation of a program’s implementation is a significant part of educational evaluation (Provus 1971). It helps educators to discover the weak aspects of a program (Dimitropoulos 1998), and could point out elements that are decisive for a program’s success. Revealing the factors that affect a program’s implementation is the first step towards a properly evaluative procedure. Solomon (1998) has pointed out that an educational procedure has a specific structure, and certain factors play an important role in the implementation of any educational program. This structure consists of: (a) school data, (b) educational procedures, and (c) outcomes (Barr and Dreeben 1983; Willms 1992). ‘School data’ refers here to school facilities, curriculum and books, and management. School facilities play an important role in the implementation of an educational program, as Vasilakis (1995) and Germanos (1996) have reported. The same holds for the books provided to support teaching (Skilbeck 1984; Vretos and Kapsalis 1997), as well as the management of

Implementation of an Olympic Education Program

429

the school (Pashiardis and Orfanou 1999; Eliophotou-Menon 2002). ‘Procedures’ includes the quality factors of relationships and educational content. ‘Relationships’ between teachers and students (Birch and Ladd 1997; Pianta 1998; Kesner 2000), among teachers, and between a school and its local community (Chapman et al. 2002) are of great importance in the implementation of an educational program. Equally important is the program’s educational procedure (educational strategies and teaching) (Hargreaves and Woods 1984). Finally, ‘outcomes’ are the program’s goals, the students’ performance, and the acceptance of the program. Additionally, educational training is considered an integral part of a program’s implementation (Sparks and Hirsh 1997; Guskey 2000). Evaluation of educational training should be carried out immediately after or during the program’s implementation, for only then is it possible to judge the effectiveness of training (Guskey 2000). Summarizing the foregoing, these six factors are considered to play decisive roles in implementation of an educational program: school facilities, school administration, relationships, educational procedure, books and educational materials to support teaching, and training. Evidently, many other factors, such as a program’s goals, students’ performance, and efficacy of teaching staff, influence planning and outcomes. These factors were not treated in the present study, as it focuses only on factors that play a determinate role in a program’s implementation. The innovative physical education program reported on in the present study is the ‘‘Olympic Education Program’’ (OEP) implemented in all Greek schools (primary and secondary, in all grades) since 2000. Goals of the OEP are that students: (a) learn the history of the Olympic Games, (b) uphold the principles and values of sports (fair play, respect for rules, respect for fellow athletes, etc.), (c) understand the importance of the Games, (d) realize the essential role of volunteerism, (e) develop skills related to the control of one’s behavior during sporting events and in life, and (f) learn the importance of exercise for health (Kioumourtzoglou et al. 2001). Moreover, through OEP activities, students are to become familiar with Olympic sporting events that are not well known in Greece, as well as with the spirit of the Paralympic Games. About 2000 physical education teachers are working on this project. The OEP makes use of modern educational approaches. Integrated curriculum (motor, theoretical, and cultural activities) and learning through projects are integral components, students being at the center of the educational process. Nevertheless, Olympic education is not a novel idea. Several approaches to Olympic education in school curricula evolved through the efforts of cities or nations that hosted the Olympic games: Los Angeles, 1984; Calgary, 1988; Lillehammer, 1994; Olympic Day and Olympic Week packages by the USOC and the German National Olympic Committee (Binder 2001). The Greek OEP seeks to establish a congruence between the goals inherent in the ‘philosophy’ of the Olympic Games and current international educational

430

Vasilios Grammatikopoulos et al.

priorities, while at the same time providing physical education teachers with useful teaching suggestions and aids. To be prepared properly for this new project, physical education teachers participated in a five-day training program. Many studies have underlined the necessity of the existence of evaluation procedures in planning and implementing a program (Worthen and Sanders 1987; Nevo 1994, 2001; Dimitropoulos 1998). Thus, evaluation of the OEP’s implementation was undertaken to provide information that will help to improve the program. The aim of the present study was to develop an instrument for the evaluation of the OEP’s implementation by means of internal evaluation procedures. Such an instrument could prove helpful for future program improvement. In addition, after appropriate adjustments the instrument could also be applied to other educational programs. What follows are two studies that describe methods and data pertinent to the development, content validity and reliability of the scale.

Study 1: Development and content validity Methods and procedures To develop the instrument used for the evaluation of the OEP’s implementation, the following approach was used. Based on a literature review of educational evaluation six factors affecting the implementation of an educational program were identified: (a) facilities, (b) educational material, (c) administration, (d) training, (e) relationships, and (f) educational procedure. A panel of experts consisting of eight specialists in physical education (each with a doctorate in education) was asked to develop at least four questions for each factor. The result was a pool of about 200 items. Three individuals with particular expertise in psychometric research reviewed the resultant list in order to limit the size of the item pool. This yielded 44 items that were included in the initial instrument. The reduction of the items was desirable because the aim of this study was to develop a measure that is easy to use and amenable to multiple application. These items were presented to five researchers (university professors) who had a broad range of experience in educational research. They were asked to rate the relevance and specification of each item, and to match each of them to one of the proposed factors. The criterion for retaining an item was the complete agreement of the five researchers as to its relevance. Ultimately, seven items were removed because there was disagreement about them among the researchers. A survey was conducted of 45 physical education teachers from a large district in Greece who had adequate educational experience (15–30 years, M = 21.41). They were asked to determine the extent to which they felt that each of the items was relevant to the evaluation of the program’s implementation. These data revealed that teachers’ assessments essentially paralleled the experts’ judgments; thus, all 37 items were retained. These items formed

Implementation of an Olympic Education Program

431

a scale called the Evaluation Scale of the Educational Program’s Implementation (ESEPI). Results and discussion Netenmeyer et al. (2003: 73) argued that ‘‘development of a content valid measure is enhanced during the early stages of scale development by effective creation of an ‘item pool’ and the subsequent evaluation of items by lay and/or expert judges’’. For the 37 items chosen for use in the ESEPI, there was complete agreement among the judges that all these items had a bearing on the evaluation of the OEP’s implementation. Similarly, 85% of the teachers agreed that these were relevant to the evaluation of the OEP’s implementation. For 21 items, this percentage increased to 94%. In light of these data, it was concluded that these items were representative of the evaluation of the OEP’s implementation, and that the instrument could be subjected to further psychometric examination.

Study 2: Factor structure Study 2 was conducted to develop hypotheses about the factor structure of the ESEPI. To this end, a large sample of physical education teachers working on the OEP used the ESEPI to evaluate the program’s implementation. The rating for the items on the ESEPI was based on a 5-point Likert scale – from 1 (=strongly disagree) to 5 (=strongly agree). Participants and procedures The sample was composed of 254 physical education teachers (about 13% of the population) working for the OEP. Among them were 147 males and 107 females. The mean age of participants was 35.4 ± 7.8 years. They were asked to complete the questionnaire evaluating the OEP’s implementation immediately after the conclusion of the OEP activities (May 2002). Physical education teachers participated voluntarily in the study. Results To determine the final number of factors, an initial Principal Component Analysis without rotation was conducted. Analysis showed that six factors had eigenvalues greater than 1.0. Scree-plot also supported the maintenance of six factors. Exploratory factor analysis followed by varimax was employed to examine the underlying structure of ESEPI. The factor loadings are presented in Table 1. Analysis revealed that 11 items had low loadings (