a study on hungarian 6th and 8th graders' proficiency ...

Nikolov, M. & Szabó, G. (2015). A study on Hungarian 6th and 8th graders’ proficiency in English and German at dual-language schools. In D. Holló & K. Károly (Eds.), Inspirations in foreign language teaching: Studies in applied linguistics, language pedagogy and language teaching (pp. 184-206). Harlow: Pearson Education

A STUDY ON HUNGARIAN 6TH AND 8TH GRADERS’ PROFICIENCY IN ENGLISH AND GERMAN AT DUAL-LANGUAGE SCHOOLS Marianne NIKOLOV and Gábor SZABÓ University of Pécs

Abstract: Dual-language education has a long tradition in Hungarian schools; the first schools opened in 1987. However, hardly any study has compared its impact on students’ development in the target languages and the content areas they study in a foreign language. This chapter gives an account of a large-scale assessment project initiated by decision makers in order to tap into the proficiency levels students achieve in years 6 and 8 in dual-language programmes. A total of 2,911 students were tested at 81 schools offering dual-language education in June 2014. The aim was to find out how many of the students learning English and German achieved the second and third levels of proficiency (A2 and B1) on the six-point scale defined in the Common European Framework of Reference for Languages (CEFR, Council of Europe, 2001). The first part of the chapter outlines the construct of dual-language education in the Hungarian context and overviews research conducted on it. The second part gives the background to the large-scale survey by drawing on language policy documents. The third part analyzes the results of the testing project and discusses the outcomes by comparing them to other studies on Hungarian learners’ achievements. The last section offers critical reflections on the findings, especially in light of what was not surveyed, and offers recommendations for further research and language policy. The main findings are impressive: the majority of the participants in both years and languages achieved the required proficiency levels. The chapter provides insights into reasons why the outcomes should be taken with a pinch of salt.

1 Introduction This chapter gives an account of a large-scale cross-sectional testing project implemented in June 2014 at 81 schools offering dual-language education in Hungary. This was the very first systematic enquiry into what levels of proficiency students achieved in dual-language programmes in years 6 and 8. First, we outline what dual-language education involves and how it has been implemented in the Hungarian context over the last three decades since its introduction in 1987. Then, we provide the background to the survey by discussing the rationale offered in language policy documents. In the empirical part of the chapter we analyze the findings of the examinations taken by 2,911 candidates in English and German by focusing on the following areas: (1) how students performed at the two levels in English and German; (2) more specifically, how they performed in listening comprehension, reading comprehension and writing; (3) how many candidates achieved the required A2 and B1 levels; (4) how students’ performances compare to one another in the two target languages in the three skills; (5) what the relationships are between the scores on the whole tests and the skills; and (6) what patterns emerged for individual schools in the

181

M01_HOLL3838_01_CUS_C01.indd 184

30/07/2015 11:04

NIKOLOV and SZABÓ: A study on Hungarian 6th and 8th graders’ proficiency in English and German185

large dataset. In the next sections we discuss the main findings by comparing them to outcomes in other projects and suggest ways in which triangulation could have been implemented. In the very last section we outline recommendations for further research and language policy.

2 Dual-language education in Hungary Dual-language programmes were launched at ten secondary grammar schools in Hungary in 1987. The target languages included English, French, German and Russian (Vámos, 2008). The new curricula were a modified version of bilingual education previously offered only to children of ethnic minority background. These bilingual opportunities were often favoured by highly motivated professional parents who considered proficiency in a second language an asset for their offspring; therefore, bilingual schools were seen as attractive options not only for ethnic minorities. The popularity of these has been unchallenged during the past decades. 2.1 Main features The term dual-language education (két tanítási nyelvű oktatás) has been used to distinguish this particular form of partial immersion from other forms of bilingual education. The main features distinguishing dual-language education from traditional foreign language teaching concern five domains. (1) The target language is used as a vehicle to carry curricular content by teaching three or more compulsory school subjects in the new language; (2) thereby, the intensity of language learning is much higher (over one third of total weekly classes in contrast with foreign language instruction in two to five classes a week). (3) Teachers’ roles vary: subjects are taught either by non-native speaker specialists (sometimes a subject and foreign language major in one) or a native speaker specialist teacher. (4) The language may also vary within a subject: part of the syllabus is taught in the target language, for example, world history, whereas students study Hungarian history in Hungarian. (5) Two curricular models exist for secondary schools: students start their new language from scratch and an additional school year is inserted for intensive foreign language study before they start learning school subjects in the target language. This model was applied in the year of intensive language learning project launched in 2004 (Medgyes & Öveges, 2004; Nikolov, Ottó & Öveges, 2009a). In the other model students start the dual-language programme in the foreign language they learnt at primary school, therefore no extra year is devoted to proficiency development. Since the early 1990s, when Russian ceased to be mandatory, schools launching dual-language programmes, most frequently in English and German, have mushroomed. In the first wave of expansion, comprehensive secondary schools joined the band wagon, whereas over the last decade more and more primary schools decided to teach school subjects in a foreign language, often starting as early as in the very first grade. In 2004 a HungarianChinese primary school was also opened. By 2014, the dynamic expansion has resulted in 81

182


30/07/2015 11:04

186

Inspirations in Foreign Language Teaching

dual-language primary schools, the foci of our study. This gradual increase has coincided with two other trends emerging not only in Hungary, but also around the globe: the lowering of age when children start learning a foreign language, which is overwhelmingly English, and a gradual shift towards content and language integrated learning (CLIL). 2.2 Research on dual-language schools Despite the fact that dual-language schools were definitely innovative in the late 1980’s, for years no research was published on how they worked except for an ethnographic study conducted by Duff (1995, 1996). Her enquiry provided new insights into socialisation and discourse practices at three English-Hungarian grammar schools. Over the years few studies examined classroom processes; an exception was a longitudinal study by Sebestyénné Kereszthidi (2010, 2011). She observed and interviewed three German teachers to examine how they implemented CLIL in their lower primary classrooms. Her study revealed that the teachers applied grammar-translation techniques and spoonfed the language instead of focusing on content and meaning. Vámos (2007, 2008) published extensively on dual-language schools. In her analysis (2007) she discussed the proficiency levels students graduating from dual-language schools achieved on the advanced-level school-leaving examinations (at the B2 level in the Common European Framework of Reference for Languages: Learning, Teaching, Assessment (CEFR), (Council of Europe, 2001). She concluded that 94.4% of the students achieved a top grade on a 1 to 5 scale in 2006, and over 78% of the candidates performed at the 100% level. In addition to language exams, students excelled at school-leaving examinations in other subjects in the target languages as well. The data, in her view, offer convincing evidence on the overwhelmingly outstanding results dual-language education offered students by the end of their secondary schools. Data for 2011 in a government document show a slightly less favourable picture: 80% of the candidates achieved the highest grade (A nemzeti idegennyelv-oktatás fejlesztésének stratégiája az általános iskolától a diplomáig: Fehér könyv 2012–2018, 2012, p. 18). So far, no survey has been available on what levels students achieve in primary schools. Most probably the excellent results of dual-language programmes served as one of the sources of motivation for schools launching new programmes. The other reason why an increasing number of institutions, especially primary schools, decided to experiment was competition for new students. Since schools are supported per capita, and there are fewer children at all levels of education, dual-language classes tend to attract more applicants than other programmes. In addition to these reasons, and most importantly, dual-language schools also benefited from special financial support to allow them to meet requirements specified in documents. These include the teaching of 35-50 percent of the overall number of classes in the target language and hiring at least one native-speaker staff member. Despite these complex requirements, the number of dual-language primary schools skyrocketed in a short time: it was 60 in 2002/2003, 99 in 2010/2011 and 121 in 2011/2012 (A nemzeti idegennyelvoktatás fejlesztésének stratégiája az általános iskolától a diplomáig: Fehér könyv 2012– 2018, 2012, p. 13). The numbers document extremely fast growth, especially in the last year. The reason why only 81 schools were involved in the project must be that school districts

183


30/07/2015 11:04


were reorganized in 2013; however, it is unclear how the centralization of educational institutions impacted dual-language programs. The dynamic increase in the number of new dual-language classes triggered the introduction of gate keeping quality control measures.

3 Background to the testing project Two language policy documents served as a basis for developing the framework and launching the project. The first one, issued in 1997, is a ministry decree (26/1997. (VII. 10.) MKM rendelet a két tanítási nyelvű iskolai oktatás irányelvének kiadásáról, 1997) comprising the main curricular principles of dual-language education. In a nutshell, duallanguage programmes aim to develop students’ proficiency in their mother tongue and in a foreign language to allow them to think and gain new knowledge in their new language, to make them open, tolerant and sensitive to other cultures. The document did not specify achievement targets, and some of these aims are clearly beyond what proficiency tests can measure. The second, more recent document (Az emberi erőforrások minisztere 4/2013. (I. 11.) EMMI rendelete a két tanítási nyelvű iskolai oktatás irányelvének kiadásáról, 2013, p. 609) further specified and elaborated on the above aims by adding the ability to use learning strategies, cultural awareness and empathy. The latter decree defines what CEFR levels students are expected to achieve in dual-language schools: A2 by the end of year 6, and B1 by the end of year 8. It also states that unless 60% of the students achieve the prescribed levels, the schools are not allowed to continue their dual-language programme. Due to this rule, the examination is considered high stakes. The construct to be measured is based on the main principles outlined in the CEFR and the curricula for foreign languages based on it (for a detailed analysis of how national curricular goals changed over time see Medgyes & Nikolov, 2010). The measuring instruments developed for the project aimed to assess students’ communicative competences in listening and reading comprehension and writing; decision makers excluded the assessment of speaking skills in years 6 and 8. The examination aimed to assess students’ language ability: the extent to which they are able to use the foreign language in contexts. All the tests used in the project focused on meaning in line with the main principles outlined in the testing literature (e.g., Alderson, Clapham & Wall, 1995; Bachman & Palmer, 2010; McKay, 2006). In addition to these principles the tests were age appropriate: the topics were chosen from the list in the curriculum and during the process of test development findings were integrated from previous research on testing Hungarian students of the same age but not at dual-language schools (Nikolov, 2011; Nikolov & Vígh, 2012; Nikolov & Szabó, 2011a, b, 2012a, b; Szabó & Nikolov, 2013). The tests aimed to tap into real life-like tasks students would have to do in order to comprehend the gist and specific points of what they heard and read, and the types of texts they would have to be able to write to express their ideas in line with the specifications in the curriculum. As speaking was not assessed, since the language policy documents required the testing of listening and reading comprehension and writing only, we aimed to find out how strong the relationships were among the three skills. Previous studies found weak or moderate correlations among these skills (Csapó & Nikolov, 2009; Józsa & Nikolov, 2005) suggesting that all skills develop along different trajectories; therefore, they should be assessed separately. The exclusion of speaking was particularly

184


30/07/2015 11:04

188


worrisome since oral skills are particularly emphasised in the curriculum and are the key to success in early language learning. Thus, the project does not allow us to gain any insights into how students would be able to use their foreign language for expressing their ideas orally in interaction with others.

4 The study 4.1 Research questions The main aim of the project was to assess the proficiency levels 6 th and 8th graders achieved in dual-language classes in English and German. More specifically the aim was to establish how many of the students achieved the A2 level in grade 6 and the B1 level in grade 8 in the two target languages. Therefore, we sought answers to the following research questions: How did the students perform on the English and German exams and on the tests measuring their listening and reading comprehension and writing skills? How many and what ratio of the students achieved levels A2 and B1? What relationships were revealed between the students’ overall performances on the exam and in the three skills? How did the students’ achievements compare across schools? What conclusions can be drawn and generalized from the results? What recommendations can be put forth for language policy? 4.2 Participants Table 1 shows data on the participants in years 6 and 8. All year 6 students took the exam at A2 level, whereas all year 8 participants took it at B1 level. As for the numbers of participants at the 81 schools, three types of data are included: the total numbers of students in years 6 and 8; the actual numbers of students who took the exam and their ratio in per cent. The numbers of participants vary to a large extent: in both years many more students learn English (1,502 in year 6; 886 in year 8) than German (418; 290, respectively). In year 6, over three and a half times more students study English than German, whereas in year 8 there are three times more learners of English, indicating more interest in English. Also, there are many more students in year 6 than in year 8; the numbers show the dynamic increase over two years in dual-language classes. The ratio of test-takers also varied: 93.4% of English learners took the examination in contrast with 96.2% of German learners. Ratios were somewhat lower for both language groups in year 8 data: 92.4% and 93.1%, respectively. The total number of participants was 2,911.

185


30/07/2015 11:04


Year 6 6 6 8 8 8

Level A2 A2 A2 B1 B1 B1

N students’ total actual ratio (%) students’ total actual ratio (%)

English German 1,502 418 1,420 402 93.4 96.2 886 290 819 270 92.4 93.1

Table 1. Participants in the survey learning English and German at A2 and B1 levels

4.3 Data collection instruments The tests The usedtests in theused project wereproject developed the project detailedalong test specifications. in the werefor developed foralong the project detailed test All focused onAllmeaning than form); cognitively specifications. focused (rather on meaning (rather they than were form); motivating they were and motivating and unchallenging for childrenfor aged 12-14;aged and12-14; they did or specific cognitively unchallenging children and not theyrequire did notcreativity require creativity or background knowledge. This alsoThis meant thatmeant the subject domains in English specific background knowledge. also that the subjectstudents domainsstudied students studied or were not covered in the tests, since subjects from school school; thus, in German English or German were not covered in thethe tests, since varied the subjects varied to from school to it was impossible find common topics covered by all students. tests were meant be school; thus, it wastoimpossible to find common topics covered by The all students. The tests to were applicable both developing testing young language learners’ proficiency in the hope meant to befor applicable for both and developing and testing young language learners’ proficiency that washback wouldeffect be positive on positive classroom Allpractice. the tasks most in thetheir hope that theireffect washback would be onpractice. classroom Allwere the tasks probably from textbooks. listeningThe and listening reading tests discrete items were mostfamiliar probably familiar fromThe textbooks. and comprised reading tests comprised and wereitems assessed Theobjectively. writing tests assessed three criteria both discrete and objectively. were assessed Thewere writing tests along were assessed alongat three levels: achievement, andvocabulary grammar. and grammar. criteriatask at both levels: taskvocabulary achievement, The exam booklets comprised the same number of tests and items at A2 and B2 levels in English and German. All skills were measured by two tests, each worth 10 points, thus students could achieve a maximum of 20 points in each skill and a total of 60 points. The full tests including the listening components can be accessed at the project website (http://www.oktatas.hu/kozneveles/meresek/celnyelvi_meres/feladatsorok/celnymeres_2014) (Célnyelvi mérés 2013/14).

4.4 Procedure The framework and test specifications were developed by a team of experts based on the language policy documents and special guidelines worded by policy makers. Unfortunately, as is so often the case with large-scale studies, the project was launched late, in the spring of 2014, whereas the candidates took the tests on June 4. The types of texts and tests were specified early on and the team of test developers designed pilot tests. The listening tests were all recorded with native speakers; both male and female voices were used. All listening tests were heard twice with short intervals between items. Due to the limited time scale, the tests were piloted with small groups of students with similar proficiency levels to those expected in the documents. However, no dual-language schools could be used for piloting the booklets, since all institutions had to participate in the live project. After piloting and editing the tests, standard setting was conducted with the

186


30/07/2015 11:04

190


participation of teachers and test developers to establish the pass rates at which students achieved the A2 and B1 levels. As a result of standard setting, the pass rate for the examinations was set at 60% both in English and German and at both proficiency levels. The time allocated for the examinations was 90 minutes at both levels; students filled in the printed test booklets in three 30-minute sessions with short breaks between them at all 81 schools at the same time. Listening tests were delivered from CDs. After administering the tests, language teachers at each school assessed their students’ two writings along criteria detailed in rating scales sent to the schools with the instructions. No training was available to teachers prior to rating the texts; this major drawback was beyond the control of the test developers. Then, the teachers entered their students’ data into data files for further analysis. As indicated above, the tests were developed centrally, but administered and assessed locally. 4.5 Students’ performances on the exam and in the three skills Table 2 summarizes the main findings on how participants performed in years 6 and 8 in English and German. Since the measuring instruments were based on the same test specifications, the results are considered to be directly comparable. The means in year 6 at level A2 are 78.714% in English and 67.802% in German; in year 8 at B1 level they are 82.534% and 76.933%, respectively. Three main trends emerge from the results: (1) At both levels the means are higher for English than for German, and (2) the standard deviations (SD) indicate less variation in English than in German results, especially among the younger testtakers. This is an important outcome especially in the light of the numbers of candidates: although many more students took the English test in year 6, the range of performances is less wide among learners of English than among their peers studying German. (3) Standard deviations indicate different trends in the two years and the two target languages: the SD is lower in year 6 for English and higher for German than for year 8. This means that there is more variation among the younger learners of German than among the older ones, whereas the opposite is true for English learners. Despite the fact that many more students’ performances are collapsed in the dataset of younger learners of English than their peers learning German, their proficiency levels indicate less variation within this group of candidates. Year 6 8

Level A2 B1

English Mean (SD) 78.714% (16.821) 82.534% (17.586)

German Mean (SD) 67.802% (20.728) 76.933% (18.572)

Table 2. Students’ performances on English and German proficiency test booklets

In Tables 3–6 the same datasets are analyzed by the three skills. The four tables comprise the students’ mean scores on the listening comprehension, reading comprehension and writing tests. The three skills, at both A2 and B1 level, were measured by two tasks each worth ten points, thus, the maximum scores were 20. The numbers (N) indicate how many candidates filled in the tests measuring the skills; all means and standard deviations are in

187


30/07/2015 11:04


raw scores. The reliability of the tests was calculated by Cronbach’s alpha values for the discrete item tests, but not for writing. The students’ two writings were assessed along a scale of 1–10; the criteria included task achievement, vocabulary and grammar. Table 3 shows the candidates’ mean scores in English in year 6 at A2 level. All means are high; the highest scores were found for writing (16.388), but listening results over 16 are also high, whereas reading scores are around 15. Standard deviations vary: they are the highest for reading (4.631), and lower (3.933; 3.282) for writing and listening. The mean per cent of correct answers is high in all three skills (between 75 and 82). The reliability coefficients are acceptable (.791) for listening and good for reading comprehension (.884). Skill

N

Mean

SD

Listening Reading Writing

1,417 1,413 1,411

16.043 15.068 16.388

3.282 4.631 3.966

Mean % correct 80 75 82

Cronbach’s alpha .791 .884

-

Table 3. Students’ achievements in English listening and reading comprehension and writing in year 6 at A2 level

Table 4 comprises the data for English learners in year 8 at B1 level. The mean values are high in all three skills, especially in listening comprehension (17.5); the standard deviation is low (2,846); and the mean per cent of correct answers is 88; the test was reliable (.822). The mean is lower (16.02) but still high in reading comprehension with a higher SD value (4.718); the Cronbach’s alpha (0,905) indicates high reliability. The mean on the writing tasks is a bit higher than on the reading tests (16.178). The mean per cent of correct answers in these two skills is 80 and 81, respectively. Skill

N

Mean

SD


819 819 819

17.501 16.02 16.178

2.846 4.718 4.555


Cronbach’s alpha .822 .905 -

Table 4. Students’ achievements in English listening and reading comprehension and writing in year 8 at B1 level

The results on the German tests in grade 6 (level A2) are shown in Table 5. Although the means are quite high, they are lower than those in Table 3. The highest mean was achieved (14.506) on writing tests; the second mean in the rank order was calculated for listening comprehension (14.133). Standard deviations are higher than in the case of the English A2 level (4.790; 4.033). The mean per cent of correct answers is 73 and 71). The Cronbach’s alpha value indicates that the listening tests were reliable (0.814). Means on the reading component were lower (12.218), whereas the SD was higher (5.668). The mean per cent of correct answers is only 61, whereas the Cronbach’s alpha value indicates high (0.904) reliability. The reading scores in Table 5 are the lowest in the dataset; this means that 6 th

188


30/07/2015 11:04

192


grade learners of German demonstrated the poorest performance in reading comprehension in the survey. Skill

N

Mean

SD


399 399 397

14.133 12.218 14.506

4.033 5.668 4.79



Table 5. Students’ achievements in German listening and reading comprehension and writing in year 6 at A2 level

Table 6 offers insights into year 8 results in German at B1 level. Scores on the listening and reading comprehension components are almost identical, a little over 16 in both skills; the standard deviations are higher for reading (4.179) than for listening (3.749). The mean per cent of correct answers is 80 in both cases and reliabilities (Cronbach’s alpha: .833; .870) are acceptable. The lowest scores (14.402) were achieved on the writing tasks with the highest SD (4.868), indicating more variance than in listening and writing; the 72% ratio of correct answers underpins the claim and makes the picture similar to the case of English B1 writing results, where the means were higher. Skill

N

Mean

SD


269 269 269

16.022 16.071 14.402

3.749 4.179 4.868



Table 6. Students’ achievements in German listening and reading comprehension and writing in year 8 at B1 level

The very same datasets were used to generate Figures 1–4 to illustrate the distributions of the scores. The number of candidates is shown on the vertical axis, whereas the scores (totals for the three skills 1 to 60) are on the horizontal axis. Since the numbers of candidates varied in the four tables, the heights of columns also vary in the figures: the scale is 100 in the case of English A2, 80 for English B1; both figures illustrating German test results use a scale of 25. Distributions are clearly different in Figures 1–2 and 3–4: in the first two figures distributions are negatively skewed, indicating mostly high achievements; the datasets do not resemble the typical bell curve indicating normal distribution. This means that the majority of English learners achieved high scores. The patterns are also negatively skewed in the German datasets in Figures 3–4, but the columns are more evenly distributed. The results of year 6 learners are closest to normal distributions.

189


30/07/2015 11:04


Figure 1. Students’ achievements in English listening and reading comprehension and writing in year 6 at A2 level

190


30/07/2015 11:04

194


Figure 2. Students’ achievements in English listening and reading comprehension and writing in year 8 at B1 level

191


30/07/2015 11:04


Figure 3. Students’ achievements in German listening and reading comprehension and writing in year 6 at A2 level

192


30/07/2015 11:04

196


Figure 4. Students’ achievements in German listening and reading comprehension and writing in year 8 at B1 level

After analyzing the overall results on the four exams as well as on the three skills, the next focal question concerns the ratio of the candidates who passed the examinations at A2 and B2 levels. 4.6 The ratio of candidates achieving A2 and B1 levels After piloting the four exam booklets and conducting standard setting, the pass rate for both years and languages was set at 60 percent. In this section we discuss how many of the candidates demonstrated proficiency at the A2 and B1 levels by achieving the pass rate or higher scores. Table 7 comprises data on the actual numbers of test-takers, the numbers of students who fulfilled the requirements, and their ratio in per cent. Nearly 90% of the candidates learning English achieved A2 level, whereas only 71% of their peers learning German did so. At B1 level, the results are 91.2% and 85.6%, respectively. The ratios are higher at B1 level (year 8) than at A2 level (year 6); however, the differences are minimal in English. In the case of German, year 6 students’ difference is close to 15% lower than that of

193


30/07/2015 11:04


year 8 candidates. The difference between year 6 and year 8 results in German is significant (p≤0.001). Comparing the results for years 6 and 8 in the two languages, the ratio of successful candidates learning English is over 91% in both years, whereas it is lower than 86% in both cohorts of German learners. These differences are also significant (p≤0.001); thus, students of English performed significantly better than their German-learning peers in year 6 as well as in year 8. Differences between English and German scores turned out to be significant (p≤0.001) in the three skills except at the B1 level reading comprehension where the difference was statistically not significant. Year 6 6 6 8 8 8

Level A2 A2 A2 B1 B1 B1

N actual passed % actual passed %

English 1,420 1,268 89.3 819 747 91.2

German 402 285 70.9 270 231 85.6

Table 7. Number of candidates, number and ratio of successful candidates in grades 6 and 8 learning English and German

4.7 Relationships between skills and overall results After analyzing the achievements and comparing the numbers and ratios of students achieving A2 and B1 levels, in this section we focus on the relationships between scores concerning the whole examination and the three skills. Tables 8–11 show the correlation coefficients for years 6 and 8 (levels A2 and B1) for English and German. All values of Spearman’s rho were found significant (p≤0.01). Quite strong relationships are indicated between the tests measuring listening, reading and writing skills and the whole tests; Spearman’s rho values in the Exam columns range between .729 and .904 in the four tables. The closer the number is to 1 the stronger the relationship between the test measuring a skill and the examination. Strong relationships indicate that the candidates achieving high scores in the particular skill, also performed well on the whole examination and vice versa: weaker students tended to get low total scores and skill scores. In other words, high correlations between skills and the total scores mean that high achievers’ language abilities tend to be high in the three skills, and the patterns of different developmental levels are less typical.

Skill Skill ListeningListening Reading Reading Writing Writing

Exam .804 .870 .729

Exam .804 .870 .729

Listening Listening 1 1 .623 .623 .415 .415

Reading Reading .623 .623 1 1 .450 .450

Table (Spearman’s rho) rho) between between English English listening, listening, reading reading and and writing writing Table 8. 8. Correlation Correlation coefficients coefficients (Spearman’s tests A2 level level tests and and whole whole exam exam in in grade grade 66 at at A2

194


30/07/2015 11:04

198



Exam .753 .859 .802

Exam .753 .859 .802


Reading Reading .582 .582 1 1 .489 .489

Table 9. Correlation coefficients (Spearman’s rho) rho) between between English English listening, listening, reading reading and and writing writing tests and whole exam in grade 8 at B1 level


Exam .870 .904 .782

Exam .870 .904 .782


Reading Reading .734 .734 1 1 .551 .551

Table 10. Correlation coefficients (Spearman’s rho) between German listening, reading and writing tests and whole exam in grade 6 at A2 level


Exam .850 .812 .814

Exam .850 .812 .814


Reading Reading .677 .677 1 1 .439 .439

Table 11. Correlation coefficients (Spearman’s rho) between German Table German listening, listening, reading reading and and writing writing tests and whole exam in grade 8 at B1 level

Correlation coefficients are below .8, indicating somewhat weaker relationships in the case of the English exam and the writing test at A2 level in Table 8, and between listening comprehension and the exam at B1 level in Table 9, and in the case of the German exam at A2 level and writing in Table 10. All three correlations are higher than .812 in Table 11 for German at B1 level. The strongest relationship is .904 in the four tables for A2 reading scores and the whole exam (Table 10). As was shown in Table 5, scores on the German reading comprehension test were the lowest in the project. The strong correlation means that the students who achieved high or low on the reading component also achieved very similar total scores. Correlations are much lower between the skills in Tables 8–11 indicating weaker relationships, in line with our expectations. Patterns, however, vary to a large extent if we analyze relationships between receptive and productive skills. Receptive skills (listening and reading) are somewhat more closely related to one another at the A2 level than at the more advanced B1 level. Also, the German data are higher at both levels than in the case of English, showing stronger relationships. A comparison between the two languages at the A2 and B1 levels reveals different patterns: correlations are higher between both receptive skills and writing at B1 level than at A2 level in the English dataset, whereas the German data show stronger relationships at A2 level. As for the comparison between receptive and productive skills, relationships tend to be stronger for German than for English at both proficiency levels. These correlations are the

195


30/07/2015 11:04


lowest in Tables 8–11. The lower correlations mean weaker relationships between listening and reading and writing scores. This trend has important implications for the overall findings of the assessment project. The results on listening and reading tests hardly overlap with results on writing; therefore, an even weaker relationship can be hypothesized between these scores and candidates’ speaking abilities. The examination did not include any measure of speaking skills, thus, one of the basic skills was excluded from the project. It is especially worrying in light of the fact that in the early years of language education the development of oral skills is a must, as is emphasized in the Hungarian core curriculum. If the students’ speaking skills are not assessed, this may exert a negative washback effect on classroom practices. 4.8 Results at the level of schools At the end of the project all schools received information on the following results: what means their students achieved on the exam and on the tests measuring the three skills, and the same data for the relevant national cohort in the project. These pieces of information were meant to allow schools to compare their students’ performances to those of other institutions teaching the same age groups and languages. Also, the structured dataset is useful for decision makers in order to analyze how schools performed in comparison with one another. This chapter does not allow us to scrutinize the datasets of all 81 institutions in detail, but some of the main trends are worth a short discussion. The general findings masked a lot of variation in the results at the level of schools. At most schools both 6th and 8th graders participated in English or German, but there were a few schools where the programme must have been launched recently, thus, only year 6 candidates were surveyed. A different extreme is found in a few institutions where both English and German are offered in both years. It must be more challenging to teach and learn in a context where dual-language education is an innovation in contrast to a school where it is a well-established tradition with experienced teachers. Shocking differences were found in the numbers of students at various institutions: at 13 schools the number of candidates was below ten; the smallest group comprised three learners. At the other end of the scale, numbers ranged between 50 and 89. The data indicate very different teaching and learning conditions at schools; it must be a very different challenge to teach three or 19 students in a group. Another trend was revealed in the number of absences: typically either no or one student was absent at most schools, but, for example, five out of 23 and twelve out of 80 did not take the test at two schools. Patterns in means also varied a lot. (1) At a school means for the three skills were very similar (e.g., at A2 level in English 69%, 69%, 65%; mean on exam 67%), below the national mean, especially in writing. In this case all three skills have to be developed. (2) A different patterns emerged where one skill lagged behind the others (e.g., English writing at A2: 56%, B1: 54%, whereas the means for the other two skills are better than the mean for the whole cohort); further analysis is necessary on how writing scores were distributed on the three criteria. (3) Means were very different in years 6 and 8 at a school teaching German: they were low, especially for reading at A2 level (59%, 41%, 64%), whereas at B1 level

196


30/07/2015 11:04

200


means were way beyond the national means (92%, 97%, 81%). This may mean that the exam was a huge challenge for 6th graders, especially in reading, but this school is probably doing something outstanding, assuming that the 8th graders were similarly low achievers two years earlier. These examples indicate that the findings cannot be interpreted without considering contextual factors and learners’ individual differences. (4) In some cases means are extremely high, for example, 97% or 100% on writing (national data 81%). It would be interesting to find out what the means would have been had the texts been assessed externally. The schools’ results should be compared and contrasted along other contextual criteria. As none were available, this kind of analysis has to be conducted by decision makers and school administrators in order to gain deeper insights into what the data mean and how the findings can be used to improve dual-language education in particular contexts.

5 Discussion In this section, we first discuss the most important findings by comparing them to outcomes in other projects. Then, we point out numerous limitations of the study and suggest ways in which triangulation could have been implemented. The most impressive result is that the overwhelming majority of the candidates achieved the required proficiency levels of A2 and B1 in both target languages. In sum: in grade 6 almost 90% of the candidates taking the English exam, and 71% of the test takers taking the German tests achieved level A2. The results are even better in grade 8: nearly 91% in English and 86% in German were at B1 level. The ratios are higher in year 8 in both target languages; however, the differences are minimal in English, whereas the difference in the case of German is close to 15%, indicating a significant difference (p≤0.001) at B1. In previous studies inquiring into the same age groups’ proficiency, students’ performances were measured at the A1-A2-lower half of B1 levels (Csapó, 2001; Csapó & Nikolov, 2009; Józsa & Nikolov, 2005; Nikolov, Ottó & Öveges, 2009a), but these results are not directly comparable, since in those studies no pass rates were set. However, it is clear that candidates in the present cohorts achieved a higher level according to the levels described in the previous surveys. These overwhelmingly outstanding results can be interpreted as a strong argument for dual-language education. Before looking into this issue, we discuss some additional points. Important differences were found between the two target languages in the present study. The significantly higher achievements in English than in German are in line with findings of previous large-scale surveys conducted in years 6, 8, 10 and 12 learning English and German in typical foreign language learning contexts (not dual-language schools). Candidates learning English outperformed their peers studying German in two national surveys involving representative samples of students in state schools (Csapó, 2001; Csapó & Nikolov, 2009; Józsa & Nikolov, 2005). A different testing project (Nikolov, Öveges & Ottó, 2009a) revealed consistently better achievements over a 5-year study analyzing how students’ proficiency developed during the year of intensive language learning and the subsequent four years. In all large-scale assessments, students of English achieved higher levels of proficiency. The reasons underlying this trend are not in the target languages per se, but in important differences in learners’ attitudes towards learning English and German, their

197


30/07/2015 11:04


aptitudes, socioeconomic status, and contextual variables related to the location of the schools. Learners’ attitudes are overwhelmingly more positive towards learning English than German (Dörnyei, Csizér & Németh, 2006), they are more motivated to study, set higher goals for themselves, are more able, and the higher their parents’ educational level the more probable it is that they study English. German is somewhat more popular in schools near the Austrian border where the utilitarian motives are typical and achievements also tend to be greater in contrast with other locations. Yet another trend in the findings is in line with previous research: although standard deviations in the datasets are not very high, large differences at the level of schools add more clarity to the larger picture. For example at a particular school, at A2 level the mean was 87%, and at B1 level 96% in English, way beyond the national means (79% and 83%); at two low achieving schools at A2 level in German the means were 31% and 40%, respectively (full cohort 68%). These schools need help to scaffold their students’ development more efficiently and the first step forward should be to diagnose the reasons why candidates achieved poor results. These points highlight some of the limitations of the present project. First of all, the majority of candidates demonstrated outstanding proficiency on the exams. The results indicate that students developed at the expected rate and managed to achieve the goals set by decision makers. It is important to use these data as a baseline for future assessments, since this is the first dataset available for further comparisons. As has been shown, proficiency in English seems to develop at a faster rate than in German. This has to be born in mind when students are tested after two years in years 8 and 10, especially bearing in mind that our findings were based on a cross-sectional comparison. Year 6 data forecast more variation in students’ proficiency in two years’ time. As for the validity and reliability of the results, a word of warning is necessary and this takes us back to our first point concerning the high achievements. This was a high stakes examination and all stakeholders (candidates, teachers, school administration, decision makers) were fully aware of this fact. Although the tests were externally validated, they were administered by the schools and the students’ teachers themselves assessed the writing tests. It is a realistic assumption that both groups of stakeholders had a vested interest in maintaining their programmes and jobs. Therefore, it is the decision makers’ task to consider how changes in the implementation of the testing project may have impacted the results. The relatively high number of students absent at a few schools on the day of test administration may also indicate that the conditions were not the same in all contexts. Conclusions, however, are to be drawn by decision makers after taking a close look at the findings and considering the recommendations. A similar controversy has characterized the two levels of school-leaving examinations since their introduction over a decade ago. The advanced-level exam is external, whereas the intermediate-level exam is internal, not only in foreign languages, but also in all school subjects.

198


30/07/2015 11:04

202


6 The way forward: recommendations and conclusions In thiswesection outline recommendations further and language In this section outlinewe recommendations for further for research andresearch language policy. The policy.results The main results the underpin that dual-language offer students main underpin claim the thatclaim dual-language curriculacurricula offer students better better opportunities traditional foreign language programmes.However, However,this this conclusion opportunities than than traditional foreign language programmes. cannot be drawn unless reliable data are also available on other types of curricula for all forfor example thatthat the gains were due moretointensive languagelanguage learning schools. ItItisispossible, possible, example the gains wereto due more intensive and not content in the target language. Or the aptitude motivation of students learning and notlearning content learning in the target language. Or the and aptitude and motivation of at dual-language schools are different those of other furtherFor inquiries, students at dual-language schools are from different from those learners. of other For learners. further large-scalelarge-scale surveys are neededare onneeded a regular to monitor over time inquiries, surveys onbasis a regular basis tostudents’ monitor development students’ development in years and 8 at6the of A1 to B1. Such longitudinal studies would over time6 in years andscale 8 at the scale of A1 to B1. Such longitudinal studiesprovide would insights provide into all candidates’ strengths and weakness. would offer awould baseline to examine what insights into all candidates’ strengths and Datasets weakness. Datasets offer a baseline to added value characterizes schools, and howschools, dual-language anddual-language traditional curricula compare examine what added value characterizes and how and traditional to one another. curricula compare to one another. Decision makers should also consider if the strict distinction between curricula is justified or useful. All schools should be able to offer one or two subjects in years when staff with appropriate qualification are available. Schools should be encouraged and helped to experiment to find out what works best in their contexts. They should enjoy autonomy in choosing subjects to deliver in a new language and they would need support to be able to monitor their own projects. The scope of the present survey was limited to language proficiency; no information is available on factors impacting language development over time. Without examining background variables and students’ individual differences, it is impossible to gain insights into the depth and complexity of language learning. Previous studies found enormous differences in language learning opportunities in different school and also between classes in schools (e.g., Csapó, 2001; Józsa & Nikolov, 2005; Nikolov, Ottó & Öveges, 2009a, b). International and national surveys include the following areas and future studies should also integrate some or all of them. (1) Students’ individual differences impact their success in language learning to a large extent. Their socioeconomic status (measured by parents’ educational or income level), aptitude, motivation, anxiety, goals, self-assessment, strategies have been found to be most important. Teachers can influence some of these. (2) Contextual factors also play an important role: the age when students have access to ageappropriate language learning opportunities (meaning-focused intrinsically motivating and cognitively challenging tasks), frequency of classes, curriculum, etc. (3) Teachers’ competences interact in the language teaching process: their methodology, motivation, proficiency in both languages, beliefs, etc. (4) The least known area is how extracurricular activities, including private tuition and free voluntary reading, internet browsing, watching of films, chatting, role plays, music, etc. in the target language contribute to motivation and proficiency. (5) Hungarian surveys tend to include school grades to allow comparison between formative and proficiency assessment in first and foreign languages and other subjects. (6) An obvious factor to take into account concerns the subjects candidates learnt in the target language for years. Most probably important insights were missed by not including components covering the actual construct of subject domain knowledge in the target

199


30/07/2015 11:04


language, the main distinguishing feature of dual-language education. Unfortunately, it was not known which school subjects’ candidates had studied for how long and in how many hours before they took the exam. It would have been a must to explore to what extent various subjects contributed to proficiency development. (7) Finally, teachers’, students’ and parents’ voices should have been heard about the benefits and challenges of dual-language learning. Without their views it is impossible to interpret the test results. A further recommendation concerns speaking. As was pointed out, no insights were gained into candidates’ speaking skills. This is unacceptable, not only because the survey failed to reveal anything about their proverbial ability to ask for a glass or water, but also because the ability to interact orally is a key competence, especially at the early stages of language development. Excluding speaking may result in negative washback; only what is tested matters, therefore, speaking does not. The testing of speaking is not a simple task. Teachers need special training to be able to implement oral tasks appropriately and to assess their students’ speaking abilities along criteria. Such training would result in more efficient teaching of oral skills. Examiner training is necessary for all skills, especially for speaking and writing. In the future, all teachers should be trained regularly on how to assess their students’ progress and proficiency and how to apply the data for diagnostic purposes. It is an important step forward that the tests are freely available to all stakeholders. The next steps should allow everyone to have access to and search students’ writings and oral performances. Building a corpus could be used for teacher education as well as for classroom teaching. In addition to learner corpora, a bank of tests should be available with calibrated tests for practice purposes. Such an online bank would offer students and teachers many more learning opportunities to boost all four skills. It would be useful to compare test results across cohorts assuming that the survey will be repeated annually or biannually. Since the tests are publicly available, anchor items cannot be applied. A particular type of Rasch analysis (Many-Facet Rasch Measurement), however, would make it possible to link tests in different ways, allowing all the items to be placed on a common scale and thus making them comparable over test versions. Finally, paper and pencil tests were used in the survey. In the future, computer-based tests would be more appropriate and more efficient. It would offer not only better opportunities for practice and security, but also a smart way to build learner corpora and test banks.

References 26/1997. (VII. 10.) MKM rendelet a két tanítási nyelvű iskolai oktatás irányelvének kiadásáról [Ministry of Education decree on issuing the basic principles of duallanguage schools] (1997). Retrieved December 12, 2013 from http://net.jogtar.hu/jr/gen/hjegy_doc.cgi?docid=99700026.MKM

200


30/07/2015 11:04

204


Alderson, C., Clapham, C., & Wall, D. (1995). Language test construction and evaluation. Cambridge: Cambridge University Press. A nemzeti idegennyelv-oktatás fejlesztésének stratégiája az általános iskolától a diplomáig: Fehér könyv 2012–2018 [A strategy for the development of national foreign language education from the primary school to the degree: White book 2012–2018]. Retrieved May 5, 2014 from http://nyelvtudasert.hu/cms/data/uploads/idegennyelv-oktatas-feher-konyv.pdf Az emberi erőforrások minisztere 4/2013. (I. 11.) EMMI rendelete a két tanítási nyelvű iskolai oktatás irányelvének kiadásáról [4/2013. (I. 11.) decree of the minister of human resources on issuig the basic principles of dual-language schools] (2013). Magyar közlöny, 2013, 6, 609–611. Bachman, L., & Palmer, A. (2010). Language assessment in practice. Oxford: Oxford University Press. Célnyelvi mérés 2013/2014: feladatsorok és javítókulcsok [2013/2014 assessment of target languages: test booklets and keys]. Project website. Retrieved May 12, 2014, from http://www.oktatas.hu/kozneveles/meresek/celnyelvi_meres/feladatsorok/celnyme res_2014 Csapó, B. (2001). A nyelvtanulást és nyelvtudást befolyásoló tényezők [Factors impacting language learning and proficiency]. Iskolakultúra, XI(8), 25–35. Csapó, B., & Nikolov, M. (2009). The cognitive contribution to the development of proficiency in a foreign language. Learning and Individual Differences, 19, 203– 218. doi:10.1016/j.lindif.2009.01.002. Council of Europe (2001). Common European framework of reference for languages: Learning, teaching, assessment. Cambridge: Cambridge University Press. Dörnyei, Z., Csizér, K., & Németh N. (2006). Motivation, language attitudes and globalization: A Hungarian perspective. Clevedon: Multilingual Matters. Duff, P. (1995). An ethnography of communication in immersion classrooms in Hungary. TESOL Quarterly, 29(3), 505–536. Duff, P. (1996). Different languages, different practices: Socialization of discourse competence in dual-language school classrooms in Hungary. In K. M. Bailey, & D. Nunan (Eds.), Voices from the language classroom: Qualitative research in second language education (pp. 407–433). Cambridge: Cambridge University Press. Józsa, K., & Nikolov, M. (2005). Az angol és német nyelvi készségek fejlettségét befolyásoló tényezők [Factors impacting the developmental levels of English and German language skills]. Magyar Pedagógia, 105(3), 307–337. McKay, P. (2006). Assessing young language learners. Cambridge: Cambridge University Press. Medgyes, P., & Öveges, E. (2004). Paved with good intentions: Foreign language policy in Hungary (pp. 279–290). In Cs. Dobos, Á. Kis, Y. Lengyel, G. Székely, & Sz. Tóth (Eds.), Mindent fordítunk és mindenki fordít: Értékek teremtése és közvetítése a nyelvészetben. (pp. 279–290). Bicske: SZAK. Medgyes, P., & Nikolov, M. (2010). Curriculum development: The interface between political and professional decisions. In R. Kaplan (Ed.), The Oxford handbook of applied linguistics (2nd ed., pp. 264–274). Oxford: Oxford University Press.

201


30/07/2015 11:04


Nikolov, M. (2011). Az angol nyelvtudás fejlesztésének és értékelésének keretei az általános iskola első hat évfolyamán. Modern Nyelvoktatás, XVII(1), 9–31. Nikolov, M., Ottó, I., & Öveges, E. (2009a). A nyelvi előkészítő évfolyam értékelése 2004/2005–2008/2009. Budapest: Oktatási és Kulturális Minisztérium. Nikolov, M., Ottó, I., & Öveges, E. (2009b). Szakma és nyelv? Az idegennyelv-tanulás és tanítás helyzete és fejlesztésének lehetőségei a szakképző intézményekben [Profession and language? The teaching and learning of foreign languages and opportunities for their development at vocational schools]. Budapest: Oktatásért Közalapítvány. Nikolov, M., & Szabó, G. (2011a). Az angol nyelvtudás diagnosztikus mérésének és fejlesztésének lehetőségei az általános iskola 1–6. évfolyamán [Opportunities for the development of diagnostic assessments of English as a foreign language in years 1 to 6 in primary schools]. In B. Csapó, & A. Zsolnai (Eds.), A kognitív és affektív fejlődés diagnosztikus mérése az iskola kezdő szakaszában (pp. 13–40). Budapest: Nemzeti Tankönyvkiadó. Nikolov, M., & Szabó, G. (2011b). Establishing difficulty levels of diagnostic listening comprehension tests for young learners of English. In J. Horváth (Ed.), UPRT 2011: Empirical studies in English applied linguistics (pp. 73–82). Pécs: Lingua Franca Csoport. Nikolov, M., & Szabó, G. (2012a). Developing diagnostic tests for young learners of EFL in grades 1 to 6. In E. D. Galaczi, & C. J. Weir (Eds.), Voices in language assessment: Exploring the impact of language frameworks on learning, teaching and assessment–Policies, procedures and challenges, Proceedings of the ALTE Krakow Conference, July 2011 (pp. 347–363). Cambridge: UCLES/Cambridge University Press. Nikolov, M., & Szabó, G. (2012b). Assessing young learners’ writing skills: A pilot study of developing diagnostic tests in EFL. In G. Pusztai, Z. Tóth, & I. Csépes (Eds.), Current research in the field of disciplinary didactics. Hungarian Educational Research Journal, Special Issue, 2, 50–62. Nikolov, M., & Vigh, T. T. (2012). (2012). Az Az idegen nyelvek tanulásának eredményessége [The success of learning foreign foreign languages]. languages]. In: In Csapó Benő (Ed.): Mérlegen a magyar iskola (pp. 241–288). Budapest: Nemzeti Tankönyvkiadó. Sebestyénné Kereszthidi, Á. (2010). Alsótagozatos németórákon szerzett tapasztalataimról [Experiences in lower-primary German classes]. Modern Nyelvoktatás, 16(2–3), 56–70. Sebestyénné Kereszthidi, Á. (2011). Kéttanítási nyelvű képzésben, alsótagozaton oktató nyelvtanárok meggyőződéseinek elemzése [Teachers’ beliefs in German duallanguage lower-primary classes]. Magyar Pedagógia, 111(1), 79–103. Szabó, G., & Nikolov, M. (2013). An analysis of young learners’ feedback on diagnostic listening comprehension tests. In J. Mihaljevic Djigunovic, & M. Medved Krajnovic, (Eds.), UZRT 2012: Empirical studies in English applied linguistics (pp. 7–21). Zagreb: FF Press. Vámos, Á. (2007). Kétszintű érettségi vizsga a két tanítási nyelvű középiskolákban [The twolevel school-leaving examination at dual-language secondary schools]. Új Pedagógiai Szemle, 57(3–4), 104–113.

202


30/07/2015 11:04

206


Vámos, Á. (2008). A kétnyelvű oktatás tannyelv-politikai problématörténete és jelenkora [The past and present problems in dual-language education]. Budapest: Nemzeti Tankönyvkiadó.

203


30/07/2015 11:04