Assessing the use of sophisticated EFL writing

John Benjamins Publishing Company

This is a contribution from Dutch Journal of Applied Linguistics 2:1 © 2013. John Benjamins Publishing Company This electronic file may not be altered in any way. The author(s) of this article is/are permitted to use this PDF file to generate printed copies to be used by way of offprints, for their personal use only. Permission is granted by the publishers to post this file on a closed server which is accessible to members (students and staff) only of the author’s/s’ institute, it is not permitted to post this PDF on the open internet. For any other use of this material prior written permission should be obtained from the publishers or through the Copyright Clearance Center (for USA: www.copyright.com). Please contact [email protected] or consult our website: www.benjamins.com Tables of Contents, abstracts and guidelines are available at www.benjamins.com

Assessing the use of sophisticated EFL writing A longitudinal study Pieter de Haan and Monique van der Haagen† Radboud University Nijmegen

Even very advanced EFL writing tends to be less sophisticated than native writing. One of the problems seems to be finding the right collocations and the correct register. The aim of this article is to pinpoint what characterizes the development in very advanced Dutch EFL students’ written language production. We discuss the development of students’ ability to use appropriate intensifiers. Compared to their native English speaking contemporaries, the Dutch students initially tend to use intensifiers that are found typically in spoken English, such as really and a bit, but they gradually replace them by modifiers more suitable to academic writing. It is argued that the use of appropriate intensifiers can be seen to be a measure of advancedness and hence be used as a criterion in the assessment of advanced EFL writing quality. Keywords: CEFR, EFL writing, sophistication, L2 development, longitudinal, corpora

1. Introduction: Writing, native English writing, non-native writing Numerous studies have been conducted into non-native use of vocabulary and syntactic patterns (e.g. Calabrese, 2008; Granger, 1998; Nesselhauf, 2003). In all of these studies the general conclusion would appear to be that L2/FL writing is on the whole less sophisticated than L1 writing, in that it is less fluent (i.e. shorter), less accurate (it contains more morphology and syntax errors), less effective (it receives lower holistic scores), and distinctly simpler in structure than L1 writing. What these studies have so far not been able to reveal is how FL/L2 writing develops, whether it is possible for FL/L2 writers to acquire a level of sophistication that can be called native-like, and how the latter can be assessed. De Haan and Van Esch (2005) did establish that progress at this advanced level cannot be measured by crude features like word length or type/token ratio, but that it may manifest itself in more subtle aspects of vocabulary use. Dutch Journal of Applied Linguistics 2:1 (2013), 16–27. doi 10.1075/dujal.2.1.04deh issn 2211–7245 / e-issn 2211–7253 © John Benjamins Publishing Company

Assessing the use of sophisticated EFL writing

Grant and Ginther (2000) looked at different levels of EFL writing proficiency and their characteristics, but they did not look at the development of individual learners’ proficiency. By contrast, De Haan and Van Esch (2005; 2008) studied developmental aspects of EFL writing and found differences in linguistic features such as essay length and sentence length, between first year essays and second year essays written by students of a single cohort in the English department of Radboud University Nijmegen in the Netherlands. These differences could all be said to be directly related to an assumed development of writing skills. The second year essays also received higher holistic scores. One of the observations that De Haan and Van Esch made is that what apparently applies to lower levels of foreign language proficiency does not appear to apply to very advanced Dutch EFL levels. For instance, they found that none of the Dutch EFL writers in their study made many serious grammar or vocabulary errors, such as tense or morphology errors, whether they are in the first, second, or third year. So development at this level, if any, cannot be measured by features that apply at lower levels of proficiency. However, there must be something that develops, because blind holistic assessment of these data by experienced writing class lecturers, native and non-native alike, shows that the third year essays tend to be rated higher than the first year’s. Moreover, lecturers frequently comment on the lack of maturity in the first year essays (for a discussion of holistic assessment, see e.g. Casanave, 2004; De Haan & Van Esch, 2008). In order to characterize development at this very advanced level, we need to consider (the interaction of) different measures, such as those that show how many really native characteristics (in terms of idiomatic and syntactic preferences (De Haan, 1997; 1998; Lorenz, 1999) the learners have mastered at various levels of their L2 proficiency. In doing so, we might eventually also help to formulate the existing CEFR definitions of C1 and C2 levels in more concrete terms (Council of Europe, 2009). Exactly what sophisticated writing is, is not easy to define. A common method is to define sophistication on the basis of learners’ institutional status (Ortega & Byrnes, 2008, p. 7). When Dutch students enter university, they have already had six years of formal instruction in English at secondary school, given that English is a compulsory foreign language for secondary school pupils in the Netherlands. They have also been taught English for two years at primary school. So, prior to their academic study of English, they have already been exposed to the English language for eight years. In combination with their keen interest and strong motivation to master English, we may assume that they have already accrued a rather high level of proficiency at the time when they produce their first essay. They are already advanced learners, whose writing proficiency level corresponded to the B2+ (upper intermediate) level of the Written Assessment Criteria Grid of the © 2013. John Benjamins Publishing Company All rights reserved

17

18

Pieter de Haan and Monique van der Haagen†

CEFR (Melissen, 2007). Considering that the students take courses intended to improve their written communication skills, it is safe to assume that their competence in English will develop even more during their Bachelor’s study. In our teaching, we therefore aim at a written proficiency at C2 [upper advanced] level when they graduate. The CEFR descriptions are rather vague; therefore it is worth investigating what happens between CEFR levels B2 and C2. We encourage our graduates to keep improving their skills in their MA study, beyond the CEFR C2 (Mastery) level, because there is a need for non-native speakers with a near-native command of English (EFL professionals, see below), who can help other non-native speakers (EFL users) in their EFL communication. 2. The EFL professional The aim of this article is to pinpoint what characterizes the development in the written language production of these very advanced EFL learners. Dutch university students of English are fulltime majors. They are not merely students who are improving their English skills to perform at an academic level in any academic discipline, but they are training to become what we would like to call EFL professionals. We make a distinction between EFL users and EFL professionals. EFL users are non-native speakers of English who use English to communicate with native (or other non-native) speakers of English. EFL professionals, on the other hand, are non-native speakers of English who are employed as language teachers, language trainers, translators, or editors, usually in a non-native English environment. Dutch EFL professionals, in our view, should not merely have a very advanced proficiency of English (CEFR C2), but a native-like command, as they will often be consulted about EFL matters that go beyond the Mastery level. While it is true that not all English graduates in the Netherlands choose a career as language professionals, the majority do. The EFL writers that were studied for the current article are young, post-adolescent university students of English, in their late teens or early twenties, who have already attained a high command of English. It can be assumed that they are not only becoming very proficient EFL writers, but are still maturing as human beings and as academics, gradually attaining an academic level of thinking. They will vary in the way they develop individually, but the academic development is fed by their classes, their reading, and the interaction with lecturers and fellow students. A gradual maturing in academic terms will lead to a more balanced presentation of ideas and viewpoints, by means of increasingly more subtle distinctions in formulation. At the same time, they are acquiring the sensitivity and the vocabulary to express these distinctions. © 2013. John Benjamins Publishing Company All rights reserved


In an earlier article (De Haan & Van der Haagen, 2012) we studied the use of intensified adjectives in very advanced Dutch students’ essays, and the way in which this use changed over time, comparing the students’ use with that of native British sixth-formers and university students. Intensification of adjectives had been found to be a good indicator of the level of sophistication of EFL writing (Lorenz, 1999). On the basis of even limited data we were able to show that there is a development in this use in the EFL writers which is comparable, but not equal to that of native English writers. Most of these findings were consistent with earlier findings (e.g. De Haan, 1999) in which it had been found that Dutch EFL learners use far more adverbs than native student writers. More specifically, adverb-adjective and adverb-adverb combinations are greatly over-used by Dutch EFL writers. Lorenz had made similar observations for German EFL learners, and he had also found that the German EFL writers use far more adjectives in the first place. We concluded that all writers obviously mature as writers, but that EFL writers also need to mature as users of the foreign language. For EFL writers this means that development takes place along a dual track, combining the development of the language user and the (academic) writer. Native writers, by contrast, only need to develop as (academic) writers. In the current article we want to study the use of intensified adjectives and adverbs in EFL writing, in order to find out whether these can be used as a means of assessing the level of sophistication. In view of the above considerations, the specific research questions that we wish to address in this article are: 1. Can we observe any differences between native English student writers and Dutch EFL student writers in the intensification of adjectives and adverbs, and do these differences relate to an assumed lower level of sophistication in the Dutch EFL writers? 2. Do these differences decrease over time as the Dutch EFL student writers become more advanced? 3. Can we relate these differences to the nature of the writing tasks? 3. Method We studied all the occurrences of intensified adjectives and adverbs in native and non-native student essays, on the assumption that the way in which adjectives and adverbs are intensified is indicative of sophistication of English writing. The nonnative data used for this study are student essays collected since 2009. These essays were collected in the context of the LONGDALE project, which aims to “build a large longitudinal database of learner English containing data from learners from

© 2013. John Benjamins Publishing Company All rights reserved

19

20 Pieter de Haan and Monique van der Haagen†

a wide range of mother tongue backgrounds and thereby contribute to filling a major gap in corpus-based SLA studies” (Granger, 2009). We started collecting essays for the LONGDALE project in July 2009, beginning with the then-first year students at Radboud University Nijmegen. Following a suggestion by Paul K. Matsuda (pers. comm., 2008), we collected varied material, including spontaneous writing, literature essays and other course work. We used WordSmith (Scott, 2004) to generate concordance lines for the adjective and adverb intensifiers. The list of intensifiers that we used was based on Lorenz (1999). It should be noted that a few idiosyncratically used intensifiers may have been missed. The non-native data discussed in this article come from two groups of students that we are still following, the 2009 and 2010 cohorts of students of English at Radboud University Nijmegen. We have analysed essays they wrote at the beginning and at the end of the first semester (an overview is presented in Table 1). For the first essay, students in both cohorts were asked to write an unprepared essay in class about their expectations of the coming year, for which they were given 30 minutes. The second essay written by the 2009 cohort was a take-home essay exam for the introductory American Studies course. The second essay written by the 2010 cohort was another in-class assignment, for which the students were, again, given 30 minutes. The topic of the latter assignment was “Critical thinking is important in academic writing”. Although students in both cohorts wrote a clearly more academically oriented essay at the end of the semester, the students in the 2009 cohort were given the opportunity to incorporate their course reading in their essay, whereas those in the 2010 cohort were not. These Dutch student essays were compared to a new selection of native student essays about a variety of topics that served as reference material. These were British and American undergraduates’ argumentative essays on a variety Table 1. Overview of the student essays Dutch EFL: Cohort 2009

Dutch EFL: Cohort 2010

text 1

text 2

text 1

timed expectations

untimed timed timed Am.Lit expectations argument.

timed untimed argument. argument.

# essays

63

26

96

42

33

25

# words

27,393

19,179

29,136

9,337

18,828

28,096

mean words/ 435 essay

738

304

222

571

1124

# intensifiers/ 100,000 wds

547

834

589

552

441

967


Native control corpus

text 2


of general and literary topics, which are part of LOCNESS (LOuvain Corpus of Native English eSSays) (Granger, 1998). The American students wrote their essays without a time limit at home and their essays can thus be compared to the second text written by the 2009 EFL cohort. The British students wrote their essays timed in class, and their essays can be seen as being similar to the other EFL essays. We take the native undergraduate essays as a norm — beyond CEFR C2 — against which our students’ non-native writing is measured because they have comparable academic backgrounds. Admittedly, the native student writers themselves are still developing as academic writers (Hamp-Lyons, 2011), so their writing may not be perfect, but it seems better to compare the Dutch students to native speaking students, as EFL students cannot be expected to have reached a level of academic sophistication that even native university students have not yet reached. 4. Results and discussion Table 2 shows the most frequent adjective and adverb intensifiers, in decreasing order of occurrence found in our data. Scores have been normalised per 100,000 words in order to avoid an effect of text length. Absolute numbers of occurrences were too small to perform any reliable statistical tests. We see that the EFL writers in their first text use nearly twice as many intensifiers as the native writers, and that the number for the EFL writers goes down in the second text, especially in the untimed 2010 essay exam. For all writers the intensifier very is preferred over all others, but this preference is more outspoken in the EFL groups than in the native control groups. We also see that the Dutch EFL students use a different selection than the natives, but that in their first text they use many more and a wider range of (sometimes inappropriate) intensifiers than in their second text. In the second text, the variety of intensifiers is much smaller, also compared to the native usage, suggesting that four months of academic input has heightened student awareness about the kind of modifiers found in English academic texts. Let us now consider a number of differences in use of individual intensifiers. The use of most is fairly high in all texts, but it is extremely high in both the 2009t2 untimed EFL essays and the native untimed essays. There is also a development in the way in which most is being used over time; in all EFL texts most frequently collocates with important, but whereas in the initial texts it is frequently used predicatively (1) or attributively with the word thing (2), in the untimed texts more meaningful nouns are used (3). In the native materials most collocates with a wider variety of adjectives, e.g. (4) and (5). © 2013. John Benjamins Publishing Company All rights reserved

21

22


Table 2. Intensifiers per 100,000 words Cohort 2009 text 1

Cohort 2010 text 2

text 1

Native control corpus text 2

timed untimed timed timed timed untimed expectations Am. Lit. expectations argument. argument. argument. N intensifiers per 100,000 words

967

547

834

589

552

441

very

230

209

251

203

138

103

most

40

109

45

21

58

96

16

18

16

43

quite

66

42

89

so

51

36

34

21

really

91

5

51

21

too

62

24

4 27

well

4

16

21

16

25

rather

55

10

34

16

7

16

24

11

4

21

10

completely

33

a bit

66

just

15

even

11

not very

26

extremely almost

11

161

25

38

7 21

5

21 10

10

16

10

21

5

directly

53

4

5

4

enough

26

3

pretty

22

14

5

7

32

far fully

11

not that

11

not … enough

11

4

32 11 14

7

more and more 7

21

21

not too fairly

14

11

5

5

11

32

10

5

3

11

not quite

11

5

7

perfectly

4

5

3

5

7

5

10

5

4

somewhat other intens.

99

36

79

54

74

64

total # different 38 intensifiers

21

42

13

34

29



(1) Overall I think it is most important to know that I enjoy studying English language and culture. (2010t1)

(2) However in my eyes that is not the most important thing. (2009t1)

(3) This, together with transcendentalism, are the most important changes caused by romanticism. (2009t2)

(4) The next step is to take the most appropriate action, which will not be difficult … (native, untimed)

(5) … she is seen variously as the most obstructive, bellicose, insular power, thwarting the path of progress, … (native, timed)

The intensifier so is also used almost equally frequently by the EFL and native writers in the untimed essays, and is usually found in combination with correlative that (as in …so boring that…). However, we do find some signs of unsophisticated use of so without that in both the non-native and native essays.

(6) A university is so different, compared to secondary school. (2009t1)

(7) Trying to find release in the past of America was an arduous task, since it was so limited. (2009t2)

(8) This usually occurs because courts are so eager to find a scapegoat for violent crimes. (native, untimed)

The use of quite and rather is very high in the spontaneous first texts in both cohorts, but drops considerably in the untimed text 2, and has disappeared altogether in the timed EFL text 2. In the first text about the students’ expectations, things were going to be quite different and quite difficult or rather hard at University. Of course in our students’ perception, quite and rather are stereotypes of Britishness, and in their first week at university students are probably keen to demonstrate this knowledge. Elsewhere we found (De Haan & Van der Haagen, forthcoming) that British sixth-formers, too, use more intensifiers in general than more mature British undergraduate writers, and we linked this to the notion that native student writers, too, are still developing their writing skills. We also found in that connection that the less mature sixth-formers use many intensifiers that are commonly seen in spoken language, a case in point being the use of really (Longman Dictionary of Contemporary English, 2009). This reflects the findings by Shaughnessy (1977, p. 201), who mentions really as one of the “common intensifiers […] which are overused” by native novice writers. We find that in the Dutch EFL essays the word really collocates especially with the most common general qualifiers, such as good, bad, and important. Below are a number of occurrences of really good and really important that were found in the timed Dutch first texts. © 2013. John Benjamins Publishing Company All rights reserved

23

24


(9) … in this direction: something I am really good at.

(10) However, losing contact with the people who are really important to me, … (11) Because my friends are really important to me. (12) This is really important, because without friends I would not feel happy. (13) The quick practice for the Cambridge exam proved insufficient for a really good mark, so … (14) .., because a really good friend of mine will buy a house there next year … (15) I expect my phonetics class to be a really important class, where I can learn …

The very few occurrences of really we found in the second texts and in the native speaker texts all used really in the sense of genuinely, as in (16). (16) We do not know though if the writer was really happy or if he just searched for an escape from his problems into nature … (2009t2)

Similarly, typically spoken intensifiers such as pretty, a bit and just (cf. Longman Dictionary of Contemporary English, 2009) occur in the first texts but no longer occur in the second texts. (17) I figured out pretty quickly that American Studies was something of my interest (2010t1) (18) Perhaps this year will be a bit chaotic (2009t1) (19) Also, I like listening to people with a British accent. It’s just beautiful. (2009t1) (20) Philosophy is just dashingly interesting and I couldn’t bear the idea of losing the interest (2010t1)

The intensifiers well and enough do not follow the general pattern of occurring more often in the initial texts and less frequently in the second essays. We see that well is the second most frequent intensifier in the timed 2010 second text, and a look at the collocations reveals that apparently, students feel that critical thinking in academic writing needs well thought out and well-reasoned arguments, presented in well-developed and well-written essays. The intensifier enough occurs frequently (alongside not … enough) in the 2009 first texts where students are expressing their worries about being (not) good enough, or the study being (not) interesting enough. Oddly enough this does not occur in the 2010 first texts, and in the 2010 second texts the usage is comparable to native usage, as in:



(21) Once students, especially first year students, are clever enough to use their sunken knowledge they will become better academic writers. (2010t2) (22) The Community was quick enough to remind the Italians that (native, timed)

Finally it is worth noting that directly does not occur in the learner data, but frequently in the timed native data in the combinations directly applicable (6 times), directly effective (2x) and directly elected. The absence of this particular intensifier, plus all the findings of infelicitous choices of intensifiers lead us to think that we should pay special attention to these features of intensification in our teaching. 5. Conclusion Let us now see how we can answer our research questions. There are noticeable quantitative and qualitative differences between native English student writers and Dutch EFL writers. Not only do the Dutch writers use more intensifiers than the native writers, they also greatly overuse the all-purpose intensifier very. However, the longitudinal Dutch EFL data show how this overuse, both in the use of intensifiers in general, and in the use of very, decreases after four months of university experience. It is interesting to observe that the Dutch EFL writers are apparently keen to show off their knowledge of English when they enter university, and try to be creative. After four months they have become more careful in their use of intensifiers, having gathered that not all intensifiers are equally appropriate in academic writing. This is shown most clearly in the 2009 cohort, who show a dramatic drop in the use of intensifiers overall, but hardly any drop in the use of very, suggesting that they do not want to take any risk in their exam essay. The comparison of the 2009 and 2010 cohorts reveals a number of differences that can help us determine what is appropriate for academic writing in a range of situations and purposes. The second text written by the 2009 cohort was an exam essay, meaning that students would be graded, primarily for content. However, students were aware that in order to express themselves clearly they would have to use the appropriate language. Moreover, they could use their course reading as input for their essays, and they had the opportunity to revise their work. For example, the high incidence of the intensifier most in the second text of this cohort is very much like that in the untimed native essays, reflecting attention to detail. In contrast, the second text written by the 2010 cohort, though academically motivated, was not graded, and students could not draw upon any course reading while they were writing it. Nor did they have the opportunity to revise. This may account for the extremely small range of intensifiers that they used overall (13 as © 2013. John Benjamins Publishing Company All rights reserved

25

26 Pieter de Haan and Monique van der Haagen†

opposed to 21in the second untimed text of the 2009 cohort), which suggests that they did not feel confident enough to use a variety of intensifiers, but were aware of the inappropriateness of many of the intensifiers they used in their first text. The comparison of either cohort of our Dutch EFL writers with the native writers, furthermore, leads us to believe that our students should be made more explicitly aware of the range of academically appropriate intensifiers. The use of these appropriate intensifiers can then seen to be a measure of advancedness and hence be used as a criterion in the assessment of advanced EFL writing quality. In order to fine-tune the measure and to monitor the effects of our teaching, we are continuing our collection of Dutch written EFL material. We are currently following the development of the 2009 and 2010 cohorts beyond the first year (De Haan & Van der Haagen, forthcoming).

References Calabrese, R. (2008). Insights into the lexicon-syntax interface in Italian learners’ English. A generative framework for a corpus-based analysis. Rome: Aracne. Casanave, C.P. (2004). Controversies in second language writing: Dilemmas and decisions in research and instruction. Ann Arbor: University of Michigan Press. Council of Europe (2009). Relating language examination to the Common European Framework of Reference for languages: Learning, teaching, assessment (CEFR): A manual. Retrieved on 19 January 2012, from http://www.coe.int/t/dg4/linguistic/CADRE_EN.asp De Haan, P. (1997). An experiment in English learner data analysis. In J. Aarts, I. de Mönnink, & H. Wekker (Eds.), Studies in English language and teaching (pp. 215–229). Amsterdam: Rodopi. De Haan, P. (1998). How native-like are advanced learners of English? In A. Renouf (Ed.), Explorations in corpus linguistics (pp. 57–67). Amsterdam: Rodopi. De Haan, P. (1999). English writing by Dutch-speaking students. In H. Hasselgård, & S. Oksefjell (Eds.), Out of corpus: Studies in honour of Stig Johansson (pp. 203–212). Amsterdam: Rodopi. De Haan, P., & Van der Haagen, M. (forthcoming). The search for sophisticated language in advanced EFL writing: A longitudinal study. In S. Granger, G. Gilquin, & F. Meunier (Eds.), Twenty years of learner corpus research: Looking back, moving ahead. Louvain-la-Neuve: Presses Universitaires de Louvain. De Haan, P., & Van der Haagen, M. (2012). Modification of adjectives in very advanced Dutch EFL writing: A development study. The European Journal of Applied Linguistics and TEFL, 1(1), 129–142. De Haan, P., & Van Esch, K. (2005). The development of writing in English and Spanish as foreign languages. Assessing Writing, 10, 100–116. De Haan, P., & Van Esch, K. (2008). Measuring and assessing the development of foreign language writing competence. Porta Linguarum, 9, 7–21. Granger, S. (2009). LONGDALE. Retrieved on 22 October 2010, from http://www.uclouvain.be/ en-cecl-longdale.html Granger, S. (Ed.). (1998). Learner English on computer. London-New York: Addison Wesley Longman.



Grant, L., & Ginther, A. (2000). Using computer-tagged linguistic features to describe L2 writing differences. Journal of Second Language Writing, 9, 123–145. Hamp-Lyons, L. (2011). Assessing the ineffable. Paper presented at the Writing Assessment in Higher Education Symposium, Amsterdam. Longman Dictionary of Contemporary English. (2009). (5th ed.). Harlow: Pearson Education Limited. Lorenz, G. (1999). Adjective intensification — Learners versus native speakers. AmsterdamAtlanta, GA: Rodopi. Melissen, M. (2007). Exameneisen havo-vwo nieuwe stijl 2007. Alphen aan den Rijn, the Netherlands: Kluwer. Nesselhauf, N. (2003). The use of collocations by advanced learners of English and some implications for teaching. Applied Linguistics, 24(2), 269–286. Ortega, L., & Byrnes, H. (Eds.). (2008). The longitudinal study of advanced L2 capacities. New York: Routledge. Scott, M. (2004). WordSmith Tools version 4.0. Oxford: Oxford University Press. Shaughnessy, M.P. (1977). Errors and expectations: A guide for the teacher of basic writing. New York: Oxford University Press.


27