Fluency versus accuracy in advanced spoken learner

John Benjamins Publishing Company

This is a contribution from International Journal of Corpus Linguistics 16:2 © 2011. John Benjamins Publishing Company This electronic file may not be altered in any way. The author(s) of this article is/are permitted to use this PDF file to generate printed copies to be used by way of offprints, for their personal use only. Permission is granted by the publishers to post this file on a closed server which is accessible to members (students and staff) only of the author’s/s’ institute, it is not permitted to post this PDF on the open internet. For any other use of this material prior written permission should be obtained from the publishers or through the Copyright Clearance Center (for USA: www.copyright.com). Please contact [email protected] or consult our website: www.benjamins.com Tables of Contents, abstracts and guidelines are available at www.benjamins.com

Fluency versus accuracy in advanced spoken learner language A multi-method approach* Christiane Brand and Sandra Götz

Justus Liebig University, Giessen / Justus Liebig University, Giessen and Macquarie University Sydney, Australia

In this paper we present a possible multi-method approach towards the description of a potential correlation between errors and temporal variables of (dys-) fluency in spoken learner language. Using the German subcorpus of the Louvain International Database of Spoken English Interlanguage (LINDSEI) and the native control corpus Louvain Corpus of Native English Conversation (LOCNEC), we first analysed errors and temporal variables of fluency quantitatively. We detected lexical and grammatical categories which are especially error-prone as well as problematic aspects of fluency for all learners in the LINDSEI subcorpus, e.g. confusion in tense agreement across clauses or an overuse of unfilled pauses. In the ensuing qualitative analysis of five prototypical learners, no trend for a possible correlation of accuracy and fluency could be observed. Fifty native speakers’ ratings of these five learners revealed that the learner with an average performance across the investigated variables received the highest ratings for overall oral proficiency. Keywords: fluency, accuracy, errors, error analysis, learner corpus, LINDSEI

1. Introduction The correlations of temporal fluency and accuracy have been discussed widely among researchers in the context of investigating foreign language proficiency. While some researchers make a clear distinction between fluency (i.e. temporal variables such as speech rate or number of filled and unfilled pauses) and accuracy (i.e. the number of errors), stating that fluency “constitutes an isolatable component of language proficiency” which is “measurable in a series of quantifiable phenomena” (Gut 2009: 79), for others fluency cannot be completely distinguished International Journal of Corpus Linguistics 16:2 (2011), 255–275. doi 10.1075/ijcl.16.2.05bra issn 1384–6655 / e-issn 1569–9811 © John Benjamins Publishing Company

256 Christiane Brand and Sandra Götz

from other factors of oral proficiency, i.e. “error and dysfluency may be difficult to separate” (Lennon 1990: 395). In the present paper, we would like to investigate whether there is a possible correlation between the number and type of errors and temporal aspects of (dys-)fluency in spoken learner language, i.e. whether the error rate increases or decreases with the degree of temporal fluency of the learner. After taking a brief look at the theoretical background of the concepts of accuracy and fluency in spoken learner language, we will present a corpus study based on the German component of the Louvain International Database of Spoken English Interlanguage, (LINDSEI). The German subcorpus, LINDSEI-GE, has recently been tagged for learner errors (cf. Kämmerer 2009) and we will investigate the frequency of errors and temporal variables to identify parallel patterns in the corpus. Selected learners will then be analysed in detail, with a focus on their most common errors and their fluency performance. In addition, we will include an experimental set-up, in which native speakers of English rate the degree of overall oral proficiency of the selected learners. The corpus findings will thus be related to the question of whether the perception of the learners’ overall oral proficiency depends on the rate and type of errors committed or whether the number and type of temporal fluency variables lead to the overall oral proficiency ratings that native speakers attribute to learners. Overall, the focus of this pilot study is methodological as we aim to discuss one possible approach to the way in which the aspect of accuracy could be merged with fluency in the evaluation of spoken learner language. 2. Aspects of fluency and accuracy 2.1 Fluency The first (and predominant) approach in linguistic fluency research is based on the definition of fluency as a “smooth, rapid, effortless use of language” (Crystal 1987: 421) and the “continuity of speech” (Koponen & Riggenbach 2000: 8). Measurements in this approach to fluency in the “narrow sense” (Lennon 1990, 2000) thus focus on automaticity and speed of speech production, and temporal variables of fluency are analysed. Since this is the mainstream approach, we have adopted it for the fluency analysis in this pilot study. Temporal fluency variables such as speech rate, length of speech runs or the number and length of filled and unfilled pauses have been found and generally accepted, among other related variables, as being the best indicators of a learner’s degree of fluency (cf. e.g. Lennon 1990, Riggenbach 1991, Chambers 1997, Cucchiarini et al. 2000 and 2002, Osborne 2008 and this issue, Gut 2009). Apart from this focus on temporal variables, there are other approaches towards fluency in speech that merit consideration,

© 2011. John Benjamins Publishing Company All rights reserved

Fluency versus accuracy in advanced spoken learner language 257

especially when fluency in the “broad sense” (Lennon 1990, 2000) is analysed, that is in the sense of overall oral proficiency. Although we do not adopt these in the context of this pilot study, it is certainly worthwhile to discuss them briefly. Fluency has often been equalled with the use of formulaic language, (semi‑) preconstructed phrases and/or recurrent sequences since they do not require extensive planning and retrieving effort (cf. e.g. Pawley & Syder 1983 and 2000, Biber et al. 1999 and 2004, Erman 2007). They can thus be metaphorically seen as a “kind of ‘autopilot’ which the speaker can switch on to gain time for the creative and social aspects of the speech process” (Altenberg & Eeg-Olofsson 1990: 2). Empirical corpus studies have revealed that learners often show an underuse in the number and/or variability of recurrent sequences, which might contribute to a decrease of fluency (e.g. De Cock 2000, Götz & Schilk 2011). A further approach towards fluency has been to investigate the use of certain performance phenomena that contribute to an impression of naturalness in learners’ speech, which can be established e.g. by a native-like use of speech management strategies, i.e. “adaptations to the needs arising from the interactive nature of realtime conversation” (Rühlemann 2006: 402), such as repeats, self-repairs, hesitation phenomena or the use of discourse markers (e.g. Biber et al. 1999). Many of these strategies have been shown to be highly underused or misused in learner language (e.g. Hasselgren 2002, Müller 2005, Rühlemann 2006, Götz 2007, Gilquin 2008). Other approaches to fluency have suggested that it may be established by other perceptive global variables in speech, such as idiomaticity (which is established on various linguistic levels, e.g. on the lexical, lexicogrammatical or syntactic level, cf. e.g. Fernando 1996), accent (e.g. Herbst 1992), intonation (e.g. Trofimovich & Baker 2006, Wells 2006), the range of lexical diversity (e.g. Malvern & Richards 1997), syntactic complexity (e.g. Towell et al. 1996), or the use of certain pragmatic features, for instance a native-like realisation of certain speech acts (e.g. Cutting 2008). Although a combination of these approaches would certainly be promising, in the context of this pilot study we will adopt the “narrow sense” of fluency (Lennon 1990, 2000) only and will thus investigate temporal variables of fluency. 2.2 Accuracy There are different approaches to measuring accuracy in learner output. Following the most dominant view, which is also used in most standardized tests, for example by the Common European Framework of Reference for Languages (Council of Europe 2001), the level of oral competence is primarily assessed in terms of grammatical, lexical and phonological proficiency. In addition, the appropriate contextual use of lexical and syntactic items is taken into account since sociolinguistic and pragmatic competence is regarded as necessary for relating language knowledge



and skills successfully to the situations and domains of communication. The second way to measure accuracy is to focus on deficiency rather than on proficiency, that is to pay attention to the errors at the lexical and syntactic level and to the incorrect use of items and constructions. According to Corder (1974), an error is regarded as a failure in competence and thus as a systematic fault, whereas a mistake is simply seen as a flaw in performance. He explains the significance of analysing learners’ errors in three different ways: First to the teacher in that they tell him, if he undertakes a systematic analysis, how far towards the goal the learner has progressed, and consequently what remains for him to learn. Second, they provide to the researcher evidence of how language is learned or acquired, what strategies or procedures the learner is employing in his discovery of the language. Thirdly (and in a sense this is their most important aspect) they are indispensable to the learner himself, because we can regard the making of errors as a device the learner uses in order to learn. (Corder 1984: 25)

We are certainly aware that it is notoriously difficult to distinguish errors from unsystematic mistakes and infelicities, which also occur in native speech, but we decided to count every major breach of a grammatical or lexical rule as an error (see Section 3 for the detailed error-tagging procedure), while acknowledging that the proficiency of a learner is certainly to be regarded as multifaceted and its description involves more than the mere focus on deficiency. 3. Research questions, database and methodology If, as Lennon (1990) suggests, errors and dysfluencies are difficult to separate and both are indicators for the proficiency of learners, is there an observable correlation between accuracy and fluency, and, if this is so, how do the two variables relate to each other? One would certainly assume that the more errors a learner commits, the poorer their performance according to their temporal variables becomes, but could it also be the other way around? Can a learner show a high error rate and yet be very fluent as far as temporal variables are concerned? The aim of this paper is to present a multi-method investigation that addresses the nature of such a possible correlation. As mentioned in the introduction, we structured our investigation into three major parts: first a quantitative corpus analysis of error rate and temporal fluency variables was conducted, which was supplemented by a detailed qualitative investigation of the output of five German learners that were chosen on the basis of their performance in the quantitative analysis. In a second step we compared these five learners’ fluency and accuracy



performances in order to find possible trends for a correlation between the two. In a third step we asked fifty native speakers of English to rate the overall degree of oral proficiency of these learners to see if their judgements might be correlated with either the findings from the learners’ fluency or error analyses. The database for this study was the error-tagged version of the German component of the Louvain International Database of Spoken English Interlanguage (LINDSEI-GE).1 LINDSEI is the first large-scale multinational corpus of advanced spoken learner English and consists of several national subcorpora containing data gathered from learners with different mother-tongue backgrounds such as Bulgarian, French, German, Italian, Japanese, Swedish and Spanish (cf. Gilquin et al. 2010). There is also a comparable corpus of native speakers of English, the Louvain Corpus of Native English Conversation, LOCNEC, which provides a basis for the comparison of interlanguage and native language. The German component of LINDSEI was compiled, transcribed (cf. Brand & Kämmerer 2006), and error-tagged (cf. Kämmerer 2009) at the University of Giessen. In the first phase the data were collected and the recordings were transcribed according to pre-defined LINDSEI standards. LINDSEI-GE, like the other components of the corpus, includes fifty interviews which were conducted in welldefined settings to guarantee the collection of comparable data, e.g. duration of the interviews, restricted set of topics and comparable proficiency levels. In the second phase of the project, the German component was tagged for learner errors. Each error in the corpus (consisting of 86,186 words) has been manually identified, categorized and coded for a correct target hypothesis according to the Error Tagging Manual (cf. Dagneaux et al. 2005). The main categories used to tag errors in our spoken learner corpus were grammatical (G-tag), lexico-grammatical (Xtag) and lexical errors (L-tag), which are divided into sub-categories with specific tags (e.g. GADVO for Grammar, Adverbs, Order, which marks the wrong placement of adverbs). The correct form is indicated by the symbol $ (e.g. We could travel (GADVO) a little around $around a little$ [ger028]). Further categories include word redundancy, missing word or wrong word order (W-tag), as well as style and infelicities (S- and Z-tags). 4. Findings 4.1 Quantitative Analysis 4.1.1 Quantitative error analysis The first step in our pilot study was to identify the number of errors. In total, 1,365 errors were detected in our 86,186-word corpus. Figure 1 gives the error mean and


260 Christiane Brand and Sandra Götz 4 error mean 3

2

0

ger001 ger002 ger003 ger004 ger005 ger006 ger007 ger008 ger009 ger010 ger011 ger012 ger013 ger014 ger015 ger016 ger017 ger018 ger019 ger020 ger021 ger022 ger023 ger024 ger025 ger026 ger027 ger028 ger029 ger030 ger031 ger032 ger033 ger034 ger035 ger036 ger037 ger038 ger039 ger040 ger041 ger042 ger043 ger044 ger045 ger046 ger047 ger048 ger049 ger050

1

Figure 1. Errors per one hundred words in LINDSEI-GE

the number of errors per 100 words (phw) committed by the individual learners in the German component of LINDSEI. The results show that the average German learner (indicated by the dashed horizontal line) can be regarded as fairly accurate since the mean error rate is 1.63 errors phw. Obviously, there are deviations from this mean. Learner ger024 is the least accurate with 3.38 errors phw (58 errors in total), whereas learner ger027 is the most accurate with only 0.4 errors phw (6 errors in total). With an error rate of 1.2 to 1.5 phw, learners ger001, ger028 and ger041 commit between 20 and 30 errors in total, which more or less corresponds to the prototypical German learner in this corpus.2 4.1.2 Quantitative analysis of temporal fluency variables As we have outlined in Section 2.1, our corpus-based fluency analysis covers temporal fluency variables only. We investigated speech rate in words per minute (wpm), the number of filled pauses (er/erm/eh/em) phw and the number of unfilled pauses (silences) phw. However, we did not take the length of unfilled pauses into consideration since it has been shown by Cucchiarini et al. (2002: 2870) that “less fluent speakers, in general, do not make longer pauses than more fluent speakers, but they do pause more often” and that the “mean length of silent pauses seem[s] to have almost no relation at all with perceived fluency”. In the more detailed fluency analysis the study of pause frequency will be supplemented with an investigation of the positions of both filled and unfilled pauses in order to distinguish functional pauses at clause boundaries from hesitation pauses within clauses, which render the speech less fluent, cf. Lounsbury (1954) and Section 4.2.2 below.



The first temporal variable analysed is speech rate. We analysed the learners’ speech rates in words per minute following Lennon’s (1990) approach of calculating unpruned words, i.e. we included all the words uttered by the speakers, including self-corrections, repetitions and asides, as well as all filled pauses and other hesitation or performance phenomena. Short unfilled pauses under 3 seconds were not cut out, but were not counted as words in our word-count. From the learners’ output, we only excluded all sorts of non-verbal sounds (i.e. laughter, coughing, sneezing, sighing) as well as periods of silence over 3 seconds, which were clearly induced by content planning (e.g. before they started the third part of the interview, a picture story retelling). The findings for LINDSEI-GE are illustrated in Figure 2, which shows the fifty learners of LINDSEI-GE and their speech rates. The learners’ mean is 160 wpm (indicated by the dashed line), with the slowest speaker (ger046) speaking 117 wpm and the fastest speaker (ger001) 190 wpm. Compared to the native speakers’ mean of 218 wpm (obtained from the mean of the interviewees of LOCNEC and indicated by the continuous line in Figure 2), all the learners produce fewer words per minute than the native speaker mean represents. However, when listening to the sound files we find that this might not necessarily be due to the learners speaking at a much slower pace than the native speakers, but — since we did not exclude hesitations from the analysis — might rather be caused by a higher number of filled and unfilled pauses in the learners’ output which, in turn, reduces the learners’ speech rate (because filled pauses often co-occur with unfilled pauses which are not counted as words in our analysis). The findings for filled pauses phw (unpruned) are illustrated in Figure 3. From a quantitative perspective, the learners show great variation in their use of filled pauses, ranging from only 1 filled pause phw (ger011) to 14 phw (ger003). The 240 220 200 180 160 140 120 100 80

LINDSEI-GE mean LINDSEI-GE mean LOCNEC mean LOCNEC mean

60 40 20

ger001 ger002 ger003 ger004 ger005 ger006 ger007 ger008 ger009 ger010 ger011 ger012 ger013 ger014 ger015 ger016 ger017 ger018 ger019 ger020 ger021 ger022 ger023 ger024 ger025 ger026 ger027 ger028 ger029 ger030 ger031 ger032 ger033 ger034 ger035 ger036 ger037 ger038 ger039 ger040 ger041 ger042 ger043 ger044 ger045 ger046 ger047 ger048 ger049 ger050

0

Figure 2. Speech rate Figure 2. Speech rate in in words wordsper perminute minutein inLINDSEI-GE LINDSEI-GE


262 Christiane Brand and Sandra Götz 16 14

LINDSEI-GE mean

12

LOCNEC mean

10 8 6 4 2 ger001 ger002 ger003 ger004 ger005 ger006 ger007 ger008 ger009 ger010 ger011 ger012 ger013 ger014 ger015 ger016 ger017 ger018 ger019 ger020 ger021 ger022 ger023 ger024 ger025 ger026 ger027 ger028 ger029 ger030 ger031 ger032 ger033 ger034 ger035 ger036 ger037 ger038 ger039 ger040 ger041 ger042 ger043 ger044 ger045 ger046 ger047 ger048 ger049 ger050

0

Figure 3. Figure 3. Filled Filledpauses pausesper perone onehundred hundredwords wordsininLINDSEI-GE LINDSEI-GE

native-speaker mean obtained from LOCNEC is 2.27 phw and is indicated by the continuous line; the LINDSEI-GE mean, indicated by the dashed line, is 5.12 phw and as far as the totals of the two corpora are concerned there is an overall highly significant overuse of filled pauses in LINDSEI-GE (a total of 4,401 vs. 2,686 filled pauses results in G2 > 15.13, p 3.84, p 6.63, p 3.84, p 6.63, p 15.13, p learner performance) or as learner performance / NS mean (if NS mean < learner performance). We decided to do this because we wished to take into account the fact that the learners may not have internalised the complete nativelike variety of variables that contribute to fluency and, as a result, may use one variable much more frequently than another to establish their spoken fluency. In other words, they may show a very poor performance concerning one fluency variable, but “make up for that”, as it were, by a very good performance in another and thus may establish their overall fluency performance through different means. For example, while ger001’s performance in UPs in total and UPs within clauses is at respectively 40.97% and 36.84% relative to the NS-mean, the learner uses very few FPs in total and within clauses, which are at 125.41% and at 104.00% respectively and thus even better than the NS-mean, thereby increasing her overall fluency. The findings for these analyses for the five selected learners are summarised in Table 1. According to the investigated variables shown in Table 1, no overall fluency judgements for all five learners can be made since the learners’ performances vary across the temporal variables (for similar observations, see Osborne this issue). Nevertheless, some tendencies in the five learners’ fluency performances are noteworthy. Learner ger001 can be considered the most fluent speaker with the fastest speech rate and a very low number of filled and unfilled pauses, both in total and within clauses. Learner ger041 can be considered the least fluent speaker



Table 1. Temporal fluency variables of selected learners of LINDSEI-GE Learner

Speech rate

UP total

wpm relation phw to NSmean (218) in %

UP within clauses

FP total

FP within clauses

relation phw relaphw relation phw relation to NStion to to NSto NSmean NS-mean mean mean (3.99) (1.96) (2.27) (0.78) in % in % in % in %

ger001 190

87.16

9.74

40.97

5.32

36.84

1.81

125.41

0.75

104.00

ger024 175

80.28

17.29

23.08

8.79

22.30

3.14

72.29

1.80

43.33

ger027 169

77.52

19.18

20.80

7.20

27.22

6.33

35.86

2.15

36.28

ger028 154

70.64

15.54

25.68

8.02

24.44

2.08

109.13

0.59

132.20

ger041 154

70.64

14.33

27.84

8.32

23.56

6.85

33.14

4.09

19.07

despite the second lowest number of unfilled pauses phw, because she displays the slowest speech rate and the highest proportion of filled pauses both in total and within clauses, as well as the second highest number of unfilled pauses within clauses. Learners ger024, ger027 and ger028 show different fluency performances concerning the different variables. While the speech rate of learner ger028 is rather slow, she displays a low frequency of use of both types of pauses (especially filled pauses) in total and within clauses. Learner ger024 speaks very fast and her use of filled pauses in total and within clauses is relatively low, but her frequency of unfilled pauses is higher than that of the others (except ger027 for UP total), both in total and within clauses. Learner ger027’s performance shows an average speech rate and the second highest number of filled pauses, but shows the highest overall number of unfilled pauses. As the performances of the learners are not consistent across the variables, we computed an overall fluency score for each of them in order to be able to compare and rank the learners in terms of their overall temporal fluency. In order to do so, we combined the values of the five temporal variables analysed and set the mean of this calculation as their fluency score (see second column of Table 2). Table 2 shows that ger001 can be considered the most fluent speaker of the five, according to the investigated variables, with a score of 78.88% (relative to the NS-means), followed by ger028 at 72.42% and ger024 at 47.02%. Learner ger027 has the second lowest score at 39.54% and ger041 has the lowest score at 34.85%, both scores representing half and even less than half, respectively, of the most fluent learner’s. These considerable differences in fluency performance between the learners might give rise to the assumption that some speakers of LINDSEI-GE are more proficient



than others. It would thus be interesting to see if their performance deviates equally strongly from other variables we did not investigate in this pilot study. 4.3 Correlation between accuracy and fluency: some trends In order to see if we can find any trends for a possible correlation between fluency and accuracy of the five learners we investigated, we combined the findings of the accuracy and the fluency analysis, cf. Table 2. Table 2. Correlation between fluency and accuracy of selected learners of LINDSEI-GE Learner

Temporal fluency score

Accuracy (in errors phw)

ger001

78.88

1.26

ger024

47.02

3.38

ger027

39.54

0.40

ger028

72.42

1.53

ger041

34.85

1.28

Correlation

r = 0.14, p > 0.05

What becomes visible from the comparison is that learner ger024, who commits the highest number of errors, is neither the most nor the least fluent speaker and thus ranges somewhere around the average. Learner ger027 is the one with the lowest number of errors and shows the second lowest overall fluency score. The most fluent speaker, ger001, shows a relatively low number of errors, which is, however, almost equal to ger041, who was found to be the least fluent speaker of the five. Considering all the investigated variables, ger028 can be considered comparatively fluent, but ranges on an average level concerning the level of accuracy. Therefore, from the analysis of these five learners, we cannot detect any clear correlation between the learners’ performance in the temporal fluency variables and the parameters of accuracy in our pilot study. This is confirmed by a test of statistical significance (r = 0.14, p > 0.05). Correlations might, however, be detected using a larger sample. Nevertheless, in this context, our findings raise the question which of the factors, if any, might influence native speakers’ perception most when they are asked about the overall oral proficiency of the learners under scrutiny, which we would like to address in the following section.



4.4 The native-speaker perception In an online survey we asked fifty native speakers of English to listen to the sound files of the interviews carried out with the five selected learners. The native speakers were Australian, British and American English speaking staff and PhD students from the faculties of Arts and Human Sciences of Macquarie University, Sydney, consisting of linguists and non-linguists with a certain degree of “linguistic awareness”, so as to grasp the possible perceptive differences of trained vs. untrained listeners. After listening to each of the interviews they were asked to rate their perception of the overall oral proficiency of the learners from 1 (sounds like an absolute beginner) to 10 (sounds like a native speaker).5 We summarised the means of these fifty ratings (with an inter-rater reliability of 89%, calculated through an intraclass correlation of ICC = 0.89) and compared them to the accuracy and fluency findings, as shown in Table 3. Table 3. NS-ratings of overall proficiency compared to fluency and accuracy of selected learners of LINDSEI-GE Learner

Temporal fluency score

Accuracy (in errors phw)

NS-ratings of overall proficiency (N = 50, ICC = 0.89)

ger001

78.88

1.26

7.70

ger024

47.02

3.38

7.68

ger027

39.54

0.40

6.80

ger028

72.42

1.53

7.96

ger041

34.85

1.28

7.10

Correlation with NS-ratings

r = 0.788, p > 0.05

r = 0.560, p > 0.05

As can be seen in Table 3, ger024 with the highest number of errors gets a relatively high average rating of 7.68 whereas ger027, who is by far the most accurate, gets the lowest overall rating of 6.80. Accordingly, the number of errors is not strongly correlated with the native speakers’ ratings of the learners’ overall proficiency (r = 0.560, p > 0.05). This might lead to the conclusion that for advanced learners other variables of speech production might become perceptively more important than accuracy, once they have reached a certain proficiency level. This is a noteworthy finding given the high weight that is put on accuracy in foreign language teaching in Germany (e.g. as one of the central descriptors in the Common European Framework of Reference, Council of Europe 2001: 28). As far as the comparison of temporal fluency and the overall proficiency ratings is concerned, Table 3 shows that ger001, who is the most fluent speaker



according to our analysis, received the second highest rating of 7.70, whereas the least fluent speaker, ger041, received the second lowest rating of 7.10 (which is, however, only 0.6 points lower than that of the second most fluent speaker). Although the learners’ fluency scores are correlated more strongly with the overall proficiency ratings than their scores for accuracy, no statistical significance can be attested (r = 0.788, p > 0.05). This could indicate that temporal variables might contribute somewhat more to the impression of oral proficiency than accuracy, but this would warrant further and more detailed investigations. Most interestingly, the highest overall rating of 7.96 was given to ger028, who we found to be comparatively fluent and mid-accurate. This might lead to the assumption that other variables we did not include in this study might have an even stronger impact. Of course we have to interpret these findings with some caution and reservation, because five learners are certainly not a representative sample from which general trends can be identified. However, the ratings of these five learners showed that temporal variables seem to weigh more than the level of accuracy in a native speaker’s perception of overall oral proficiency, although neither of the variables has a significant impact and it rather suggests that a good performance across several variables might be perceptively more relevant. Also, and maybe even more importantly, when it comes to advanced learners who are already fairly fluent and accurate compared to beginners or intermediate speakers, the native speaker perception is probably more strongly based on other variables we did not investigate in this pilot study, e.g. accent, intonation, pragmatic features, idiomaticity, register, sentence structure or lexical diversity. 5. Conclusion and outlook The initial quantitative analysis of the German component of LINDSEI has shown that the fifty learners have to be regarded as fairly advanced and accurate as far as the overall (low) frequency of errors is concerned, but that we find great variation in oral performance when investigating fluency variables across the individual speakers. The qualitative error analysis of five selected learners has shown that there are clear patterns in the error distribution across the learners, which are indicative of typical problem areas of German learners that could be regarded as quite systematic. Yet again, the qualitative analysis of temporal fluency variables did not exhibit any clear trends of patterns of the five learners across the variables. Since the major focus of our paper was a methodological one, we hope that we were able to show one possible way of how various aspects of learner output can be jointly investigated. However, our combination of the findings of the accuracy and fluency analyses did not reveal a significant correlation between the two. We



thus had fifty native speakers of English rate five selected learners in order to see to what extent fluency and accuracy parameters exert an influence on native speakers’ overall assessment of learners’ oral proficiency. When the native speakers were asked for their rating of the degree of the overall oral proficiency of the five learners, the learner who received the highest average rating was the one whose performance ranged on an average level for all of the investigated variables. This suggests that the perception of overall oral proficiency is not based on good performance in one single variable only, but rather results from good performance across several variables. Native-speaker perception might in addition be influenced by other features that we could not investigate in our pilot study, e.g. intonation, pragmatic features or accent. Experimental set-ups that take these factors into consideration are certainly worthwhile for future research. One major caveat of our pilot study relates to the small number of learners we analysed qualitatively. Of course, this makes it difficult to draw general conclusions with regard to possible trends, let alone to define a well-grounded correlation between accuracy and fluency. Moreover, when fluency is investigated in the broad sense, many more and different variables need to be taken into consideration, because fluency is a metacategory that is based, apart from temporal fluency, on a range of other linguistic features and effects. This caveat notwithstanding, the present pilot study has revealed that the method we introduced in this paper, combining fluency parameters and error analysis, proved to be feasible and thus may be promising for future research. Therefore, in order to unveil more fine-grained aspects of possible correlations of variables other than accuracy and fluency in speech with the overall perception of native speakers, a detailed qualitative analysis of all fifty learners of LINDSEI-GE is needed, along with a statistical multivariate analysis of all these different variables of speech production to be able to see where possible correlations become visible. Also, it would be highly desirable to match our results with other national subcorpora of LINDSEI in order to shed light on similarities and differences across the different learner populations with different L1 backgrounds.

Notes * We would like to thank the participants of the ICAME 30 pre-conference workshop on ‘Errors and disfluencies in spoken corpora’ for a stimulating discussion, Rosemary Bock and Patrick Maiwald for proofreading this manuscript, Johannes Herrmann for statistical support and Joybrato Mukherjee, Gaëtanelle Gilquin, Sylvie De Cock and two anonymous reviewers for their various valuable comments on earlier versions of this paper. Of course, all remaining errors or infelicities are our responsibility alone.



1. This study is based on a previous version of LINDSEI-GE, which may differ slightly from the final version included on the LINDSEI CD-ROM. 2. The types of errors committed by the learners are discussed in the qualitative analysis in Section 4.2.1. 3. Please see for reference and detailed description: Error Tagging Manual Version 1.2 (Dagneaux et al. 2005). 4. In the examples taken from LINDSEI-GE, consider some of the transcription conventions of LINDSEI: Unfilled pauses are transcribed with a single dot “.” for a pause under 1 second, two dots “..” for an unfilled pause of 1–3 seconds and three dots “…” for an unfilled pause of over 3 seconds. Non-verbal vocal sounds are put in ; when an error occurred, the error category is put in () before the error; a suggested correct target hypothesis is put between $-signs afterwards. 5. The ethical aspects of this study have been approved by the Macquarie University Ethics Review Committee (Human Research). If you have any complaints or reservations about any ethical aspect of this research, you may contact the Ethics Review Committee through its Secretary (Tel: +61 (0)2 9850 7854; Email: [email protected]). Any complaint you make will be treated in confidence and investigated, and you will be informed of the outcome.

References Altenberg, B. & Eeg-Olofsson, M. 1990. “Phraseology in spoken English: Presentation of a project”. In B. Aarts & W. Meijs (Eds.), Theory and Practice in Corpus Linguistics. Amsterdam: Rodopi, 1–26. Biber, D., Johansson, S., Leech, G., Conrad, S. & Finegan, E. 1999. Longman Grammar of Spoken and Written English. Harlow: Pearson Education. Biber, D., Conrad, S. & Cortes, V. 2004. “If you look at…: Lexical bundles in university teaching and textbooks”. Applied Linguistics, 25 (3), 371–405. Brand, C. & Kämmerer, S. 2006. “The Louvain International Database of Spoken English Interlanguage (LINDSEI): Compiling the German component”. In S. Braun, K. Kohn & J. Mukherjee (Eds.), Corpus Technology and Language Pedagogy: New Resources, New Tools, New Methods. Frankfurt am Main: Peter Lang, 127–140. Butcher, A. 1980. “Pause and syntactic structure”. In H. W. Dechert & M. Raupach (Eds.), Temporal Variables in Speech. The Hague: Mouton de Gruyter, 85–90. Chambers, F. 1997. “What do we mean by fluency?”. System, 25 (4), 535–544. Corder, S. P. 1974. “Error analysis”. In J. P. B. Allen & S. P. Corder (Eds.), Techniques in Applied Linguistics. London: Oxford University Press, 122–154. Corder, S. P. 1984. “The significance of learners’ errors”. In J. C. Richards (Ed.), Error Analysis: Perspectives on Second Language Acquisition. Essex: Longman, 19–27. Council of Europe. 2001. Common European Framework of Reference for Languages: Learning, Teaching, Assessment. Cambridge: Cambridge University Press. Crystal, D. 1987. The Cambridge Encyclopedia of Language. Cambridge: Cambridge University Press.


274 Christiane Brand and Sandra Götz Cucchiarini, C., Strik, H. & Boves, L. 2000. “Quantitative assessment of second language learners’ fluency by means of automatic speech recognition technology”. Journal of the Acoustical Society of America, 107 (2), 989–999. Cucchiarini, C., Strik, H. & Boves, L. 2002. “Quantitative assessment of second language learners’ fluency: Comparisons between read and spontaneous speech”. Journal of the Acoustical Society of America, 111 (6), 2862–2873. Cutting, J. 2008[2002]. Pragmatics and Discourse: A Resource Book for Students. 2nd ed. London: Routledge. Dagneaux, E., Denness, S., Granger, S., Meunier, F., Neff, J. & Thewissen, J. 2005. Error Tagging Manual Version 1.2. Université catholique de Louvain. De Cock S. 2000. “Repetitive phrasal chunkiness and advanced EFL speech and writing”. In C. Mair & M. Hundt (Eds.), Corpus Linguistics and Linguistic Theory. Papers from the Twentieth International Conference on English Language Research on Computerized Corpora (ICAME 20). Amsterdam: Rodopi, 51–68. Erman, B. 2007. “Cognitive processes as evidence of the idiom principle”. International Journal of Corpus Linguistics, 12 (1), 25–53. Fernando, C. 1996. Idioms and Idiomaticity. Oxford: Oxford University Press. Gilquin, G. 2008. “Hesitation markers among EFL learners: Pragmatic deficiency or difference?”. In J. Romero-Trillo (Ed.), Pragmatics and Corpus Linguistics: A Mutualistic Entente. Berlin/ Heidelberg/New York: Mouton de Gruyter, 119–149. Gilquin, G., De Cock, S. & Granger, S. (Eds.) 2010. The Louvain International Database of Spoken English Interlanguage. Handbook and CD-ROM. Louvain-la-Neuve: Presses universitaires de Louvain. Götz, S. 2007. “Performanzphänomene in gesprochenem Lernerenglisch: Eine korpusbasierte Pilotstudie”. Zeitschrift für Fremdsprachenforschung, 18 (1), 67–84. Götz, S. & Schilk, M. 2011. “Formulaic sequences in spoken ENL, ESL and EFL: Focus on British English, Indian English and Learner English of advanced German learners”. In J. Mukherjee & M. Hundt (Eds.), Exploring Second-language Varieties of English and Learner Englishes: Bridging a Paradigm Gap. Amsterdam/Philadelphia: John Benjamins, 81–102. Gut, U. 2009. Non-native Speech. A Corpus-based Analysis of Phonological and Phonetic Properties of L2 English and German. Frankfurt am Main: Peter Lang. Hasselgren, A. 2002. “Learner corpora and language testing: Smallwords as markers of learner fluency”. In S. Granger, J. Hung & S. Petch-Tyson (Eds.), Computer Learner Corpora, Second Language Acquisition and Foreign Language Teaching. Amsterdam/Philadelphia: John Benjamins, 143–173. Herbst, T. 1992. “Pro-Nunciation: Zur Bedeutung einer guten Aussprache in der Fremdsprache”. Die Neueren Sprachen, 91 (1), 2–18. Kämmerer, S. 2009. “Error-tagging spoken features of (learner) language: The UCL Error Editor ‘revised’ ”. Paper presented at the 30th annual conference of the International Computer Archive of Modern and Medieval English (ICAME30), Lancaster, 27–31 May. Koponen, M. & Riggenbach, H. 2000. “Overview: Varying perspectives on fluency”. In H. Riggenbach (Ed.), Perspectives on Fluency. Ann Arbor: The University of Michigan Press, 5–25. Lennon, P. 1990. “Investigating fluency in EFL: A quantitative approach”. Language Learning, 40 (3), 387–417.



Lennon, P. 2000. “The lexical element in spoken second language fluency”. In H. Riggenbach (Ed.), Perspectives on Fluency. Ann Arbor: The University of Michigan Press, 25–42. Lounsbury, F. 1954. “Transitional probability, linguistic structure and systems of habit-family hierarchies”. In C. Osgood & T. Sebok (Eds.), Psycholinguistics: A Survey of Theory and Research Problems. Bloomington: Indiana University Press, 93–101. Malvern, D. D. & Richards, B. J. 1997. “A new measure of lexical diversity”. In A. Ryan & A. Wray (Eds.), Evolving Models of Language. Papers from the Annual Meeting of the British Association of Applied Linguists held at the University of Wales, Swansea, September 1996. Clevedon: Multilingual Matters, 58–71. Müller, S. 2005. Discourse Markers in Native and Non-native English Discourse. Amsterdam/ Philadelphia: John Benjamins. O’Connell, D. C. & Kowal, S. 2004. “The history of research on the filled pause as evidence of The Written Language Bias in Linguistics (Linell, 1982)”. Journal of Psycholinguistic Research, 33 (6), 459–474. Osborne, J. 2008: online. “Oral learner corpora and assessment of speaking skills”. Paper presented at the 8th Teaching and Language Corpora Conference, Lisbon, 3–6 July. Available at: http://www.webcef.eu/files/Talc8.pdf (accessed August 2010). Pawley, A. & Syder, F. H. 1983. “Two puzzles for linguistic theory: Native-like selection and native-like fluency”. In J. Richards & R. Schmidt (Eds.), Language and Communication. London: Longman, 191–226. Pawley, A. & Syder, F. H. 2000. “The one-clause-at-a-time hypothesis”. In H. Riggenbach (Ed.), Perspectives on Fluency. Ann Arbor: The University of Michigan Press, 163–199. Riggenbach, H. 1991. “Toward an understanding of fluency: A microanalysis of nonnative speaker conversations”. Discourse Processes, 14 (4), 423–441. Rühlemann, C. 2006. “Coming to terms with conversational grammar: ‘Dislocation’ and ‘dysfluency’ ”. International Journal of Corpus Linguistics, 11(4), 385–409. Towell, R., Hawkins, R. & Bazergui, N. 1996. “The development of fluency in advanced learners of French”. Applied Linguistics, 17 (1), 84–119. Trofimovich, P. & Baker, W. 2006. “Learning second language suprasegmentals: Effect of L2 experience on prosody and fluency characteristics of L2 speech”. Studies in Second Language Acquisition, 28 (1), 1–30. Wells, J. C. 2006. English Intonation — An Introduction. Cambridge: Cambridge University Press.

Authors’ addresses Christiane Brand Justus Liebig University, Giessen Department of English English Language and Linguistics Otto-Behaghel-Str. 10 B 35390 Giessen, Germany

Sandra Götz Justus Liebig University, Giessen Department of English English Language and Linguistics Otto-Behaghel-Str. 10 B 35390 Giessen, Germany

[email protected]

[email protected]
