Testing in EAP: Progress? Achievement? Proficiency? In G. Blue, J.Milton, and J. Saville (Eds.), Assessing English for Academic. Purposes Berne: Peter Lang.
Short-term changes in Complexity, Accuracy and Fluency: Developing Progress-sensitive Proficiency Measures Alan Tonkyn Department of Applied Linguistics, University of Reading, UK. BACKGROUND There is continuing uncertainty about the nature and extent of progress that instructed second/foreign language (L2) learners can make in a given period. Recent e-mail correspondence to the LTEST-L list has indicated that language assessors are often pessimistic about being able to measure progress, even on intensive courses, with one scholar (A. Huhta) citing as reasons: the insensitivity of the tests, the shortness of the courses, the relatively high level of initial proficiency of the students and the lack of explicit language teaching. Alderson (2000) has urged language testers to develop progress-sensitive proficiency tests, and this study aims to contribute to that. Theory suggests that instruction and feedback should promote attention to form (Schmidt, 2001), and thereby complexity and accuracy; more frequent use of forms in and out of class should promote fluency (Ellis 2002; Johnson, 1996). However, theory also suggests that skill-focused courses may provide fewer opportunities for noticing and intensive practice, and that pressure to use the existing interlanguage in the L2 country may automatise error (Klein, 1986)
Figure 7: Significant band group differences: accuracy features (Group av.s)
Figure 1: Significant gains in overall complexity (Group av.s)
70
800 673
700
50
500 Int 1
400
THE DATA •24 learners on a university pre-study English course, 19 male, 5 female, median age:30 •2 parallel structured interviews (Int 1 & 2), 9 weeks apart, based on data regarding the Ss’ own discipline and their English language learning experiences •Audio recordings of data transcribed, segmented into Analysis-of-Speech Units (ASU’s) (Foster et al. 2000) and 66 ASU’s analysed for the features shown in Table 1 • Ratings of the recordings (in random order) by a panel of 4 IELTS assessors: overall IELTS band, and C, A and F ratings.
Grammatical Complexity Words Botel Dawkins Granowski (BDG) complexity measure Subordinate clauses NP: a) Premod.s; b) Postmod.s VP: a) Primary aux. b) Modal aux. c) Catenatives Adverbials: a) Adverbs; b) Prep. P’s
100 10
0 Words
BDG
4.8
7.1
0
Words/error
Figure 2: Significant gains in specific complexity features (Group av.s)
Words/VP error
Words/syntax error
Figure 8: Significant band group differences: fluency features (Group av.s) 200
45
39
40 30 25
160
27
24 18
15
120
Int 1
100
Int 2
80
Band 5 Band 6
60
9
10
40
4
2
5
20
0
5.7
7.1
13.7
3.8
0
Sub. Cl.
Modals
Caten. Vs
Speaking rate (sylls/min)
Adv.s
Figure 3: Significant accuracy gains (Group av.s) 60
55.4
40
29.9
28.6
30
24.1
Int 1 Int 2
20
6.4
7.6
0
Words/error
Words/VP error
Words/NP error
Figure 4: Significant fluency gains (Group av.s) 8
7.4
7
147.5
140
20
10
178.4
180
35
35
6.6
6 5 4
4
Int 1 Int 2
3
3
Fluent runs (syll.s)
Pause clusters
RESULTS and DISCUSSION Complexity: The more general complexity measures (Words, Subordinate clauses, or the BDG measure, which draws on several complexity features) seem to be better progress-sensitive indices and better aligned with judges’ assessments of adjacent proficiency levels than specific VP, NP or Adverbial features. (Fig.s 1, 2, 5 and 6) Accuracy: Overall Error density, and Error frequency in the VP seem to be promising indices of progress, and to be aligned with judges’ views of level. Syntax errors (e.g. word order errors or constituent omission) are less likely to show shortterm gains, but seem very influential in judges’ assessment of level. (Fig.s 3 and 7) Fluency: These data showed surprisingly limited fluency gains over time, with Fluent runs and Turn length the only significant cases of short-term progress. However, Speaking rate and frequencies of ‘weighty’ Pause clusters appeared to be strong influences on judges’ assessments of band group. (Fig.s 4 and 8)
2 1 0 Fluent runs (syll.s)
Turn length (ASU's)
700
600
623 554
500
400
Band 5 Band 6
300
172.9
200
206.6
100
0
BDG
Figure 6: Significant band group differences: specific complexity features (Group av.s) 30
26.2
25
Fluency Rate of speaking (syll.s / min.) Length of fluent runs (syll.s) Phonation time / Total speaking time Proportion of pause time at ‘text unit’ boundaries (Garman 1990) Av. Turn length (ASU’s) Non-extraneous words / Total words Pause clusters (silent-filled-silent)
Band 5 Band 6
28.2 20.8
20
Words
Language Accuracy Words / error Error-free ASU’s / Total ASU’s Words / VP error Words / NP error Words / syntax error Words / lexical error Words / error-free ASU
34.1
30
226
192
200
Figure 5: Significant band group differences (Int 1): Overall complexity (Group av.s) Table 1: Objective measures
40
Int 2
300
50
THE QUESTIONS 1. What changes in the oral proficiency of instructed intermediate/upper intermediate learners of English as L2 occur during a typical intensive EAP course? 2. How are objective measurements of the preand post-instruction performances of such learners related to subjective ratings?
63.5
60
598 600
20
19.2
15
13.1
10
7.2
5 0 Sub. Cl.
Primary aux.
Band 5 Band 6
Progress-sensitive rating: This study revealed again the problematic nature of ‘online’ subjective assessment of proficiency if sensitivity to progress is important. Judges’ comments revealed, for example, that high Complexity may not be perceived if turns are short, if speech is disfluent, or if a particular structure is repeated frequently. On the other hand, relatively simple syntax may be over-rated if it is delivered in a confident and fluent manner. These ‘halo’ effects suggest that, if subjective ratings of speaking are used for progress assessment, such features as Complexity, Accuracy and Fluency need to be rated in series, rather than in parallel, and raters’ attention needs to be drawn to the sort of indices of development highlighted in this study.
REFERENCES. • Alderson, J.C. (2000). Testing in EAP: Progress? Achievement? Proficiency? In G. Blue, J.Milton, and J. Saville (Eds.), Assessing English for Academic Purposes Berne: Peter Lang • Botel, M., Dawkins, J., & Granowski, A. (1973). A syntactic complexity formula. In W.MacGinitie (Ed.) Assessment Problems in Reading. Newark DED: International Reading Association. • Ellis, N.(2002). Frequency effects in language processing: A review with implications for theories of implicit and explicit language acquisition. Studies in Second Language Acquisition, 24(2), 143-188. • Foster, P., Tonkyn, A. and Wigglesworth, G. (2000). Measuring spoken Language: A unit for all reasons. Applied Linguistics, 21(3), 354-375. • Garman, M. (1990)Psycholinguistics. Cambridge: Cambridge University Press. • Johnson, K. (1996).Language Teaching and Skill Learning. Oxford: Blackwell. • Klein, W. (1986). Second Language Acquisition. Cambridge: Cambridge University Press . • Schmidt, R. 2001. Attention. In P.Robinson (Ed.).Cognition and Second Language Instruction. New York: Cambridge University Press.
