complexity

Measuring textual complexity in student writing Mathias Schulze Waterloo Centre for German Studies University of Waterloo http://www.wcgs.ca/~mschulze 1

outline • What do we mean by ‘complexity’? • Why would you want to measure it? • How would you measure it? • Does this work? • What did not work?

2

Text Text

Text

Text

Text Text

Text

Text

• proficiency ↔ complexity, accuracy, fluency − fluency captures how much text you produce in a certain time interval (and that you don’t pause too often) − accuracy captures how well your text conforms to our experience with other texts − complexity captures how predictable your next text stage is for others 3

CAF? • to analyze task outcomes • to measure indicators of proficiency at time t • to plot proficiency development over time • to model individual, complex, dynamic systems of language learning 4

complexity •The more different elements per unit of analysis we find, the more complex the text is. •The less information element n carries about element n+1, the more complex the text is. •textual complexity ↔ lexical complexity, syntactic complexity ⇨ two measures for each 5

lexical complexity: mean word length • “silent” characters and character combinations: malen mahlen, Mine – Miene; • inflection: Tisch – Tischen; schön - schöneres • derivation: Tisch – Tischchen, Professor – Professorin • compounding: Tischbein, Professorentitel Equation 1: Mean word length

v1 =

l w

with 1 ≤ v1 ≤ l 6

lexical complexity: type-token ratio Ich habe einen Bruder.

Ich habe eine Schwester.

Ich habe eine Schwester.

Sie ist noch klein.

Ich habe eine Mutter.

Sie heißt Petra.

7/12 → 1.43

10/11 → 2.13

Carroll’s type token ratio

v = 2

t 2w

with

1 ≤v ≤ 2 2w

w 2

7

syntactic complexity: mean comma unit length • longer clauses are more complex than shorter ones Mean comma unit length

w v3 = c

with 1 ≤ v3 ≤ w

8

syntactic complexity: comma units per period unit Ich kenne die Frau.

Ich kenne die Frau, die du mir beschrieben hast.

Mit viel Arbeit werden wir es schaffen.

Mit viel Arbeit, ein wenig Glück und deiner Hilfe werden wir es schaffen.

Peter kam zu spät.

Peter, der Kapitän der Mannschaft, kam zu spät.

Mean period unit length

c v4 = p

with 1 ≤ v4 ≤ c 9

balanced complexity Equation 1: Rectilinear distance

AB = |xa − xb | +| y a − yb | 1

Equation 1: Raw complexity CR

⏐ CR = V −Vmin 1 = |v1 −1|+⏐v2 − ⏐

1 ⏐ ⏐+|v3 −1|+|v4 −1| (2w)⏐

Equation 1: Balanced complexity CB

CB = C R − (max(v1 , v2 , v3 , v4 ) − min(v1, v2 , v3 , v4 ) ) ⎛ ⏐ CB = ⎜ |v1 − 1|+⏐v 2 − ⎜ ⏐ ⎝

⎞ 1 ⏐ ⏐+ |v3 − 1|+ |v4 − 1|⎟ − (max(v1, v2 , v3 , v4 ) − min(v1, v2 , v3 , v4 ) ) ⎟ (2w )⏐ ⎠ 10

GER 101

GER 102

GER 201

Student(s)

A

B

both

C

D

both

E

F

both

Balanced Complexity

7.6

7.0

7.3

8.1

8.1

8.1

10.1

9.6

9.8

Raw Complexity

12.4

11.1

11.7

13.5

13.8

13.7

15.8

15.3

15.6

11

problems • further tests are necessary − 3 terms x 3 courses, 316 students, 3143 texts

• lexical and syntactic complexity are not yet balanced: lexical: mean length and ratio; syntactic 2 x mean length 12

unique bigram ratio and mean period unit length Unique bigram ratio

u v = 3 2w

with

u 1 ≤v ≤ 3 2 2w

Mean period unit length

w v4 = p

with 1 ≤ v 4 ≤ w 13

14

balanced complexity • is a complex, conglomerate variable situated in a fourdimensional vector space • is based on four previously validated, simple text surface indicators • is the simple addition of the distances covered by the learner along the paths of mean word length, typetoken ratio, mean period unit length, and bigram ratio • is balanced through a penalty for skewedness • is balanced along the lexicon-grammar continuum 15

Bibliography (selection) Attali, Y., & Burstein, J. (2006). Automated Essay Scoring with e-rater® V.2. Journal of Technology, Learning, and Assessment, 4(3). Retrieved from http://escholarship.bc.edu/jtla/vol4/3/ Birjandi, P., & Ahangari, S. (2008). Effects of Task Repetition on the Fluency, Complexity and Accuracy of Iranian EFL Learners' Oral Discourse Asian EFL Journal, 10(3), 28-52. Burstein, J., & Chodorow, M. (1999). Automatic Essay Scoring for Non-native English Speakers. In M. B. Olsen (Ed.), Computer Mediated Language Assessment and Evaluation in Natural Language Processing. Proceedings of a Symposium Sponsored by the Association for Computational Linguistics and the International Association of Language Learning Technologies (pp. 68-75). College Park, Maryland: Association for Computational Linguistics. Burstein, J., & Marcu, D. (2000). Benefits of Modularity in an Automated Scoring System. Paper presented at the Proceedings of the Workshop on Using Toolsets and Architectures to Build NLP Systems. 18th International Conference on Computational Linguistics, Luxembourg. Chodorow, M., & Burstein, J. (2004). Beyond Essay Length: Evaluating erater's Performance on TOEFL Essays (TOEFL Research Rep. No. RR-73, ETS RR-04-04). Princeton, NJ: Educational Testing Services. Crookes, G. (1990). The Utterance, and Other Basic Units for Second Language Discourse Analysis. Applied Linguistics, 11(2), 183-199. Ellis, R., & Yuan, F. (2004). The Effects of Planning on Fluency, Complexity, and Accuracy in Second Language Narrative Writing. Studies in Second Language Acquisition, 26(1), 59-84. Foster, P., & Skehan, P. (1996). The Influence of Planning and Task Type on Second Language Performance. Studies in Second Language Acquisition, 18(3), 299-323. Foster, P., & Skehan, P. (1999). The Influence of Source of Planning and Focus on Task-Based Performance. Language Teaching Research, 3(3), 215-247. Foster, P., Tonkyn, A., & Wigglesworth, G. (2000). Measuring Spoken Language: A Unit for All Reasons. Applied Linguistics, 21, 354-375. Kaplan, R. M., Wolff, S. E., Burstein, J. C., Lu, C., Rock, D. A., & Kaplan, B. A. (1998). Scoring Essays Automatically Using Surface Features (GRE Board Professional Rep. No 94-21P; ETS RR-98-39). Princeton, NJ: ETS.

Kuiken, F., Mos, M., & Vedder, I. (2005). Cognitive Task Complexity and Second Language Writing Performance. Eurosla Yearbook (Vol. 5, pp. 195-222). Larsen-Freeman, D. (2006). The Emergence of Complexity, Fluency, and Accuracy in the Oral and Written Production of Five Chinese Learners of English. Applied Linguistics, 2006, 27, 4, Dec, 27(4), 590-619. Leal, D. J. (2005). The Word Writing CAFE: Assessing Student Writing for Complexity, Accuracy, and Fluency. Reading Teacher, 59(4), 340-350. Li, Y. (2000). Linguistic Characteristics of ESL Writing in Task-Based E-Mail Activities. System, 28(2), 229-245. Michel, M. C., Kuiken, F., & Vedder, I. (2007). The Influence of Complexity in Monologic versus Dialogic Tasks in Dutch L2. , 45(3), 59. IRAL, 45(3), 241-259. Ortega, L. (2003). Syntactic Complexity Measures and Their Relationship to L2 Proficiency: A Research Synthesis of College-level L2 Writing. Applied Linguistics, 24(4), 492-518. Shermis, M. D., & Burstein, J. (2003). Automated Essay Scoring: A Crossdisciplinary Perspective. Mahwah, NJ; London: Lawrence Erlbaum Associates. Skehan, P., & Foster, P. (1997). Task Type and Task Processing Conditions as Influences on Foreign Language Performance. Language Teaching Research, 1(3), 185-211. Skehan, P., & Foster, P. (1999). The influence of task structure and processing conditions on narrative retellings. Language Learning, 49(1), 93-120. Skehan, P., & Foster, P. (2005). Strategic and On-line Planning: The Influence of Surprise information and Task Time on Second Language Performance. In R. Ellis (Ed.), Planning and task performance in a second language (pp. 193-216). Philadelphia: John Benjamins Pub. Co. Sotillo, S. M. (2000). Discourse Functions and Syntactic Complexity in Synchronous and Asynchronous Communication. Language Learning & Technology: A Refereed Journal for Second and Foreign Language Educators, 4(2000), 82-119. Tavakoli, P., & Skehan, P. (2005). Strategic Planning, Task Structure, and Performance Testing. In R. Ellis (Ed.), Planning and Task Performance in a Second Language (pp. 239-273). Amsterdam, Netherlands: John Benjamins Publishing Company. Wolfe-Quintero, K., Inagaki, S., & Kim, H.-Y. (1998). Second Language Development in Writing: Measures of Fluency, Accuracy, and Complexity. Honolulu, HI: Second Language Teaching & Curriculum Center, University of Hawaii at Manoa.

16