Dynamic development of lexical sophistication

Dynamic development of lexical sophistication through a series of academic tasks: A semester-long study Masaki Eguchi Waseda University University of Hawaii at Manoa

The Seventh International Conference on Task-Based Language Teaching @ University of Barcelona 04/19/2017

Agenda 1) Literature review

A. Lexical Richness and Lexical sophistication B. Multidimensional lexical sophistication C. Usage-based lexical studies

2) Methodology 3) Results 4) Discussion 5) Conclusion

3

Literature Review 1

EMI and linguistic outcome English-Medium Instruction (EMI) • “Non-language subject taught through English” (Hellekjær, 2010, p.1) • A series of unfocused tasks (Ellis, 2003) • Language learning = “by product” (Taguchi, 2014a) Research in EMI settings q Pragmatic development q Academic literacy q Attitude, self-perception and L2 speech (Mahboob, 2014; Pessoa & Miller, 2014; Taguchi, 2014b; Suzuki et al., 2017)

Vocabulary retrieval : Most frustrating aspect for students in EMI in Japan (Suzuki et al., 2017)

4

Literature Review 2

Lexical Richness

(Daller et al., 2003; Lu, 2012; Skehan, 2009ab; see also Read, 2000)

What is rich vocabulary use?? Text-internal measures Diversity: the variety of words

Text-external measures Sophistication: the quality of words

# of different words

Corpus

(COCA,BNC etc.)

# of total words Oral/Written

Oral/Written

The two approaches seem to be independent constructs (Skehan, 2009a).

1000 75% 2000: 15% 3000: 7% Off: 3%

5

Literature Review 3

Multidimensional Lexical Sophistication Li and Lorenzo-Dus (2014): Conceptualization of “Lexical sophistication” • 25 raters commented on “sophistication” Búlte & Housen (2012) : Ø Sophistication extensively relies on

Frequency

e.g., Lexical Frequency Profile (Laufer & Nation, 1995), Lambda (Meara & Bell, 2001)

Need for investigating multidimensional lexical sophistication

Word length Technical vocabulary

Lexical sophistication

Colloquial words

Abstract concept

Descriptive vocabulary

6

Literature Review 4

Lexical Sophistication Revised • Multidimensional Lexical Sophistication has been shown to capture learners’ proficiency (Kyle & Crossley, 2015, 2016). Frequency

Range (dispersion)

Psycholinguistic property

•How infrequent a word is. •(Crossley, Cobb & McNamara, 2013)

Proportion of Multiword Units (Ngrams)

•How much a learner use Ngrams (contiguous words) in production •(Pawley & Syder, 1983; Wray, 2002)

•“How widely a word is used” •Indication of domain specific vocabulary •(Kyle & Crossley, 2015, p.4)

Association Strength of Multiword Units (Ngrams)

•How MWUs co-occur together in large corpora. •(Bestgen & Granger, 2014; Brezina et al., 2015)

•Concreteness, familiarity etc. •Crossley et al., (2011)

Among others (see Kyle & Crossley, 2015)… • Sense relations (hypernymy, polysemy) • Contextual distinctiveness

7

Literature Review 5

Usage-based, dynamic lexical development Usage-based model of L2 acquisition: (Ellis, 2002; Ellis & Cadierno, 2009; Ellis & Larsen-Freeman, 2006)

Study

Settings

Mode/Genre

Constructs

Findings

Crossley , ESL Kyle, Salsbury (1 yr) (2016)

Oral/ casual conversation in lab

Psycholinguistic Properties (e.g., Concreteness, Familiarity)

Less concrete, less familiar words produced (time and proficiency)

Bestgen & Granger (2014)

ESL (Semester )

Written

Ngram association strength (e.g., T score and MI)

• Decrease the overreliance of highly frequent Bigrams • Did not develop the use of infrequent Bigrams

Zheng (2016)

Intensive EFL (10 mths)

Written

Diversity Frequency-based Sophistication Lexical bundles

Dynamic development • Increase in D and Soph • U-shaped: Lexical bundles

Hou, Loerts & EFL Written No study in(18 mths: Few studies on Verspoor CBI classroom (2016) pre/post) oral mode

Range of chunk measures • Few study ratio measured multifaceted • Chunk lexical sophistication • Lexical/grammatical • chunks

Lexical chunks strongly correlated with holistic Significant improvement of lexical chunk and

8

Literature Review 6

Research Question How do learners develop their multidimensional lexical sophistication throughout the in-class opinion exchange/reasoning tasks in EMI, in terms of: a) b) c) d) e)

Frequency Range Concreteness Ngram proportion Ngram strength

9

Methodology 1

Participants • Three students (juniors) in an EMI course in a private university in Tokyo. • “Immersion (bilingual) education” (13 week * 2 hours = 26-28 hours/ semester) Gen TOEFL # of EMI der ITP taken previously

# of words/w (SD)

VLT 30

A F

530

6 (2)

46 (21)

113

B F

530

7 (2)

147.75 (83.47)

95

10

C M

570

8 (5)

349.18 (130.08) 112

**( ) shows the number of student-centered EMI

28 23

25 20

Vocabulary Levels Test

29 29 30

26 25

29 30 29 21

14

15

2

5

3

2

0

A

2000

B

3000

AWL

C

5000

10000

**98% of the token in textbook were within 6-7000 level

Learner A gave few opinions in the small group discussion (< 100 words/week). => Unable to process the text (due to reliability of the measures)

10

Methodology 2

Tasks in the target EMI

Pre

Beginning

Classroom

• Assigned readings (English: 10 – 15 pages / weekly)

• Weekly written quizzes (2 definitions & 1 discussion) • Two student presentations • Small group discussions (3-12 Qs) • Informal lecture by the instructor

Data collection site: 4 groups of 4 – 5 students One IC recorder in each group

11

Methodology 3

Data collection & handling • Audio recording of the in-class small discussions – 13 weeks of instruction (1 class canceled and combined).

• Transcribed by the researcher, and exclusion based on… – Verbatim responses by the participants – Repetitions of the other speakers – Direct quotation from the textbooks

• Discussion questions were coded by the researcher – Based on cognitive processes involved (Ellis, 2003) – Also allowing for open-coding to maximize the flexibility

12

Methodology 4

Types of discussion tasks in the current EMI Question types Opinion exchange (Ellis, 2003) Explaining/reasoning (Ellis, 2003) Definition of a term/ what is XX? Review questions Factual question based on the reading Causal connection/ inference by the data Compare/Contrast Role play Experiences/ preference? Past decision Explaining concepts with example Impression about the particular part of reading Impression about a model lesson Unidentifiable Total

# of Qs 34 22 16 4 3 1 1 1 1 1 1 1 4 90

Since these two tasks can elicit extended turn by the learners, production from these two tasks were combined and analyzed. => Constraints on text lengths

13

Methodology 5

Indices of Lexical Sophistication • Tool for Automatic Analysis of Lexical Sophistication 2.4 (TAALES, Kyle & Crossley, 2015) – Corpus of Contemporary American (COCA) Academic/ Spoken (Davies, 2009)

Frequency CW

• # of occurrence in Ref corpus

Ngram Proportion

•# of Ngram found in corpus/ total Ngrams •Similar to Chunk Ratio (Hou et al., 2016; )

Range CW

• # of texts that contain words in Ref corpus

Ngram Association strength

•MI2 (Evert, 2005) •(see also Bestgen & Granger, 2014, Li & Schmitt, 2010)

Concreteness CW

• How concrete mental image of a word is • (Brysbaert, 2014)

Lemmatized (run, runs, and ran gets the same score) Type counts = the same word counted only once in the same text (=> Controlled for the repetitions) çè Lexical diversity

14

Methodology 6

Indices of Lexical Sophistication| Range High Range score : General vocab know

70

Beg.

TAALES calculates

Int.

Adv.

policy term recruit formative recess

36 35 9 2 1

Low Range score: Domain Specific

the mean range score of the words for each text. e.g., I know about the policy. know => 70 policy => 36

Range score: 70 + 36 / 2 = 53

15

Methodology 7

Indices of Lexical Sophistication| Concreteness High concreteness score: Concrete

Beg.

Int.

Adv.

child graph

581 553

context impression definition normal responsibility

314 288 262 237 222

Low concreteness score: Abstract

TAALES calculates

the mean concreteness score of the words for each text.

16

Methodology 8

Indices of Lexical Sophistication| Association Strength High score: Strong association Adv.

Int.

academic achievement the kind of more emphasis on

13.73 11.25 10.01

but if they to deepen the the purpose be

6.80 5.30 4.79

Beg.

Low score: Weak association

TAALES calculates

the mean Ngram Association strength score of the words for each text.

17

Methodology 9

Analysis | Dynamic System Theory • Describing individual trajectories (= development in context) – Individual variability: “intrinsic property of the developmental process” (Verspoor et al., 2011). – “Flux and variability leading to stability signal self-organization and emergence” (LarsenFreeman, 2012, p.80; Larsen-Freeman & Cameron, 2008; Verspoor et al., 2011)

• Moving Min-Max graph (Verspoor et al., 2011; see example for Polat & Kim, 2013; Zheng, 2016) 2) Maximum among the 5 adjacent observations

10000

Increase in Maximum, but large variability ↓ Restructuring of the system

9000 8000 7000 6000

1) Observed data

5000 4000 3000 2000

3) Minimum among the 5 adjacent observations

1000 0

1

2

3

4

5 Data

6

7 Max2

8

9 Min2

10

11

12

13

Results

19

Results 1

Results | Overview A

B

C

Academic

Spoken

Academic

Spoken

Frequency

Fluctuate

→

Range

Shift to more general vocabulary

Slight trends to general vocab

Concreteness

→

→

Bigrams prop

↗

↗

→

→

Trigram prop

→

↗

→

→

Bigram Strength

→

→

→

↗

↗

↗

Trigram Strength

Fluctuate -> stable

** Excluding A from the analysis does not mean A’s development is invalid.

20

Results 2 | Range

Range ü B’s range shows a shift to more general words at the end. ü C’s range shows slight trends to general vocabulary.

B and C: Spoken Range CW

B and C: Academic Range CW

Ad v.

100

90 80 70

B

60

C

50

maxB

40

minB

30

MaxC

20 10

MinC

Discipline Specific

0 1

2

3

4

5

6

7

Time

MICASE 8

9

10

11

12

13

Percentage of texts in corpus (%)

Int.

Widely used

100

Percentage of texts in corpus (%)

Beg .

90 80 70

B

60

C

50

maxB

40

minB

30

MaxC

20

MinC

10

MICASE

0 1

2

3

4

5

6

7

Time

8

9

10

11

12

13

21

Results 3 | Bigram Proportion

Bigram Proportion ü B produces more bigrams towards the end (both academic and spoken). ü C produces relatively stable amount of bigrams. B: Spoken Bigram Proportion

Beg.

100

100

90

90

80

80

70

70

60

Data

50

Max

40

Min

30

MICASE

20 10 0

Proportion (%)

Int.

Proportion (%)

Adv

C_COCA_lemma_spoken_bi_prop_90k_TP

60

Data

50

Max2

40

Min2

30

MICASE

20 10

1

2

3

4

5

6

7

Time

8

9

10

11

12

13

0

1

2

3

4

5

6

7

Time

8

9

10

11

12

13

22

Results 4 | Trigram Proportion

Trigram Proportion ü B produces more Spoken Trigrams towards the end.. C: Spoken Trigram Proportion

Int. Beg.

100

100

90

90

80

80

70

70

60

Data

50

Max

40

Min

30

MICASE

20 10 0

Proportion (%)

Adv

Proportion (%)

B: Spoken Trigram Proportion

60

Data

50

Max2

40

Min2

30

MICASE

20 10

1

2

3

4

5

6

7

Time

8

9

10

11

12

13

0

1

2

3

4

5

6

7

Time

8

9

10

11

12

13

23

Results 5 | Trigram Association

Trigram Association Strength ü B and C experienced high variability ü C’s Trigram approaching to target like. Increase in upper boundary B and C: Academic Trigram Strength (MI2) 10

8.5

B

8

C

7.5

maxB

7

minB

6.5

Beg.

MaxC

6

Variability

5.5

MinC MICASE

5 1

2

3

4

5

6

7

Time

8

9

10

11

12

13

9.5

MI2 (Degree of associasion)

9

MI2

Int.

10

Approach more target like use

9.5

Adv

B and C: Spoken Trigram Strength (MI2)

9 8.5

B

8

C

7.5

More Stable

7 6.5

maxB minB MaxC

6

Variability

5.5

MinC MICASE

5 1

2

3

4

5

6

7

Time

8

9

10

11

12

13

24

Results 6 | Summary

Summary of Results C’s production 1) Stable production of less frequent 2) Ngram proportion: same 3) Ngram Ass Str (Spoken) increase

B’s production 1) General vocabulary 2) Ngram proportion increase 3) Ngram Ass Str more stable

A

B

C

Academic

Spoken

Academic

Spoken

Frequency

Fluctuate

→

Range

Shift to more general vocabulary

Slight trends to general vocab

Concreteness

→

→

Bigrams prop

↗

↗

→

→

Trigram prop

→

↗

→

→

Bigram Strength

→

→

→

↗

↗

↗

Trigram Strength

Fluctuate -> stable

25

Discussion 1

Discussion | Overview B’s production 1) General vocabulary 2) Ngram proportion increase 3) Ngram Ass Str more stable

C’s production 1) Stable production of less frequent 2) Ngram proportion: same 3) Ngram Ass Str (Spoken) increase

A. Individual trajectories 1) Development of Ngram proportion 2) Development of Ngram Association strength

B. The shift to general vocabulary use

26

Discussion 2

A) Individual trajectories of Ngrams TOEF L ITP B 530

VLT (5000 level)

# of past EMI

Lecture -based EMI

Studentcentered EMI

Amount of production in EMI

Ngram Proportion

Ngram Strength (MI2)

95 (14)

7

5

2

147.75 (83.47)

Increase (510%)

Less variability

C 570

112 8 3 5 349.18 (130.08) No change Increase (21) Usage-based acquisition of construction (Ellis, 2002) B’s development in Ngram proportion • increase the amount of “Formula” Low-scope Formula Construction • B learned conventionalized units of utterance patterns (Hou et al., 2016; Verspoor et al., 2012)

C’s development in Ngram Association Strength • increase strongly associated collocations (MI2) • ó Bestgen & Granger (2014) => C’s prior proficiency & Active participation in EMI

27

Discussion 3

A) Individual trajectories of Ngrams TOEF L ITP

VLT (5000 level)

# of past EMI

Lecture -based EMI

Studentcentered EMI

Amount of production in EMI

Ngram Proportion

Ngram Strength (MI2)

B 530

95 (14)

7

5

2

147.75 (83.47)

Increase (510%)

Less variability

C 570

112 (21)

8

3

5

349.18 (130.08)

No change

Increase

Teacher centered and student centered EMI 1) Initial proficiency level & receptive vocab

(Assigned Reading)

2) The type of EMI (exposure)

Lecture

Assigned Reading Student Presentation Discussion

Exam

3) The amount of production in the EMI classroom => Different usage experiences

Exam/ Term papers

Interactions between learning environment and individual trajectories can be dynamic.

28

Discussion 4

B) The shift to general vocabulary use ASSUMPTION: academic discourse = domain specific vocabulary (see Kyle & Crossley, 2015). FINDING: a slight shift to general vocabulary 1) Task types: Opinion exchange/ reasoning (interactive) – Decision making (interactive) => frequent words (Skehan, 2009ab, 2014)

2) Teacher’s role: encouragement for negotiation (e.g., confirmation checks & clarification requests etc.) – => strategic use of general vocabulary use enhanced

Type of vocabulary use required by the immediate task needs (see also Polat & Kim, 2014 for diversity measure)

29

Conclusion

Conclusion üThe unfocused academic tasks provided opportunity for the two learners to develop multifaceted lexical sophistication. 1) Relationships between individual profiles and development: ØInitial proficiency / Previous experiences in EMI ØThe amount of production in EMI

2) The interactive tasks and teacher’s encouragement for negotiation ØImmediate communicative needs => more general words towards the end of the semester.

30

Limitations

Limitations [Measurement] üWe need measures to see the student A’s trajectory, who produced less amount of production.

[Analysis] üItem analyses of the Ngrams the learner produced in each session üInferential statistics about the individual variability (e.g., Monte Carlo analysis)

[Generalizability] üThe results should be interpreted with caution because of the nature of the study (exploratory case study). To study the

generalizable pattern of development, further studies should investigate the learners with different profiles (Larsen-Freeman, 2012)

31

Further research

Further research [For the current project] ØTriangulation with interview data, learning logs, quasi-experimental speech data, as well as analysis of discourse of the classroom discussions

[Future projects] üHow different design and implementation of tasks influences: ü Learners’ usage-experiences in EMI (frequency, association, perceptual learning, interaction among the factors). ü Development of multidimensional lexical sophistication (potential interactions)

üMore longitudinal, dense learner corpora that document various aspects of learner characteristics (e.g., previous experiences, moment-by-moment experience in EMI etc.) to reveal a richer picture of how learning happens in EMI (see also Ellis et al., 2016).

Acknowledgements Dr. Tetsuo Harada (Waseda University) Dr. Kristopher Kyle (University of Hawaii at Manoa) Dr. Nicole Ziegler (University of Hawaii at Manoa) Shuhei Kudo and Ryo Moriya (Waseda University) for supporting the data collection

33

References • Bestgen, Y., & Granger, S. (2014). Quantifying the development of phraseological competence in L2 English writing: An automated approach. Journal of Second Language Writing, 26, 28–41. https://doi.org/10.1016/j.jslw.2014.09.004 • Brezina, V., Mcenery, T., & Wattam, S. (2015). Collocations in context A new perspective on collocation networks*. International Journal of Corpus Linguistics, 202(2015), 139–173. https://doi.org/10.1075/ijcl.20.2.01bre • Bulté, B., & Housen, A. (2012). Defining and operationalising L2 complexity. Dimensions of L2 Performance and Proficiency: Complexity; Accuracy and Fluency in SLA, 21–46. https://doi.org/10.1075/lllt.32.02bul • Chang, J.-Y., Kim, W., & Lee, H. (2015). A language support program for English-medium instruction courses: Its development and evaluation in an EFL setting. International Journal of Bilingual Education and Bilingualism, 50(September 2015), 1–19. https://doi.org/10.1080/13670050.2015.1080658 • Coyle, D. (2007). Content and language integrated learning: Towards a connected research agenda for CLIL pedagogies. International Journal of Bilingual Education and Bilingualism, 10(5), 543–562. https://doi.org/10.2167/beb459.0

34

References • Crossley, S. A., Cobb, T., & McNamara, D. S. (2013). Comparing count-based and bandbased indices of word frequency: Implications for active vocabulary research and pedagogical applications. System, 41(4), 965–981. https://doi.org/10.1016/j.system.2013.08.002 • Crossley, S. A., Salsbury, T., & Kyle, K. (2016). A Usage-Based Investigation of L2 Lexical Acquisition: The Role of Input and Output. The Modern Language Journal. https://doi.org/10.1111/modl.12344 • Crossley, S. A., Salsbury, T., Mcnamara, D. S., & Jarvis, S. (2011). Predicting lexical proficiency in language learner texts using computational indices. Language Testing, 28(4), 561–580. https://doi.org/10.1177/0265532210378031 • Daller, H., Van Hout, R., & Treffers-Daller, J. (2003). Lexical Richness in the Spontaneous Speech of Bilinguals. Applied Linguistics, 24(2), 197–222+267. https://doi.org/10.1093/applin/24.2.197 • Doiz, A., Lasagabaster, D., & Sierra, J. M. (2013). English-Medium Instruction at Universities: Global Challenges. Bristol, UK: Multilingual Matters. https://doi.org/10.1093/elt/cct045

35

References • Ellis, N. C. (2016). Cognition, Corpora, and Computing: Triangulating Research in Usage-Based Language Learning. Language Learning, 1–26. https://doi.org/10.1111/lang.12215\ • Ellis, R. (2003). Task-based language learning and teaching. Oxford University Press. https://doi.org/10.1017/S0272263104293056 • Evert, S. (2005). The Statistics of Word Cooccurrences Word Pairs and Collocations. Unpublished Doctoral Dissertation Institut Fur Maschinelle Sprachverarbeitung Universitat Stuttgart, 98(August 2004), 353. https://doi.org/10.1073/pnas.141413598 • Harada, T. (2017). Developing a content-based English as a foreign language program: Needs analysis and curriculum design at the university level. In M. A. Snow & D. M. Brinton (Eds.), The content-based classroom. Ann Arbor, MI: University of Michigan Press. • Hellekjaer, G. O. (2010). Lecture comprehension in English-medium higher education. Hermes, 45(45), 11–34. • Kyle, K., & Crossley, S. A. (2015). Automatically Assessing Lexical Sophistication: Indices, Tools, Findings, and Application. TESOL Quarterly, 49(4), 757–786. https://doi.org/10.1002/tesq.194 • Li, H., & Lorenzo-Dus, N. (2014). Investigating how vocabulary is assessed in a narrative task through raters’ verbal protocols. System, 46(1), 1–13. https://doi.org/10.1016/j.system.2014.06.006

36

References • Lu, X. (2012). The Relationship of Lexical Richness to the Quality of ESL Learners’ Oral Narratives. Modern Language Journal, 96(2), 190–208. https://doi.org/10.1111/j.15404781.2011.01232_1.x • Pawley, A., & Syder, F. H. (1983). Two puzzles for linguistic theory: Nativelike selection and nativelike fluency. Language and Communication, 191, 225. • Polat, B., & Kim, Y. (2014). Dynamics of complexity and accuracy: A longitudinal case study of advanced untutored development. Applied Linguistics, 35(2), 184–207. https://doi.org/10.1093/applin/amt013 • Read, J. A. S. (2000). Assessing vocabulary. Cambridge : Cambridge . • Skehan, P. (2009a). Lexical performance by native and non-native speakers on languagelearning tasks. Vocabulary Studies in First and Second Language Acquisition: The Interface between Theory and Application, 107–124. https://doi.org/10.1057/9780230242258 • Skehan, P. (2009b). Modelling second language performance: Integrating complexity, accuracy, fluency, and lexis. Applied Linguistics, 30(4), 510–532. https://doi.org/10.1093/applin/amp047

37

References • Taguchi, N. (2014a). English-medium education in the global society. International Review of Applied Linguistics in Language Teaching, 52(2), 89–98. https://doi.org/10.1515/iral-2014-0004 • Taguchi, N. (2014b). Pragmatic socialization in an English-medium university in Japan. IRAL - International Review of Applied Linguistics in Language Teaching, 52(2), 157–181. https://doi.org/10.1515/iral-2014-0007 • Zheng, Y. (2016). The complex, dynamic development of L2 lexical use: A longitudinal study on Chinese learners of English. System, 56(220), 40–53. https://doi.org/10.1016/j.system.2015.11.007 • Verspoor, M., De Bot, K., & Lowie, W. (2011). A dynamic approach to second language development: Methods and techniques (Vol. 29). John Benjamins Publishing. • Verspoor, M., & van Dijk, M. (2013). Variability in a Dynamic Systems Theory Approach to Second Language acquisition. The Encyclopedia of Applied Linguistics, 1–9. https://doi.org/10.1002/9781405198431 • Wray, A. (2002). Formulaic language and the lexicon. Cambridge, U.K. : Cambridge, U.K. . • Zheng, Y. (2016). The complex, dynamic development of L2 lexical use: A longitudinal study on Chinese learners of English. System, 56(220), 40–53. https://doi.org/10.1016/j.system.2015.11.007

Mahalo! Thank you for your attention!! TBLT 2017 Barcelona