methodological constraints of metadiscourse and inter

TOWARDS CRITERION VALIDITY IN CLASSROOM L A N G U A G E ANALYSIS: METHODOLOGICAL CONSTRAINTS O F METADISCOURSE AND I N T E R - R A T E R A G R E E M E N T 1

Douglas A l t a m i r o C O N S O L O

2

• ABSTRACT: This paper reports on a process to validate a revised version of a system for coding classroom discourse i n foreign language lessons, a context i n w h i c h the dual role of language (as content and means of communication) and the speakers' specific pedagogical aims lead to a certain degree of ambiguity i n language analysis. The language used by teachers and students has been extensively studied, and a framework of concepts concerning classroom discourse well-established. Models for coding classroom language need, however, to be revised w h e n they are applied to specific research contexts. The application and revision of an initial framework can lead to the development of earlier models, and to the re-definition of previously established categories of analysis that have to be validated. The procedures followed to validate a coding system are related here as guidelines for conducting research under similar circumstances. The advantages of using instruments that incorporate t w o types of data, that is, quantitative measures and qualitative information from raters' metadiscourse, are discussed, and i t is suggested that such procedure can contribute to the process of validation itself, towards attaining reliability of research results, as well as indicate some constraints of the adopted research methodology. • KEYWORDS: Discourse analysis; classroom discourse; metadiscourse; validation; reliability.

1

A n earlier v e r s i o n of t h i s p a p e r w a s p r e s e n t e d i n t h e poster s e s s i o n at t h e X I A I L A W o r l d C o n gress, Jyvâskylà, F i n l a n d , 4-9 A u g u s t 1996.

2

D e p a r t a m e n t o de L e t r a s M o d e r n a s - I n s t i t u t o de Biociências, L e t r a s e Ciências E x a t a s - U N E S P - 15054-000 - São José d o Rio Preto - SE E - m a i l : d c o n s o l o @ I e m . i b i l c e . u n e s p . b r .

Alfa, São Paulo, 43: 113-134, 1999

113

Categorization of discourse a c t s in classroom interaction T h i s p a p e r r e p o r t s o n a p r o c e s s to v a l i d a t e a s y s t e m for c o d i n g c l a s ­ s r o o m d i s c o u r s e i n f o r e i g n l a n g u a g e lessons. T h e c a t e g o r i e s o f a n a l y s i s for t e a c h e r t a l k a n d s t u d e n t s p e e c h p r o p o s e d for v a l i d a t i o n i n t h i s s t u d y w e r e a d a p t e d f r o m a r a n g e of m o d e l s a n d s t u d i e s o n c l a s s r o o m d i s ­ 3

course, m a i n l y from the categories devised i n earlier models to classify a n d describe different c o m m u n i c a t i v e aims a n d pedagogical purposes of t e a c h e r t a l k a n d s t u d e n t s p e e c h . T h e t y p o l o g y o f d i s c o u r s e c a t e g o r i e s d e a l t w i t h i n t h i s p a p e r i s b a s e d o n t h e w o r k o f C o n s o l o (1996), a n e x t e n ­ ded a n d revised version of a m o d e l originally proposed by Sinclair & C o u l t h a r d (1975, 1992) for a n a l y s i n g c l a s s r o o m i n t e r a c t i o n . T h e u n i t s o f a n a l y s i s a t t h e l o w e s t r a n k o f d i s c o u r s e are c a l l e d " c o m ­ m u n i c a t i v e acts". B e y o n d t h e i r f o r m a l s y n t a c t i c a n d p h o n o l o g i c a l p r o ­ p e r t i e s , a c t s are l i n g u i s t i c i t e m s at t h e l e v e l o f d i s c o u r s e , t h a t is, t h e y are a n a l y s e d a c c o r d i n g t o t h e f u n c t i o n a l p r o p e r t i e s w i t h w h a t s p e a k e r s u s e a c t s for. O n e a s s u m p t i o n i n t h i s c a t e g o r i z a t i o n o f d i s c o u r s e e l e m e n t s is t h a t the discourse value of an act depends o n w h a t acts have preceded it, w h a t are e x p e c t e d t o follow, a n d t h e w a y a c t s r e l a t e t o e a c h other. S u c h s e q u e n t i a l r e l a t i o n s h i p is considered to analyse t h e discourse s t r u c t u r e b y m e a n s o f c a t e g o r i e s d e f i n e d n o t o n l y s t r u c t u r a l l y , b u t also for class­ r o o m - s p e c i f i c s i t u a t i o n s . T h e s o c i o l i n g u i s t i c c o n t e x t of f o r m a l EFL l e s ­ sons, i n c l u d i n g n o n - v e r b a l b e h a v i o u r a n d p a r a l i n g u i s t i c a s p e c t s of t h e e n v i r o n m e n t d e t e r m i n e t h e s t r u c t u r a l a n d f u n c t i o n a l c a t e g o r i e s i n class­ r o o m d i s c o u r s e ( h e n c e f o r t h CD). T h e r e l e v a n t " n o n - l i n g u i s t i c " f a c t o r s t o i n f l u e n c e t h e p r o d u c t i o n a n d a n a l y s i s of c l a s s r o o m l a n g u a g e -

the

social conventions of the environment and the shared experience of t h e s p e a k e r s - are t r e a t e d i n t h e S i n c l a i r & C o u l t h a r d ( h e n c e f o r t h S&C) m o d e l (1975) as " s i t u a t i o n " . S u c h f a c t o r s i m p l y a n a n a l y s i s o f l a n g u a g e a n d i n t e r a c t i o n i n EFL l e s s o n s b a s e d o n t h e a w a r e n e s s o f s o c i a l , p e d a ­ g o g i c a l , p s y c h o l o g i c a l a n d l i n g u i s t i c a s p e c t s of c l a s s r o o m i n t e r a c t i o n . Acts constitute the next higher rank of discourse categories, defi­ n e d as " m o v e s " . T h e s e c a t e g o r i s e t h e t y p i c a l i n t e r a c t i o n a l I n i t i a t i o n R e s p o n s e - F o l l o w u p (IRF) s t r u c t u r e o f lessons, w h i c h p r e d i c t s s t u d e n t m o v e s as r e s p o n s e s (R) t o t h e t e a c h e r ' s i n i t i a t i o n s (I). S t u d e n t s p e e c h is u s u a l l y e v a l u a t e d b y t h e teacher, i n f o l l o w - u p (F) m o v e s . A d i f f e r e n t i a -

Fot a d e t a i l e d r e p o r t o n t h e d e v e l o p m e n t of t h e c o d i n g s y s t e m d e a l t w i t h here, see C o n s o l o 1996 p.144-87 ( C h a p t e r 4)

114

Alfa, Säo Paulo, 43: 113-134, 1999

t i o n is m a d e i n t h e s y s t e m p r o p o s e d b y C o n s o l o (1996) b e t w e e n T T a n d s t u d e n t s p e e c h at t h e r a n k of m o v e s b y u s i n g t h e l a b e l s Is, Fs a n d Rt for i n i t i a t i o n s (Is) a n d f o l l o w - u p m o v e s (Fs) p r o d u c e d b y s t u d e n t s , a n d resp o n s e s g i v e n b y t h e t e a c h e r (Rt), i n o r d e r t o a n a l y s e t h e s p e c i f i c c o n t r i b u t i o n s of t e a c h e r s a n d s t u d e n t s t o CD. T h e i n t e r a c t i o n a l u n i t is d e f i n e d as a n e x c h a n g e , w h i c h is d e t e r m i n e d b y t h e o c c u r r e n c e of a n I or Is m o v e . E x c h a n g e s c o m b i n e t o f o r m t r a n s a c t i o n s . T h e i d e n t i f i c a t i o n of a t r a n s a c t i o n b o u n d a r y u s u a l l y c o i n cides w i t h t o p i c change, t h a t m a y be m a r k e d by b o u n d a r y moves such as F r a m e (Fr) a n d Focus (Fo). T r a n s a c t i o n s differ f r o m e x c h a n g e s i n t h a t w h i l e i t is p o s s i b l e t o e s t a b l i s h i n t e r n a l p a t t e r n s a n d a c a t e g o r i z a t i o n for exchanges,

4

l i t t l e c a n be s a i d ab | o u t t h e p a t t e r n s for t h e i n t e r n a l s t r u c -

t u r e o f t r a n s a c t i o n s . T r a n s a c t i o n s c o m b i n e t o f o r m lessons, t h e h i g h e s t u n i t o n t h e r a n k s c a l e for a n a l y s i n g CD. T h e a c t s w h i c h c a t e g o r i s e t h e f u n c t i o n s o f c l a s s r o o m l a n g u a g e , as p r o d u c e d b y t e a c h e r s a n d s t u d e n t s , are p r e s e n t e d b e l o w :

5

1. Maker { m r k }

15.5 Informative reply { i - r p l }

2. Starter { s t r }

15.£ i. Offer reply { o - r p l }

3. Elicitation { e l i }

16. Rect { r e a }

5. Confirmation check {cf-chk}

CO

4. Comprehension check { c p - c h k } 17. Protest { p r o } Correction { c o r }

6. Clarification { c l a }

19. Evaluate { e v l }

7. Directive { d i r }

20. Metastatement { m s t }

8. Informative { i n f }

21.

9. Comment { c o m }

22. Tank { t h k }

Apology { a p l }

10. Clue { c l u }

23. Encouragement { e c g }

11. Model { m d l }

24. Conclusion { c o n }

12. Bid { b i d }

25. Terminate { t e r }

13. Nomination { n o m }

26. Greeting { g r t }

14. Acnowledge { a c k }

27. Parting { p r t }

15.1. Affirmative reply { y - r p l }

28. Aside { a s i }

15.2. Negative reply { n - r p l }

29. Translation { t r l }

15.3. Choice reply { c - r p l }

30. Rhetorical aquestion { r t q }

15.4 Repetition reply { r p - r p l }

4

V a r i o u s p a t t e r n s , s u c h as IP, a n d I { R F } " , d e r i v e d from t h e bas.c I R F s t r u c t u r e are p r e s e n t e d in S i n c l a i r & C o u t h h a r d (1975) a n d C o n s o l o (1996).

5

For t h e d e f i n i t i o n s of t h e acts, see C o n s o l o , 1996, p.170-85.

Alfa, Sâo Paulo, 43: 113-134, 1999

325

T h e n e x t s e c t i o n reports o n a process to v a l i d a t e t h e revised v e r s i o n of t h e S&C m o d e l p r o p o s e d b y C o n s o l o (1996), f o c u s i n g a t t h e r a n k of t h e c o m m u n i c a t i v e acts l i s t e d above. T h e v a l i d a t i o n of these revised cate­ g o r i e s o f a n a l y s i s is e s s e n t i a l t o s u p p o r t t h e r e s u l t s a n d f u r t h e r i m p l i c a ­ t i o n s o f s u c h a study, s i n c e i t s a c h i e v e m e n t s w i l l h a v e b e e n

reached

t h r o u g h criteria other t h a n those prescribed by the original m o d e l .

Validation of categories of a n a l y s i s T h e v a l i d i t y a n d r e l i a b i l i t y of o b s e r v a t i o n a l p r o c e d u r e s , a n d o f c a t e ­ g o r i e s o f a n a l y s i s , are k e y i s s u e s i n r e s e a r c h . A m a j o r a i m i n c l a s s r o o m r e s e a r c h is, a c c o r d i n g t o C h a u d r o n (1988, p.23), to produce descriptions and interpretations of classroom events, and the rela­ tionships between them, that will be identified by others as real and meaning ful for teachers, learners, and learning. T h e a s s u m p t i o n a b o v e is f o l l o w e d i n t h i s s t u d y w i t h r e g a r d t o t h e description of h o w teachers a n d students engage i n v e r b a l i n t e r a c t i o n i n EFL lessons, a n d t o t h e e x t e n t w h i c h c l a s s r o o m o b s e r v a t i o n , r e c o r d e d lessons a n d transcripts c a n reveal the m e a n i n g s a n d goals of t h a t t y p e of i n t e r a c t i o n ( S t u b b s , 1983). V a l i d i t y a n d r e l i a b i l i t y i n d a t a a n a l y s i s are f o c u s e d o n a n d i n v e s t i ­ g a t e d here for a n u m b e r of reasons. Firstly, since b o t h q u a l i t a t i v e a n d q u a n ­ titative approaches were followed to collect and analyse the data corpus assembled

b y C o n s o l o (1996), t h e r e s e a r c h d e s i g n a n d m e t h o d o l o g y

i n c l u d e d d i f f e r e n t p r o c e d u r e s a n d r e s e a r c h i n s t r u m e n t s . A s a result, d i f f e ­ r e n t t y p e s of i n f o r m a t i o n , for example, c l a s s r o o m d a t a o b t a i n e d b y m e a n s of a u d i o - r e c o r d i n g s a n d b y d i r e c t o b s e r v a t i o n , w e r e c o m b i n e d i n l e s s o n transcripts. The w a y such information was gathered and put together by t h e researcher a l o n e is l i k e l y t o favour d e v i a t i o n s b e t w e e n t h e n a t u r e o f phenomena,

as t h e y o c c u r i n t h e f i e l d , a n d t h e i r i n t e r p r e t e d versions

( L a m p e r t & Ervin-Tripp, 1993; Ochs, 1979). Recordings reveal c l a s s r o o m phenomena

w i t h a considerably h i g h degree of reliability. Data from

field notes a n d transcripts, o n t h e other h a n d , are affected b y i n t e r v e n ­ i n g factors c o n c e r n i n g the observer's i n t e r p r e t a t i o n of phenomena,

as

w e l l as h i s or h e r i n t e r p r e t a t i o n o f r e c o r d i n g s w h i l e l e s s o n t r a n s c r i p t s are p r o d u c e d . T h e s e f a c t o r s c a n m a k e r e s e a r c h o u t c o m e s s u b j e c t i v e a n d less r e l i a b l e , t h u s l i m i t i n g g e n e r a l i z a t i o n of r e s u l t s ( L a m p e r t & E r v i n T r i p p , 1993; Tesch, 1990).

116

Alfa, Säo Paulo, 43: 113-134, 1999

T w o " v a l i d a t i o n sessions", r e f e r r e d t o as v a l i d a t i o n sessions 1 (VS1) a n d 2 (VS2) w e r e c a r r i e d o u t so as t o v e r i f y t h e v a l i d i t y a n d t h e r e l i a b i l i t y of p a r t of t h e s y s t e m of c a t e g o r i e s u s e d t o c o d e l e s s o n t r a n s c r i p t s . D u e to p r a c t i c a l c o n s t r a i n t s i n h a v i n g a l l c a t e g o r i e s of d a t a a n a l y s i s v a l i d a ­ ted, t h e d e c i s i o n w a s t o p r o c e e d w i t h u s i n g p a r t o f t h e c o d i n g s y s t e m i n the same w a y those categories h a d been used to code data i n earlier stu­ 6

dies. C a t e g o r i e s s u c h as e x c h a n g e s a n d m o v e s h a v e t h e r e f o r e n o t b e e n f u l l y v a l i d a t e d n e i t h e r i n V S l n o r i n VS2. Raters w e r e a s k e d t o focus o n those categories t h a t h a d been subject to adjustment and redefinition, as f r o m t h e p i l o t s t u d y a n d t h e p r e l i m i n a r y d a t a a n a l y s i s c a r r i e d o u t b y 7

C o n s o l o ( 1 9 9 6 ) , a l l a t t h e r a n k of acts. V S l a n d VS2 a i m e d a t (1) t h e t r a i n i n g o f e x t e r n a l r a t e r s t o use t h e s y s t e m o n l e s s o n t r a n s c r i p t s , a n d (2) t o h a v e t h e r a t e r s c o d e s a m p l e s of d a t a , t h a t is, e x t r a c t s f r o m l e s s o n t r a n s c r i p t s , b y f o l l o w i n g s p e c i f i c des­ criptions of the categories of analysis. For V S l , e i g h t p r o f e s s i o n a l s ( t e a c h e r s a n d / o r r e s e a r c h

students)

w o r k i n g i n t h e areas o f ELT/ELT w e r e c o n t a c t e d r e g a r d i n g t h e i r v o l u n ­ t a r y c o n t r i b u t i o n t o v a l i d a t e t h e s y s t e m of c a t e g o r i e s of a n a l y s i s i n t h i s i n v e s t i g a t i o n . H o w e v e r , o n l y four of t h e m w e r e a c t u a l l y p r e s e n t i n t h e s e s s i o n . T h e " e x t e r n a l r a t e r s " i n V S l are r e f e r r e d t o as r a t e r 1, r a t e r 2, r a t e r 3 a n d r a t e r 4. Rater 1 a n d r a t e r 2 are n a t i v e s p e a k e r s o f E n g l i s h , a n d h a d b e e n w o r k i n g i n ELT for m a n y y e a r s w h e n t h e y t o o k p a r t i n V S l . Rater 1 h o l d s a n M A i n TEFL, a n d h a d b e e n f o r m a l l y e x p o s e d t o s o m e o f t h e t h e o r e t i c a l b a c k g r o u n d r e v i e w e d for t h i s s t u d y w h i l e s h e d i d h e r M A . Rater 2 w a s w o r k i n g t o w a r d s h i s M A i n TEFL w h e n t h e v a l i d a t i o n s e s s i o n w a s car­ r i e d o u t . H e h a d p r e v i o u s l y w o r k e d i n ELT i n B r a z i l , w h i c h m e a n s n o t o n l y d o e s h e s p e a k s o m e P o r t u g u e s e , b u t also h e is f a m i l i a r i z e d w i t h t h e t e a c h i n g c o n t e x t s i n v e s t i g a t e d here. Rater 3 a n d r a t e r 4 w e r e w o r k i n g o n t h e i r d o c t o r a l t h e s e s at t h e t i m e . Rater 3 s p e a k s E n g l i s h as L 2 , a n d h i s r e s e a r c h w a s i n t h e a r e a of s p o k e n E n g l i s h . Rater 4 i s a n o n - n a t i v e s p e a k e r o f E n g l i s h . B o t h o f t h e m are p r o f i c i e n t i n t h e E n g l i s h l a n g u a g e . B o t h t h e t r a i n i n g of r a t e r s a n d t h e c o d i n g of d a t a s a m p l e s w e r e car­ ried out o n the same occasion due to the constraints imposed o n c o u n t i n g o n v o l u n t a r y e x t e r n a l c o o p e r a t i o n . A c o m m o n p r o b l e m is t o a s s o c i a t e t h e

6

A s for t h e d i s t i n c t i o n b e t w e e n m o v e s i n T T ( I , Rt a n d F) a n d i n s t u d e n t s p e e c h (Is, R a n d Fs), t h e c h a r a c t e r i s t i c s of t h o s e m o v e s w e r e c l e a r l y u n d e r s t o o d b y raters i n b o t h V S l a n d V S 2 .

7

See C o n s o l o . 1996, p.105 12.

Alfa, Säo Paulo, 43: 113-134, 1999

117

p r a c t i c a l i t i e s of h a v i n g t h e r e q u i r e d p h a s e s of t r a i n i n g r a t e r s a n d v a l i ­ d a t i n g c a t e g o r i e s of a n a l y s i s o n d i f f e r e n t o c c a s i o n s , w i t h t h e l i t t l e a v a i ­ l a b i l i t y offered b y m o s t p r o f e s s i o n a l s w h o m a t c h t h e n e c e s s a r y dards to be e x t e r n a l raters.

stan­

8

If " w e l l - t r a i n e d a n d l i k e - m i n d e d coders" ( L a m p e r t & E r v i n - T r i p p , 1993, p . 196) are d e s i r a b l e , f a v o u r a b l e c o n d i t i o n s are t o b e e s t a b l i s h e d so as t o t r a i n r a t e r s p r o p e r l y . T h e t r a i n i n g of r a t e r s 1 - 4 i n V S l w a s c o n c e n ­ t r a t e d i n a s h o r t t i m e slot; t h a t o n e or p o s s i b l y t w o r a t e r s d i d h a v e n o t l e a r n e d e n o u g h a b o u t t h e c a t e g o r i e s of a n a l y s i s is s t r o n g l y s u g g e s t e d by unsatisfactory results i n their u s i n g the codes w h i c h resulted i n l o w l e v e l s o f a g r e e m e n t , as r e p o r t e d b e l o w .

T h e preparation of materials The samples from lesson transcripts used to validate the categories of a n a l y s i s i n c l u d e d b o t h p l a i n a n d c o d e d t r a n s c r i p t s ; t h e s e h a d b e e n r a n d o m l y selected from the corpus, according to the criteria e x p l a i n e d below. S a m p l e s w e r e f r o m t e n w h o l e lessons p r e v i o u s l y t r a n s c r i b e d a n d c o d e d b y t h e researcher. T h e t u r n w a s i n i t i a l l y a d o p t e d as a n i n t e r a c t i o ­ n a l r e f e r e n c e w i t h i n t h e t r a n s c r i b e d lessons, s i n c e i t c a n b e c l e a r l y i d e n ­ t i f i e d as a u n i t of s p o k e n l a n g u a g e a n d v e r b a l i n t e r a c t i o n ( t h i s r e s e m ­ b l e s t h e p a t h f o l l o w e d b y S i n c l a i r a n d C o u l t h a r d , 1975, t o w a r d s t h e d e f i n i t i o n of m o v e s ) , to b e c o d e d i n t e r m s of m o v e s a n d acts. M o r e spe­ c i f i c a l l y , t e n s e q u e n c e s of t w e n t y t u r n s e a c h w e r e s e l e c t e d f r o m t h e t o t a l of 6558 t u r n s c o n t a i n e d i n t e n lesson transcripts. T h e t o t a l n u m b e r of t u r n s w a s a d j u s t e d t o 6560, f r o m w h i c h 328 t w e n t y - t u r n s e q u e n c e s w e r e o b t a i n e d . For e x a m p l e , t u r n s 0 0 1 - 0 2 0 w e r e f r o m " L e s s o n 1" a n d 9

s t a n d as " s e q u e n c e 0 0 1 " . T u r n s 6540 - 6558, f r o m " L e s s o n 1 0 " , s t a n d as " s e q u e n c e 328". T h e t e n s e q u e n c e s r a n d o m l y c h o s e n ( b y r e f e r r i n g t o a t a b l e of r a n ­ d o m n u m b e r s ) are i n d i c a t e d i n T a b l e 1 b e l o w :

8

T h o s e factors a p p l y t o l i m i t e d a v a i l a b i l i t y u n d e r t h e c o n d i t i o n s of " v o l u n t a r y p a r t i c i p a t i o n " i n research. T h e p r o b l e m m a y be e l i m i n a t e d b y h a v i n g p a r t i c i p a n t s ' a v a i l a b i l i t y a n d p r o f e s s i o n a l c o m m i t m e n t i n r e t u r n to p a i d w o r k .

9

T h e l a b e l s "Lesson 1 - L e s s o n 10" for s a m p l e s u s e d i n t h e v a l i d a t i o n sessions are d i s t i n c t f r o m t h e l a b e l s LES 1, LES2, LES3 a n d so o n , r e f e r r i n g t o lessons t a u g h t b y e a c h teacher.

118

Alfa, Sao Paulo, 43: 113-134, 1999

Table 1 - V a l i d a t i o n sessions: s a m p l e s e q u e n c e s ( f r o m a u d i o - r e c o r d e d lessons) Turns

10

Teacher /Lesson

Sequences (numbering 001-328) 1 (020)

400 -420 421 -440

2 (021)

441 - 4 6 0

3 (022)

NNSl/"Lesson 2"

021 -040 281 -300

4 (028)

NNS4/"Lesson 8"

141 - 160 581 - 6 0 0

6 (229)

721 -740

8 (258)

001 - 020 601 - 620

9 (268)

NSl/"Lesson 1"

NS2-ADV3/"Lesson 9"

5 (041) 7 (251)

10 (298)

Further decisions c o n c e r n i n g t h e use of t h e selected samples s u c h as w h i c h s a m p l e s w o u l d b e m o r e s u i t a b l e for t r a i n i n g r a t e r s or for v a l i ­ d a t i n g the categories, a n d the actual operationalization of t h e t r a i n i n g a n d t h e c o d i n g p h a s e s i n t h e v a l i d a t i o n session, l e d t o t h e f i n a l c h o i c e of s i x s a m p l e s , t h r e e o f w h i c h are p r e s e n t e d i n A p p e n d i c e s 1-3. Working materials (booklets w i t h definitions

1 1

and examples, han­

d o u t s w i t h t h e s e l e c t e d sequences) w e r e p r e p a r e d a n d u s e d for t r a i n i n g raters, a n d for t h e c h e c k i n g o n t h e r e l i a b i l i t y o f t h e c a t e g o r i e s o f a n a l y s i s .

The training phase A f u l l y c o d e d s a m p l e ( A p p e n d i x 1) w a s t h e d e p a r t u r e p o i n t for p r e ­ s e n t i n g t h e codes for t r a n s c r i p t i o n a n d e x a m p l e s o f t h e c a t e g o r i e s of analysis (exchanges, moves a n d acts). T h e s a m p l e i n A p p e n d i x 2 w a s u s e d for t r a i n i n g r a t e r s t o do t h e c o d i n g of m o v e s a n d a c t s . T h e a u d i o t a p e s w e r e also p l a y e d i n t h e v a l i d a t i o n sessions so as t o i l l u s t r a t e v a r i a -

10

Teachers i n v o l v e d i n C o n s o l o ' s (1996) s t u d y are n a t i v e speakers (NS) a n d n o n - n a t i v e speakers (NNS) of E n g l i s h .

11

A s p r e s e n t e d i n C o n s o l e 1996, p.171-86. I n V S 1 , raters h a d t o g e t a c c q u a i n t e d w i t h t h e d e f i n i ­ t i o n s d u r i n g t h e first p a r t of t h e session, a n d i t is a r g u a b l e w h e t h e r t h e y w e r e a b l e to g r a s p t h e d e f i n i t i o n s of t h e acts i n order t o a p p l y s u c h c a t e g o r i e s i n t h e c o d i n g of d a t a s a m p l e s . T h e s e c o n d g r o u p of raters (VS2) h a d t h e o p p o r t u n i t y t o s t u d y t h e d e f i n i t i o n s a t t h e i r w i l l b e t w e e n t h e first and the second day

Alfa, Sào Paulo, 43: 113-134, 1999

119

tions i n meaning conveyed by intonation contours

1 2

a n d p r o v i d e raters

w i t h a m o r e r e l i a b l e a c c o u n t of t h e p h e n o m e n a u n d e r i n v e s t i g a t i o n .

Results: V a l i d a t i o n S e s s i o n 1

The subcategories of acts dealt w i t h i n VS1 a n d levels of agreement b e t w e e n each t w o raters w h o have rated the occurrences of t h e same a c t s are p r e s e n t e d i n T a b l e 2:

T a b l e 2 - A g r e e m e n t b e t w e e n raters i n V S l Range of acts

Raters

Level of agre­

Coefficient of

ement ( p )

agreement: K

0

SUBGROUP I

rater 1 - rater 2

34.6% ( p = 0.34)

{mrk}, {str}, -fell}, {cp-chk}, {cf-chk}, {cla}, {dir}, {inf}, {com}, {clu}, {mdl}

0.25

researcher - rater 1 researcher - rater2

74% (p = 0.74) 38.4% ( p „ = 0.38)

0.65 0.29

SUBGROUP I I

rater 3 - rater 4

51.4% (p = 0.51)

0.46

researcher - rater 3

71.4% (p 0 = 0.71)

0.68

{bid}, {nom}, {ack}, {y-rpl}, {n-rpl}, {c-rpl}, {rprpl}. {i-rpl}, {rea}, {pro}

| researcher - rater 4

0

0

0

64% (p 0 = 0.64)

0.58

T h e p r o c e d u r e for v a l i d a t i n g t h e a c t s w a s t o h a v e t w o r a t e r s c a t e ­ gorizing the same data samples independently, and t h e n determine the d e g r e e a n d s i g n i f i c a n c e o f t h e r a t e r s ' a g r e e m e n t ( C o h e n , 1969; L a m p e r t & E r v i n - T r i p p , 1993). F o l l o w i n g C o h e n (1969), t h e p r o p o r t i o n o f cases a b o u t w h i c h t h e r a t e r s a g r e e d ( n o m i n a l s c a l e a g r e e m e n t ) is d e t e r m i n e d b y t h e c a l c u l a t i o n o f p . For e x a m p l e , t h e i n t e r - r a t e r a g r e e m e n t b e t w e e n 0

the researcher

a n d r a t e r 1 for a c t s r a n g i n g b e t w e e n " m a r k e r s " a n d

" m o d e l s " , as s h o w n i n T a b l e 3 b e l o w , is g i v e n b y p

0

= 0.27 + 0.06 + 0.04 + 0.02 + 0.29 + 0.02 + 0.04 = 0.74

A c e r t a i n a m o u n t o f a g r e e m e n t b y c h a n c e is, h o w e v e r , e x p e c t e d . T h i s c a n be d e t e r m i n e d b y m u l t i p l y i n g t h e p r o b a b i l i t i e s o f t h e m a r g i -

12 Intonation has not been indicated in transcripts, except for •?• for interrogates and standard intonation patterns such as (RISE|, [FALL], [FALL-RISE) and [RISE-FALL].

120

Alfa, Säo Paulo, 43: 113-134, 1999

n a l s . For a c t s 1 ( m a r k e r ) - 11 ( m o d e l ) , t h e r e s e a r c h e r p l a c e d 0.08 o f t h e cases u n d e r t h e c a t e g o r y " c o n f i r m a t i o n c h e c k s " ( c f - c h k ) , w h i l e r a t e r 1 p l a c e d 0.04 o f t h e cases i n t h i s category. T h e e x p e c t e d c h a n c e a g r e e m e n t for c f - c h k is t h e n (0.08)(0.04) = 0.0032. Values for c h a n c e a g r e e m e n t are t h e p a r e n t h e t i c a l e n t r i e s a l o n g t h e a g r e e m e n t d i a g o n a l i n T a b l e 3. T h e p r o p o r t i o n o f a g r e e m e n t t o b e e x p e c t e d b y c h a n c e , p , is f o u n d b y c

a d d i n g t h e parenthetical values: p = 0.1089 + 0.0084 + 0.0032 + 0.0008 + 0.1369 + 0.0008 + 0.0024 = 0.2614 c

T h e c o e f f i c i e n t o f a g r e e m e n t , K , is t h e p r o p o r t i o n o f a g r e e m e n t after a g r e e m e n t b y c h a n c e is r e m o v e d f r o m c o n s i d e r a t i o n , w h i c h c a n b e o b t a i n e d as f o l l o w s :

K=

P

0

;

P

c

= 0.65

Pc

V a l u e s o f K r a n g e d 0.25 - 0.65 for a c t s i n s u b g r o u p I a n d 0.46 - 0.68 for s u b g r o u p I I . G i v e n t h e m i n i m a l a m o u n t o f t r a i n i n g g i v e n t o r a t e r s 1 - 4 a n d c o n s i d e r i n g t h e observable characteristics of t h e i r h a n d l i n g of c a t e g o r i e s a n d d a t a s a m p l e s d u r i n g V S l , t h e l e v e l s o f a g r e e m e n t for a c t s d e a l t w i t h r a t e r s 3 a n d 4 w e r e s l i g h t l y h i g h e r t h a n t h e o n e s for s u b g r o u p I ( a n d e s p e c i a l l y for r a t e r 2) m a y b e d u e t o t h o s e r a t e r s ' p r e v i o u s k n o w ­ l e d g e o f t h e S&C m o d e l a n d c u r r e n t i n v o l v e m e n t i n r e s e a r c h . R a t e r 2's l o w l e v e l s o f a g r e e m e n t , t h e l e a s t e x p e r i e n c e d m e m b e r i n t h e g r o u p as far as b o t h d i s c o u r s e c a t e g o r i e s a n d r e s e a r c h p r o c e d u r e s a r e c o n c e r n e d , corroborate this conclusion.

Alfa, Säo Paulo, 43: 113-134, 1999

121

Selected

0.04

CO

o

CO

V

CD

CD

o

122 0.06

0.02

CO d

4. cp-chk

0.02

c

11. mdl

0.02

0.02

o S 0.02

S

0.37

9. com

0.02

10. clu

ö ©'

Pi researcher

0.02

0.29 (0.1369)

0.06

8. inf

ö

N/A

0.02 (0.0008) •

7. dir

'

10. clu

9. com

0.02

7. dir

6. cla

0.04

0.06 (0.0084)

0.27

¡2

3. eli

6. cla

0.02

0.02

11. mdl

0.04

0.04

0.37

0.04

0.04

0.14

0.33

¿T

5. cf-ctik

CD

1. mrk

°

0 U C

2. str

5. cf-chk

CD 4. cp-chk

pe 3

3. eli

CO CD CO CD

1. mrk 2. str

CD CO

ZOO

p zoo

CD

acts

CO CD

ß a W

11

5

0

& O O

CO 0 d

Alfa, Säo Paulo, 43: 113-134, 1999

Validation Session 2 B e c a u s e t h e l e v e l s of a g r e e m e n t a c h i e v e d i n V S l w e r e n o t c o n s i d e ­ red satisfactory, a n d g i v e n the chances t h a t such results m a y have been negatively affected by t h e conditions under w h i c h V S l w a s c o n d u c t e d , a s e c o n d s e s s i o n w a s c a r r i e d o u t . T h e r a t e r s i n VS2 w e r e a l l M A TEFL s t u d e n t s , w i t h t h e e x c e p t i o n of one m e m b e r of t h e a c a d e m i c staff w h o v o l u n t e e r e d t o j o i n t h e g r o u p . T h e six r a t e r s i n VS2 are r e f e r r e d t o as r a t e r 5 - r a t e r 10. U n l i k e r a t e r s 1 - 4, r a t e r s i n VS2 ( e x c e p t for t h e staff member) h a d not been explicitly exposed to the theoretical b a c k g r o u n d for c a t e g o r i e s o f d i s c o u r s e a n a l y s i s . T h u s r a t e r s i n VS2 n e e d e d

sound

t e a c h i n g a n d p r a c t i c e i n u s i n g t h e c o d e s before a t t e m p t i n g t o r a t e t h e set of acts. VS2 w a s a c t u a l l y c o n d u c t e d o n t h r e e d i f f e r e n t d a y s so as t o a l l o w for a p p r o p r i a t e t r a i n i n g o f r a t e r s a n d s a t i s f a c t o r y r a t i n g of t h e c o d i n g s y s t e m . T h e first t w o m e e t i n g s (days one a n d t w o ) w e r e e n t i r e l y d e d i c a ­ t e d t o t h e p r e s e n t a t i o n of t h e s t u d y a n d a d e s c r i p t i o n of t h e c o d i n g sys­ t e m , b y t h e researcher, t o t h e g r o u p o f raters. T h e s a m e s a m p l e s

from

lesson transcripts used to t r a i n raters i n V S l w e r e c o d e d a n d discussed, a n d r a t e r s 5 - 1 0 h a d m o r e t i m e t h a n raters 1 - 4 t o u n d e r s t a n d t h e c a t e ­ gories a n d p r o c e d u r a l a s p e c t s of c o d i n g f o l l o w e d b y t h e researcher. O n t h e t h i r d day, t h e a c t u a l r a t i n g o f t h e c a t e g o r i e s a t t h e r a n k o f acts w a s c a r r i e d o u t . Raters w e r e r a n d o m i c a l l y g r o u p e d i n p a i r s , a n d e a c h p a i r d e a l t w i t h a p r e v i o u s l y d e t e r m i n e d s u b g r o u p of a c t s , as i n V S l . I n VS2 n e a r l y a l l t h e a c t s w e r e r a t e d ,

1 3

as o p p o s e d t o V S l , i n w h i c h t h e

n u m b e r of raters w a s too s m a l l to dealt w i t h a l l types of acts. T h e proce­ d u r e of r a t i n g s u b g r o u p s of a c t s w a s a d o p t e d so as t o f a c i l i t a t e t h e h a n ­ d l i n g of d e f i n i t i o n s t o b e f o l l o w e d b y raters. I n t h i s w a y , c a t e g o r i e s for o c c u r r e n c e s of s p e c i f i c a c t s t h a t m a y l e a d t o d o u b l e - l a b e l l i n g , for e x a m ­ ple c o m m e n t s a n d informatives, were s i m u l t a n e o u s l y tested by t w o raters w o r k i n g o n t h e s a m e s a m p l e s . A p p e n d i x 3 i l l u s t r a t e s h o w t h e same sample w a s prepared to be h a n d l e d by different raters, e a c h g r o u p d e a l i n g w i t h a p r e d e t e r m i n e d set of acts. G a p s w e r e p r e p a r e d for t h e l a b e l l i n g of a c t s w i t h i n t h e r a n g e b e i n g d e a l t w i t h e a c h p a i r o f r a t e r s .

13

Four types of a c t s - g r e e t i n g s , p a r t i n g s , asides a n d t r a n s l a t i o n s - h a v e n o t b e e n v a l i d a t e d i n t h i s s t u d y d u e t o t h e r e d u c e d n u m b e r of raters a n d t h e s h o r t a m o u n t of t i m e t o d e a l w i t h t h e l a r g e n u m b e r of c a t e g o r i e s . I t has b e e n a s s u m e d t h a t , s i n c e t h o s e c a t e g o r i e s are d r a w n from s i m i l a r s t u d i e s o n c l a s s r o o m b e h a v i o u r a n d t h e i r d i s c o u r s e f u n c t i o n s are less p r o n e t o a m b i g u o u s i n t e r ­ p r e t a t i o n , t h e y w o u l d be t h e ones t o be left o u t .

Alfa, Sao Paulo, 43: 113-134, 1999

123

Because this procedure resembles the technique of t e s t i n g language by f o c u s i n g a t t e n t i o n o n one p o i n t ( g r a m m a r or v o c a b u l a r y , for e x a m p l e ) at a t i m e , t h e c o d i n g of a c t s w a s o p e r a t e d u n d e r t h e p r i n c i p l e s of a "discrete p o i n t test" (Oiler Jr., 1979), a n d i n w h i c h raters c h o s e t h e c a t e g o r i e s f r o m a p r e - e s t a b l i s h e d set of " a l t e r n a t i v e s " , as i n a m u l t i p l e c h o i c e t e s t . T h e l e v e l s o f a g r e e m e n t for e a c h set of a c t s i n V S 2 are p r e s e n t e d i n Table 4 below: T a b l e 4 - A g r e e m e n t b e t w e e n r a t e r s i n VS2 Range of acts

Raters

SUBGROUP I | m r k ) , (str), (eli), (cp-chk), (cf-chk), (cla), (dir), (inf|, (com), (clu), (mdl)

rater 5 - rater 6

Level of agreement Coefficient of agreement: K 44.2% (p = 0.44)

0.41

researcher - rater 6 researcher - rater 6

37% (p„= 0.37) 54% (p = 0.54)

0.25 0.41

SUBGROUP II (bid), (nom), (ack), (y-rpl), (n-rpl), (c-rpl), (rp-rpl), (i-rpl), (rea), (pro)

rater 7 - rater 8

80% (p = 0.8)

0.75

researcher - rater 7 researcher - rater 8

97% (p = 0.97) 83% (p = 0.83)

0.96 0.79

SUBGROUP III (ack), (cor), (evl), (mst), (apl), (thk), (ecg), (con), (ter). (grt) (prt)

rater 9 - rater 10

40% ( p = 0.4)

0.31

researcher - rater 9 researcher - rater 10

65% (p = 0.65) 45% (p„= 0.45)

0.51 0.24

0

0

0

0

0

0

0

T h e h i g h e s t l e v e l s of a g r e e m e n t for a l l t h e r a t e d a c t s , i n b o t h V S l a n d VS2, w e r e t h e r e f o r e 7 4 % for a c t s i n s u b g r o u p I ( K = .65), as s h o w n i n Tables 2 a n d 3, 9 7 % for a c t s i n s u b g r o u p I I ( K = . 9 6 ) a n d 6 5 % for a c t s i n s u b g r o u p I I I ( K = . 5 1 ) , as s h o w n i n T a b l e 4. L e v e l s of a g r e e m e n t r e a c h e d i n V S l a n d VS2 are, o n t h e w h o l e , n o t as h i g h as t h e l e v e l s r e a c h e d i n e a r l i e r s t u d i e s i n w h i c h c a t e g o r i e s for a n a l y s i n g l e s s o n t r a n s c r i p t s w e r e v a l i d a t e d , as for e x a m p l e i n M i t c h e l l e t a l . (1981) a n d R a m i r e z et a l . ( 1 9 8 6 ) .

14

The results reached here m a y

h a v e h a d t h e i n t e r v e n t i o n of t h e f o l l o w i n g f a c t o r s . Firstly, t h e c o m p l e x i t y of t h e s y s t e m of d i s c o u r s e e x c h a n g e s a n d acts, w i t h i t s l a r g e n u m b e r of p r o p o s e d c a t e g o r i e s , e s p e c i a l l y a t t h e l e v e l of a c t s . T h e s p e c i f i c s of e a c h d e f i n i t i o n t h a t , i n s o m e cases, d i s t i n g u i s h a c t s t h a t h a v e s i m i l a r c o m ­ m u n i c a t i v e f u n c t i o n s , h a v e t o b e c a r e f u l l y g r a s p e d b y coders. T h i s s e e m s

14 T h e p r o c e d u r e for d e t e r m i n i n g r e l i a b i l i t y o n t h e c o d i n g of l e s s o n t r a n s c r i p t s i n t h e s t u d y c o n d u c ­ t e d b y R a m i r e z et a l . (1986) w a s d e v e l o p e d b y C. C h a u d r o n a n d o t h e r s at t h e U n i v e r s i t y of H a w a i i , as r e p o r t e d i n C h a u d r o n (1988, p.24).

124

Alfa, Sao Paulo, 43: 113-134, 1999

to b e f e a s i b l e , h o w e v e r , o n l y after e x t e n s i v e t h e o r e t i c a l a n d m e t h o d o l o ­ g i c a l f a m i l i a r i t y w i t h t h e s y s t e m a n d t h e c o d i n g p r o c e d u r e . Secondly, l a c k of f a m i l i a r i t y w i t h t h e s y s t e m p r o b a b l y l e d m o s t r a t e r s t o m i s i n t e r ­ p r e t d e f i n i t i o n s a n d / o r h a v e d i f f i c u l t y i n c o d i n g acts t h a t are e x p e c t e d t o be p r o b l e m a t i c d u e t o t h e i r s i m i l a r f u n c t i o n s . T h e s e are, for e x a m p l e , c l a ­ r i f i c a t i o n s (cla), c o n f i r m a t i o n c h e c k s (cf-chk) a n d c o m p r e h e n s i o n c h e c k s ( c p - c h k ) ; c l u e s (clu), i n f o r m a t i v e s (inf) a n d s t a r t e r s (str); a n d a c k n o w l e d ­ ges (ack) a n d e v a l u a t e s (evl). Cases i n w h i c h t h e r e s e a r c h e r a n d r a t e r s d i s a g r e e d o n t h o s e a c t s are s h o w n i n T a b l e 5 b e l o w :

T a b l e 5 - A c t s : r a t e r s ' d i s a g r e e m e n t i n VS1 a n d VS2 Subgroup I Case n.

Researcher Rater 1

Subgroup 11

(12)

cf-chk

cp-chk

Case n.

Researcher

Rater 7

(15) (20)

str

inf

(21)

pro

inf

str

dir

(21)

eli

inf

(23)

eli

inf

Subgroup I I I

(24)

inf

eli

Case n .

Researcher

Rater 9

(27)

inf

eli

(4)

ack

i-rpl

(28)

inf

eli

(6)

ter

ack

(31) (37)

inf

-

(13)

ack

evl

com

cor

(15)

ack

evl

(38)

mdl

cor

(17)

ack

HES

(43)

eli

cp-chk

(18)

ack

evl

(44)

clu

inf

(20)

ack

evl

(45)

cla

cp-chk

A t h i r d f a c t o r a f f e c t i n g t h e l e v e l s of a g r e e m e n t w a s m i s i n t e r p r e t a ­ t i o n , b y r a t e r s , of d e f i n i t i o n s a n d c o d i n g i n s t r u c t i o n s , l e a d i n g t o d i s c r e ­ p a n c i e s i n cases s u c h as (21), (23), (24), (27) a n d (28) for s u b g r o u p I , i n w h i c h t h e r e s e a r c h e r a n d r a t e r 1 d i s a g r e e d o n e l i c i t i o n s (eli) a n d i n f o r m a t i v e s (inf); a n d i n cases (4) a n d (17) for s u b g r o u p I I I , i n w h i c h r a t e r 9 did not follow the instructions a n d used categories other t h a n the actas i n subgroup III. These d r a w b a c k s c a n be l i n k e d to the c o n d i t i o n s u n d e r w h i c h V S l w a s c a r r i e d o u t , for d e s p i t e t h e e x p e r i e n c e o f r a t e r s 1-4, as reported i n t h e i r profiles above, t h e y w e r e n o t fiben e n o u g h t r a i n i n g to deal w i t h t h e categories proposed here. N e v e r t h e l e s s , h i g h l e v e l s of a g r e e m e n t are p o s s i b l e , as s h o w n b y t h e r e s u l t s r e a c h e d b y r a t e r 7 ( s u b g r o u p II) i n c o m p a r i s o n t o t h e resear­ c h e d ( K = . 9 6 ) , as s h o w n i n T a b l e 6 b e l o w :

Alfa, Säo Paulo, 43: 113-134, 1999

125

Selectd

12. bid

0.03 0.24

S 5 ''• o & 0.14

15.2. n-rpl

0.03

v

p¡ researcer

0.24 ¡0.0576)

15.1. y-rpl

15.3. e-rpl

0.03

0.03 (0.0009}

15.4. rp-rpl

0.26

0.26 (0.0676)

15.5. i-rpl 16. rea

0.03

0.03

o o. i