Testing literacy and related language skills: Part II

k. or the se lecti on of a'

Uller,J. W., Jr. (19~2). Testing literacy and related language skills: Part II, examples of testing procedures and actlV1tles. In S. Rattanavich, R. F. Walker, & J. W . Oller, Jr. (Eds.), Teaching aU f the chIldren to read (pp. 67-94). London and New York: Open University Press.

site or a fi nent. o r acti\ity to use or foste ' eld_ r In the

CHAPTER 6

()rs suc h as interest, drama tizahili hke , the test ....Titer also needs t t), ' g, ,\ccordin g to the th take he testin eory lot be am testing I\'hateler that d . I 'C ' Oes c anllc atlon (the ler~; dennl' t' , " IOn) o f l hool expenence IS ahou t Onl ' , " I te~ts r~ and teachers alrke to rienne th . . d elr Ilor s, the tests ollgh t al\\a\'s to be eI ought to he good teac hing tool l1edlatelv preceding section are take~ 'ct on disc ourse fo rms that have b , een the ex pe nence of the children to be It helps learners to unde rstand both 110wn facts, For th is reason, it will do , ts per se and at the sa me time to use

°

Testing literacy and related language skills: Part II, Examples of testing procedures and activities JOHN W, OLLER. Jr

Introduction Before gi\ing examples of actual testing activities that might be developed it will be useful to define the term 'test', What is to count as a test? In fact, any discourse processing performance that can be judged as better or worse relative to some normative standard can be qualified as a test. All that is required is that the rater (the tester, teacher, or whatever rater or group of raters are involved) have a reliable notion of what a better or worse performance on the task in question consists of. A task that meets this minimal testing requirement may be called a scalable test. That is to say, performances can be rated on a scale ranging from better to worse. Most language processing tasks, where the factual basis (as discussed in the preceding chapter) is known, will qualify as tests in this minimal sense. Examples would include such things as telling a story, summarizing an argument, recounting a conversation. explaining a process, describing a scene, giving a speech, taking pan in a drama. and so on. A more stringent requirement for a test is that it be scorable, in other words, it must be possible to reduce a performance of the task to a number that represents right or wrong answers. or some determinable quantity. Not all scalable tasks tasks. meet this more stringent requirement, though all scorable tasks must, in pnnciple. always be scalable. Of the class of scorable tasks. a still smaller set of so-called 'objective' tests exist which can be reduced to marks on an answer sheet or other device (e.g. a keyboard or computer screen) that can be mechanically scored. Further. there probably are tasks which are so indeterminate (usually because no factual basis for them can be determined) as to be unscalable. In general. when the facts to which a given discourse performance relates are unknown or cannot be

"

l

-.

68 Teaching all the children to read determined, the performance itselfwill not be reliably scalable or sco~ It will not serve as an adequate basis for a test. . . The kinds of tasks that might qualify as tests can be summed up in Fi 6.1. Discourse tasks can be divided, loosely, into g.

as tests or as teaching

degre co ndi menL

2. those that are determinate and therefore at least scalable (which mini.

sure'

mally qualify as potential tests or as teaching activities) ; 3. those that are sufficiently determinate to be both scalable and scorable (i .e. ones that are especially useful for classroom purposes) ; 4. those that are machine-scorable (those most commonly used in widescale institutional testing) .

score items in ino num!

1. indeterminate ones (which are unsuitable

activities) ;

,\

II even 1 which

Generally, at the national, provincial, district, or even school-wide levels, only machine scorable tests will be applied, while individual classroom teachers will generally prefer tests that fall into categories (2) and (3) , that is that are determinate enough at least to be scalable and in many cases also scorable. Admittedly, for some institutional purposes, it will occasionally be necessary to use screening procedures that are not machine-scorable. Some test specialists see this as a disadvantage owing to the fact that a •subjective , element of judgement enters the rating of any non-mechanical scoring procedure. However, what such specialists overlook. is that the same sort of 'subjective' judgement enters any scoring or rating whatever including the kind that ultimately depends on an optical scanner or other mechanical device. The fact is that someone (or some group) must decide in advance of any machine scoring what the correct answers are to the test items. At the point where those decisions are made, a machine-scorable test is no more 'objective' than any merely scalable wit ~

that'

mad maId or tr ques out fillir scor sud ica!

that

any caD COl

a" poi

dd All pouible diIcoune IUb (Includiar indeterminate ones Ibal oupt DOC to be UIed In cJa.rooma)

~

Scalable diKoune IUb (pcIIIihIe choices for cJa.room 11M)

~

Scorable diKoune !alb 'O~· .

machine ICOftbIe teSII

I

-.

tnt

vet

T

~.

un su itahl e as tests ()r

d'i

1(-"1 ((

I I I ! I~

i,trict. or nen ,choo l-wide I(' \'el ' 1\' " iIe Illdilid ua l c lassroo lll tt, '~' lOll II , ,tc itT .ltegOrtes I ':2 1 an d ( 3), lh a t is [I ' , " II I' l d t dre I ) e ,lll( III Ill ~Ul\ ca,t's also scorable ,t! , purpllst'S. It \\'ill OCGtsi(J\1,tlh he n ,lh 'll ,Ire IHll lIlachin e-scorahl e , ,Icil,lllt,we (Jllina til e t'act , C ,:> to ' t I lat ' 'rs the, rating of am' nOll-me e h _ ell , ' 'a nl' ca ) ,pe .. " clallst s merlook i,' tint , th e ',line II scortng or rating whate\'er includ1I1 , lil o ptIcal scanner or other me11 (' (or some group ) must d ec ide in the co rre c t answers are to the te st I, are llIacIe, a mac hin e-scora bl e test ,ca labl e task is, (includi ng ;la [ ou ght no t

-.15\..5

~ ... r o oms)

,ks (p o ssible 'O lll

u se )

1 -

Examples of testing procedures and activities

69

In fact, there is no guarantee whatc\'er that a mac hine-sco rable tesl 1\'i11 I ' In u re IFilflh/f' ( mu c h less that it will be more l 'ftiirl) than am te,t I1, ,'>ilIU ' I\lli d, drv , ill >I Illt';lll ill,l.;', Iht"I' ,In' gell el ,l lll • l< t', ill tltt, lir 't '('Ilten('(' ill tire III till' 11t'''t tllere are Ilill e , In j0l'l11 ,gilt'I1 ,Ih()\'(, ( p, I)q ), th(" '1 ('11 (1I't' dtill ,g' ,I., ,'iCpdr,lte I\ o rd s , .I 11IVI lt'I1 I, If phOIH'llIt'S I\cre '1(' 1\ ()lI ld be 11Iuc h higher, Tht' !ird the 11 Ulll Iwr of phOI1ellll'S, Ih l ' 1l'(lnk ,\llclthe Ilullllwr ()r Illg, 'HI, \[1 11 ,1111(' i, SUfie,' ,I., ,rrllctures ,Ire l1lade lip ()f Ills titute the phrase, ,Illd SCll ' 1( >llllt, to a parti ,tl ,ICCO Ulltin ,g lits of bll gua,g t' t hat lise n an' .. p e rh 'lps thn a re tht, hnt Ir,g ullll'llt ag,l inst,co ring for t' the ullit for scoring . ,I setle ~. a recog ni/,lble but imperIlt din
T J

childre the ,an

C( As

Reading aloud ,\s s(Jon as the ch ildre ll k\\e understood the discourse well enough to reprodllce it in actually fluellt!l' reading ill' ,egmellts to be read aloud .It ,\ timt'. or Oil an O\'erhe'lel , rllt' cant." tr"nspart"ncics, llr Il'"ching acti\it\ in which the

,-\.s soon as the ch ildren are reading the big book, or e\'en some portiOll of it. the\ are in a position to begin writing it as well. There is no reasun to postpone \\Titing lliltil the second grade. On the con trar•. there are 111.111\' re ason, tl1 ,lt reading ~lIld \\Titing tasks n eed to be c lose h- linked \\ith l',lc h uther ~1I1r\ \\'ith listening

(5

Su far, the children have observed and understood the facts of the discourse from its dramatization . They ha\'e heard and understood its spoken form. Thev have listened to and reproduced the spoken form sentence-bl' ,entence. The' have seen the written forms associated with the familiar discourse. They have heard these forms associated with spoken versions of the discourse . Thev ha\'e read the written forms aloud and they have learnt to I\Tite those written forms. Thev are now ready, owing to the scaffolding now in place, to write from dictation. A spoken version of the text may be presented segment-by-segment for children to write down. They have already linked the teacher's speech forms to speech forms they themsell'es produce, They have linked those same speech forms to written ones the teacher produced. They linked those written forms the teacher produced back to the speech forms of the teacher . Subsequently they linked the written forms produced by the teacher to spoken forms the children themselves produced (reading aloud). Later, they linked the written forms produced by the teacher to written forms they themselves produced. Dictation as a testing (or teaching) task simply takes the last representational form (a written version produced by the student) and links it back to the teacher's spoken version of the familiar discourse . As in the case of all the other elicitation devices designed to assess (or to instil) familiarity . . . lth one or another form of the fact-based discourse, dictation too is an elicitation dellce. The teacher (or someone else, either live or on a tape-recording of some sort) presents a segment of the discourse in a spoken version and the task set the children is to . . . Tite down the segment that they have just heard. As in all such tasks, the difficulty may be adjusted upward or downward by giving longer or shorter segments of text respectively, or by slowing down the rate of speech, repeating the segment one or

IIl'\l'!" U'LIlt" ,

lo ,1Ilnilill ( I f lilt' e lllll'C C()lllf) lex of[il,' 'II " {J\\ II ,I lld IITittcll t, [ \\' ~IS ,Isicep, ' \l\' grandmother \\'as 'Isleep .. ' \l\' mother was ,Isleep, The hlllgLlr !Dok lhl' TY and stereo / and "'em right out the frollt door, :\ub()(\1 sal" him" hut \I'e hgured out what lnppl'lled the nex t Illorn in g, The objectiH' is to present a segment that is comprehensible, b ounded hI natural sm tactic and / or ot her junctures, and challenging enough that th e child mllst comprehend the meaning of that segment by working-up a representation of the facts referred to by it. We want dictated segments tu be associated with the facts of the discourse as well as with its surface forms both spokell and l\Titten, We do not want it to be possible for the chi ld to link the surface form of speech directly with the surface forms of writing Il'ithout understanding the meaning of either one, In general, this is n o t possib le if we work I\ith significant segments of a whole , fact-based discourse , A direct. short-circuited association between spoken and written surface forms can only occur in something like a traditional spelling test where some of the meanings of at least some of the words are not known to the child but the surface forms (both spoken and written) are known , Such a short circuit cannot obtain if we are using significant multi-syllabic words and phrases of a whole . fact-based discourse , There are many assurances from other tests ( teaching actilities) that the children really do know the meanings of the surface forms in question, Furthermore , bv presenting long enough segments of speech between pauses, we can be certain that

80

Teaching all the ch ildren to read

short-circuit surface linkages are impossible even for any single seg ment. If th e seg m e nt is too lo n g to manage in short-term mem ory without d eep le \ el co mpreh e nsion a direct assoc iation between mere surface-forms cann o t he effected b\ the c hild o The me a ning of both the spoken and written fo rm s must be understood in o rder to assoc iate them with each other. Scoring of dictation can proceed e xactly as the scoring of the other e li c itation procedures d esc ribed abmoe, so no m ore needs to be said on that aCCO llnt. In stead , it is time to Illm oe on to consider kinds of processing tasks that will ensure still deeper le\Oe ls o f pro cessingo So far , the question-based tests ;1, \,ell ~IS the e licitation devices discussed (imitation, reading aloud, cop\ing, and di ctati o n ) a ll d e pend fairly exdusi\O e ly e ith er on specifi c facts alreach re fe rred to in the di sco urse and th e ir connection to particular SUrbee forms (\\hich h;\\Oe a lso already been prmided ) or they depend On co nn ec tion s he t\\O een diffe ren t surface-forms (eogo spoke n \Oe rsus written o nes ) "hich are de ep ly interre lated with each other a t the leve l of the bcts (see Fi g o Jo ~~ anc! its explanati o n )o It is time now in our testing (o r teaching) acti\ities to go o n to co n sider tasks that more direc tly demand infere nces of an inducti\O e sort, or the production of forms not directly provided in an: o prior discourse sllstained bv short-term memoryo It is time to move on also to d edu cti\el y based se mantic and syntactic relations as well as inferences that reac h inducti\Oeh bemnd what is stated in the discourse o

r I

,,"V I

at ( \,00

tan

tat we ce l of co wr gu kr TI (f ()t'th e sides oLt rigitt-;lllgle tri,\llgle ;trt' l'qU ~ tl l() tlte 'qll ~lr e ()f the 11Ipotel1llse ) , or the proof th ~tt lhl' 'Cllldre J'()()l ()f~ i, dll Irr~Ui()ll;tlllllllllwr, If these proofs or ;\111 ()ther kld 11

g-filllll)c','rhis torrlcl lead irrto r]latters ct'rrrcerning lans th:rt pf()tect thr lrt{)l)('l-1\ rif t itizctts. llrlw those' lalr's ltre rn:rde, rr'h:lt crirne costs, anc[ s() ()n. ![:rrn ,rtlt,'r'g('ttr'('\ of cliscoLttse rnisht be introdtrcecl. Relatire to t]ie insrrr.rrrr'c r'luirrt, tlrc rcpl:rct'nrent \alue oi'the stolen soods \\'ould haie to be (lr.tcr urin('(i. f ht' u lrole ccorronric cllrestion cotrld be blgLrght int6 tIe prt tLrlc aiortg rvit[t c()Ilt]tless ()ther discolrr se possibilities. \lorertr.e r, lte(..tLt\c.lll of these p()teittial expansioris of the startins discorrrse rcl:ite t() .r( tllrl frict\ of t'rFrcr-icttce (or Jtiurrsiirle fictions that have such ties), nll of tlrt':ilror.: t('('()iltrttcrrclations cottcerning te\ts appl\ to all such forrns,rf' (lt\r ( )til \('. (

't'tlt'r-.rih

. lirl clessro()lti purp()ses. rr'hcrr tasks of the solt

cle'scribccl

.rlrore irt t'rt'tcist' I\ lri'e rrsccl. sornc generill sr:ale of aciequacr srrpplenre tttcri lrr .lrct rlit sc.tlc: ot cliagnostic tre2rtnlents of'te st perfbrntirnces rrill lre ltlt'ft'r'r't'rl ()\('l :t stt-ict scolirtg procecltrr-e. F9r ipstance. the s6rt ()f 5clrl. rc(()nlnr('lrrltrl lrtrort' ip. 71) nriuht be rrsed tl-roLrgh it rvorrld h1,,'e to bc lilil(lc a littlc rtlott'gt'ttttnl tn certaitt respeCts and r11ore spccific in .,thc|s. Irrr.rgirre. firr ex.rnrple. tlrat the tusk set (the test or teaching e \crcisi ) irrrolrcs I)la\ir)g tlre role of-Suzie's rn()ther and respon(line orallv to questiorrs 1,rrrt br the ltolite oilic't'r (u role, sar', plaied bt the teacher) takinq l r.cl)()rt aborrt thc lrrrlglan. Sr.rtrtpose firrther (just lor the sake oIthe disctr:siorr here) that it lras bceri previoLrslv determined that the date of tlrt, lobberr ivlrs 2 \overrrbcr 199 1. thc TY rvas:r 2.l-inch Soni.worth lrh()rrt S1.r0. and the stereo rr':rs li Panirsonic that cost 5650 w,hen it \\'As lrc\r. Intagitte frrlthcr th:rt tl-re children have reaci aloud, rvritten from dtctatl()lt, anci participateLl in olfrer uctivitres in the classroom that rvould assrrre rliat tlle task at h:incl is not belond rvhat thev might be expected to do rvitlr \orllc strccess. Atl Lrnfanriliar infbrmation, or facts the children rnight rrot be expecteci to renremLrer cotrld be u'ritten on the board to be referrecl to ls appr-opriate dtrling the exercise. For instance, Suzie's address and telcphone nurnber rnigl.rt be put on tl-re board along rrith the specific date ol' the crinre. and tire esrirnated cost of the T\r and stereo, Suppose, finalli, that the children tire to *'rite their responses to each of the follorvins questions put b\ the teacher (actine in the role of rhe police officer): (X)Now, \ls Doc, I Lrnclcrstand that someone broke into !our apartmenr took some thin!rs. \\hat is votrr acldress herc? \\hat is vour plronc ntrmbcr? Exactlr rr'lrcrr di,l lhis lrapperr? Ok,tr. H,,rv dirl rhc pcr.,,n get ini

\\'hat thirres dirl he take ? I see. And nhorrt horr rluch *otrld vorr sa\ the stolen items rvere rr'or-th? Dicl he take anvrlrinc else ? Or was An!one harmed?

an(1

92

Teach ing al/ the children to read

:--':otice that the questions asked in \( (and in all tests or teaching actilities that conform to the the o ry re commended here) must fo c us on the actual facts of the discourse actilitl, We do not ask about al1\thing unrelated to I\hat aClllalh' occurred, We do not il1\'ite stttdents to im'ellt sentences Or am other sort of discourse detached from the facts of experience, We might I\ant 10 include actil"ities in the curriculum and in our testing \\here children gile their o\\'n addre'ises or ,\Tite their o\\'n stories , but in ~tll ,uch cases. the setting-u p of such discourse el'ents \I'cHlid pertain to parti c ular \\'ell-determined facts, ,-\Iso, the questions to elicit the l'ariOlts ~ [dclr('"e, ,[!ld phone numbers of the sel"eral children. for instanc e , I\"()ltid be differcilt in critical respects from the ones asked here, In exercise X, quntion ( I ) COllet'rns the addre~s \I"here the hurglary occurred, ,lust al1\ ,lid arldre ,'iS or telephone numher I\ill not do, Similarh', question C~ ) concems \\'hen the burglar. occurred, Th e refore. I ~ October 1985 , is not ,I correct ans\\'el". .\Ild so nn. throughout. It is onll' because there are ,Oll1e det e rll1ined faClS upon \\'hich the discourse is hased that the discour-ie itself has al1\ particular llleaning and that meaning is the onh sufficient basis for the lalid scabhility (or scorabilit\' in other cases) of the task in question, Because the task pertains to relati\"{:,h \\'ell-determined facts, responses ({f 1/ bf scaled, That IS. response~ can be judged for different degrees of accurac\, to am ck'iired lelel of'detail. For instance , all the questions might be judged Oil ~l single fil'e-poil1t scale \\'here, sal', 'S' means that all the requested information I\as prm'ided and in a correct form, '4' means that nearh" all the Cacts were cOIl\"e\'ed in a correct or nearly correct form, '3' means that most of the facts \I'ere cOll\'el'ed but with se\'eral errors in form, '2' means that some of the facts I\'cre cOIl\'eved but with some multiple errors. and ' I ' means that little or none of the facts were correctly conveyed, Such a scale cOlild be applied in a rough and readv manner to the questions under numeral X, or almost anI' of those listed above under IX, Or, the formal aspects of the discourse task (e,g, pronunciation or legibilitv, spelling. punctuation, word-usage , grammar) might be separated, to some degree , from each other and from the comprehensibility and factual accuracy with which the examinee con\'eys the intended or desired information concerning the known facts, (Remme the known facts, incidentally, and any performance whatever will become difficult or impossible to scale or to score in any wav,) Distinct scales for factual content and formal accuracy (e,g. spelling, punctuation, vocabularv) of the spoken or written discourse forms can easily be imagined along the lines of the general scale just exemplified, However, in almost anI' classroom situation separate ratings assigned to all of the concei\'able components of any given discourse task would make no sense at all. The trouble is that there are too many components. Imagine

a~

~~

at

ill r ~l

II'

p: sc !

I

T ir 11

\

I

I

L II f( c'

\, r' r \"

c

c r

c

J

Exampies of testing procedures and activities 111,111 IDI' , " 11',\1 hill\!; ,II lililil" il" I', ,) 11111'1 f,, ( lI' " " Ihl' ,1('fll~iI ,I-. ,till'lll ,111IIh ll1\!; Illl r"!; Ill'd III Il ld"11I, III illl" 111 ,,'111 ('1]( l', " I 1\ lilt' LICl' oj t:':\p "l il'IIl'I' \\ (, ' II T it'1I111I1I ,I IHI ill (1 111 Il"IJII~ I, rilt' Iill'il ' ""11 '1

11 1'1' ( "'('111' '\()llld P"II,tlll I" (1I Il, ril' lh III ,'Ilcil Ihl' \ ,111111" tI (iJlldl't:'II, I'll ' ill'I, lIl(" '\l)III f! 'II" ,I ,I-.l'd Ill'I 'C' , III ,'''l'I CI'(' \: , il l' hlll ' ~hJ'\ ' Ill' IIITl'rI ,11"1 dill 1[ rill "il Illl 1.11'1, , qlll"l i,)J1 (:l) ',' fl 'l l' , 1:.'()III,J'CI ICJ!-i ,- l,i'lll>\ II i, 1)111< h(,( IIII'(' t il e ll' lin' 'll)lI l'(' i, h'\'l'd lildl lill' eli,11]( 1 Ih;\r Ill('.Jllillg i, lile ""I< " Idhilil\ ill (JlillT C