CLASSROOM Rajeeva L Karandikar Indian Statistical Institute
Multiple-Choice Tests, Negative Marks and an Alternative
7, SJS Sansanwal Marg New Delhi 110 016, India. Email:
[email protected]
I a n a ly se v a r io u s sc h e m e s o f n e g a tiv e m a r k in g fo r te sts c o n sis tin g o f m u ltip le -c h o ic e q u e stio n s a n d p r o p o se a sc h e m e th a t r e d u c e s th e im p a c t o f r a n d o m g u e ssin g . I a lso p r o p o se a n a lte r n a te sty le o f m u ltip le -c h o ic e q u e stio n s, w h e r e e a c h q u e stio n m a y h a v e se v e r a l c o r r e c t a n sw e r s a n d th e c a n d id a te is r e q u ir e d to tic k a ll c o r r e c t a n sw e r s in o r d e r to g e t c re d it. T ests co n sistin g o f m u ltip le-ch o ice q u estio n s a re v ery co m m o n th ese d ay s. T h ey a re co n sid ered b y m a n y to b e objective a s o p p o sed to tra d itio n a l tests w h ich a re co n sid ered to b e su bjective. A b a sic d i± cu lty w ith m u ltip lech o ice q u estio n s is th a t w h en th e a n sw er is w ro n g , w e a re su re th a t th e ca n d id a te d o es n o t k n ow th e a n sw er w h ile if th e a n sw er is co rrect, th e p o ssib ility rem a in s th a t th e ca n d id a te h a s g u essed th e a n sw er w ith o u t rea lly k n ow in g it. It is fo r th is rea so n th a t n eg a tiv e m a rk s a re th o u g h t o f. S o m e p eo p le still d o n o t a g ree w ith th e p h ilo so p h y o f n egative m arks fo r a n in co rrect a n sw er. H ow ev er, it is la rg ely a ccep ted (a n d m o st b eliev e) th a t th is ta k es ca re o f th e p ro b lem o f so m eo n e g ettin g a h ea d o n th e b a sis o f ra n d o m g u essin g . B u t o n ce w e co m e to th e q u estio n o f w h a t sh o u ld b e th e n eg a tiv e m a rk in g sch em e, th ere seem s to b e a lo t o f co n fu sio n . In m o st ex a m in a tio n s w h ere o b jectiv e tests a re u sed , th e ca n d id a tes a re n o t to ld o f th e n eg a tiv e m a rk in g sch em e. O n e a rg u m en t g o es a s fo llow s: \H ow does it m atter as to w hat schem e is u sed, it is the sam e for all!"
Keywords Multiple-choice test, negative marking.
86
In th is a rticle, I a rg u e th a t it d o es m a tter a s to w h a t sch em e is u sed a n d ev en if th e theoretically correct sch em e is u sed , ca n d id a tes ca n g et a d v a n ta g e u sin g ra n d o m ised g u essin g . I w ill p ro p o se a n a ltern a tiv e n eg a tiv e m a rk in g
RESONANCE March 2006
CLASSROOM
sch em e, w h ich im p rov es m a tters, so m ew h a t. M u l- tip lech o ice tests a re u n av o id a b le in situ a tio n s w h ere la rg e n u m b ers o f ca n d id a tes a p p ea r a n d w e n eed co m p u terized ch eck in g o f a n sw er scrip ts to h av e a q u ick resu lt. F o r d ea lin g w ith th is situ a tio n , I w ill a lso p ro p o se a n a ltern a tiv e sty le o f q u estio n { a n sw ers th a t red u ce th e in ° u en ce o f ra n d o m ised g u essin g ev en fu rth er. T o m a k e o u r d iscu ssio n co n crete, let u s co n sid er a m u ltip le-ch o ice test w ith 1 0 0 q u estio n s. E a ch q u estio n h a s 4 a ltern a tiv es o n e o f w h ich is th e co rrect a n sw er. S u p p o se th is is a screen in g test w ith 4 5 a s th e cu t-o ® , w ith ea ch co rrect a n sw er g ettin g o n e m a rk . E v ery o n e sco rin g a b ov e 4 5 q u a li¯ es fo r th e n ex t ro u n d , w h ich m ay b e a n in terv iew o r a tra d itio n a l sh o rt a n sw er test. W h ile so m e ex a m in a tio n s in th is scen a rio u se ¡ 14 a s th e sco re fo r a w ro n g a n sw er, so m e u se th e theoretically correct ¡ 13 a s th e sco re fo r a w ro n g a n sw er (th is sch em e h a s th e p ro p erty th a t th e ex p ected sco re o f a ca n d id a te w h o a n sw ers a ll q u estio n s b y ra n d o m ly ch o o sin g a n a ltern a tiv e is zero { in th is sen se, it is th eo retica lly co rrect). W e w ill ca ll sch em e I th e m a rk in g sch em e w ith n o n eg a tiv e m a rk in g , sch em e II a s th e sch em e w ith ¡ 14 a s th e sco re fo r a w ro n g a n sw er a n d sch em e III a s th e sch em e w ith ¡ 13 a s th e sco re fo r a w ro n g a n sw er. In ea ch o f th ese, th e co rrect a n sw er g ets a sco re o f 1 . L et u s co n sid er a ca n d id a te, n a m ed A in th e seq u el, w h o k n ow s a n sw ers to 4 0 q u estio n s w h ile h e ra n d o m ly g u esses th e a n sw ers to th e rem a in in g 6 0 . In th is ca se, th e ex p ected sco re o f A is 5 5 u n d er sch em e I; 4 3 .7 5 in sch em e II a n d 4 0 in sch em e III. A little re° ectio n w ill co n v in ce u s th a t it is n o t th e ex p ected sco re b u t th e probability o f A sco rin g 4 5 o r m o re th a t is releva n t { if th ere a re a la rg e n u m b er o f ca n d id a tes a p p ea rin g in a n ex a m in a tio n a n d su p p o se th ere a re 1 0 0 0 ca n d id a tes lik e A , w h a t is releva n t is n o t th e av era g e sco re o f th ese 1 0 0 0 ca n d id a tes (w h ich w ill b e clo se to 5 5 , 4 3 .7 5 a n d
RESONANCE March 2006
87
CLASSROOM
4 0 u n d er th e th ree sch em es), b u t how m an y o u t o f th ese w ill h av e a sco re a b ov e 4 5 . In th is situ a tio n , th e n u m b er o f co rrect a n sw ers X A (o u t o f 6 0 ra n d o m ly g u essed ) b y A h a s a b in o m ia l d istrib u tio n w ith n = 6 0 a n d p = 0 :2 5 a n d so th e p ro b a b ility o f n et sco re b ein g ov er 4 5 ca n b e co m p u ted : it is P (X A ¸ 5 ) u n d er sch em e I; P (X A ¸ 1 6 ) u n d er sch em e II a n d P (X A ¸ 1 9 ) u n d er sch em e III. T h ese p ro b a b ilities a re 1 , 0 .4 3 a n d 0 .1 5 . T h is sh ow s th a t w h ile w e m u st h av e n eg a tiv e m a rk s, u n d er th e sch em e II, a ca n d id a te ca n m a k e u p a g a p o f 5 m a rk s w ith a b o u t 4 3 % p ro b ab ility. E v en w ith sch em e III, th e p ro b a b ility is n ea rly 1 5 % . S o b y g u essin g , a g a p o f 5 m a rk s h a s b een m a d e u p w ith h ig h p ro b a b ility. L et u s see th e risk in v o lv ed in ra n d o m ised g u essin g fo r a ca n d id a te B w h o h a s 5 ex tra m a rk s, i.e. k n ow s a n sw ers to 5 0 q u estio n s a n d a n sw ers th e rem a in in g b y g u essin g . In th is ca se th e p ro b a b ilities a re 1 , 0 .9 9 a n d 0 .9 1 resp ectiv ely. S o u n d er sch em e II th ere is 4 3 % ch a n ce o f g a in in g 5 m a rk s a n d a b o u t 1 % ch a n ce o f lo sin g 5 m a rk s w h ile u n d er sch em e III, th ere is 1 5 % ch a n ce o f g a in in g 5 m a rk s a n d a b o u t 9 % ch a n ce o f lo sin g 5 m a rk s. S u p p o se th ere is a ca n d id a te Z w h o a n sw ers 4 4 q u estio n s co rrectly a n d d o es n o t g u ess a n y a n sw er. In th is ca se, th e ch a n ces th a t A w ill sco re m o re th a n Z d o es d ep en d u p o n th e sch em e { u n d er sch em e I it is 1 w h ile u n d er II it is 0 .4 3 a n d u n d er sch em e III it is 0 .1 5 . S o th e n eg a tiv e m a rk in g sch em e d o es h av e a n im p a ct u p o n th e o rd erin g o f th e ca n d id a tes. P eo p le w h o h av e p a rticip a ted in settin g q u estio n s fo r su ch tests w ill a g ree th a t ¯ n d in g 3 cred ib le a ltern a tiv es is o ften ra th er d i± cu lt. In m a n y q u estio n s, o n e o f th e a ltern a tiv es is su ch th a t it ca n b e ru led o u t w ith little o r n o k n ow led g e a n d h en ce w e sh o u ld n o t g iv e cred it to a ca n d id a te w h o h a s ru led o u t on e o f th e a n sw ers in m a n y q u estio n s. S o let u s co n sid er a ca n d id a te C w h o a n sw ers 4 0 q u estio n s b a sed o n h is kn ow ledge. In 5 0 q u estio n s h e
88
RESONANCE March 2006
CLASSROOM
ru les o u t co rrectly o n e a ltern a tiv e a n d ch o o ses o n e o u t o f th ree a ltern a tiv es ra n d o m ly a n d rem a in in g 1 0 q u estio n s h e h a s n o clu e a n d ch o o ses a n a n sw er ra n d o m ly o u t o f 4 a ltern a tiv es. A s w e h av e a rg u ed a b ov e, w e w o u ld n o t lik e to select h im fo r n ex t ro u n d , a s w e d o n o t w ish to g iv e cred it to h is h av in g ru led o u t o n e a ltern a tiv e in 5 0 q u estio n s. In th is ca se, th e n u m b er o f co rrect a n sw ers X C o u t o f th e o n es C h a s g u essed n o lo n g er h a s a b in o m ia l d istrib u tio n b u t is a su m o f tw o b in o m ia l d istrib u tio n s w ith p a ra m eters n = 5 0 , p = 0 :3 3 3 3 3 a n d n = 1 0 , p = 0 :2 5 resp ectiv ely. T h e d istrib u tio n o f th e su m ca n b e w o rk ed o u t b u t it is m u ch ea sier to co m p u te th e p ro b a b ility th a t X C is a t lea st 4 5 v ia sim u la tio n (see [1 ]). T h ey a re 1 u n d er sch em e I, 0 .8 5 u n d er sch em e II a n d 0 .5 7 u n d er sch em e III. L et u s a lso ex a m in e a ca n d id a te D , w h o k n ow s a n sw ers to 5 0 q u estio n s, in 4 0 q u estio n s h e ru les o u t co rrectly o n e a ltern a tiv e a n d ch o o ses o n e o u t o f th ree a ltern a tiv es ra n d o m ly a n d rem a in in g 1 0 q u estio n s h e h a s n o clu e a n d ch o o ses a n a n sw er ra n d o m ly o u t o f 4 a ltern a tiv es. F o r D , th e p ro b a b ility th a t X D ex ceed s 4 5 a re 1 ,1 a n d 0 .9 9 u n d er th e th ree sch em es resp ectiv ely. T h u s, if w e th in k th a t in m o st q u estio n s, o n e o p tio n ca n b e ru led o u t w ith o u t m u ch k n ow led g e o f th e su b ject m a tter, o r to p u t it d i® eren tly, w e d o n o t w ish to g iv e p a rtia l cred it to a ca n d id a te fo r ru lin g o u t o n e a ltern a tiv e o u t o f 4 ; th en w e sh o u ld n o t u se sch em e III. F o r a ca n d id a te ca n h av e a stra teg y o f ra n d o m ly g u essin g o u t o f 3 a ltern a tiv es in a ll q u estio n s th a t th e ca n d id a te d o es n o t k n ow th e co rrect a n sw er b u t ca n ru le o u t o n e a ltern a tiv e. W ith th is stra teg y, th e ca n d id a te ca n m a k e u p a sh o rtfa ll o f 5 m a rk s w ith p ro b a b ility 0 .5 7 u n d er sch em e III b u t th e p ro b a b ility o f lo sin g 5 m a rk s is o n ly 0 .0 1 . W e ca n th in k o f sch em e III a s fo llow s: A ra n d o m g u ess o u t o f 4 d o es n o t g et a n y cred it (o n th e av era g e) w h ile ru lin g o u t o n e a ltern a tiv e co rrectly a n d g u essin g o u t o f th e rem a in in g th ree g ets a cred it o f 19 o n th e a v era g e
RESONANCE March 2006
89
CLASSROOM
( 13 (1 ¡ 13 ¡ 13 )) w h ile ru lin g o u t 2 o p tio n s co rrectly a n d g u essin g o u t o f th e rem a in in g tw o g ets a cred it o f 12 (1 ¡ 1 ) = 13 o n th e av era g e. 3 In th is lig h t, co n sisten t w ith th e v iew th a t ru lin g o u t o n e a ltern a tiv e sh o u ld n o t g et a n y cred it, w h ile ru lin g o u t tw o a ltern a tiv es sh o u ld g et so m e p a rtia l cred it, w e ca n co n sid er a sch em e IV w h ere a n in co rrect a n sw er g ets a sco re ¡ 12 . U n d er th is sch em e, a stu d en t w ill b e p en a lised if h e ra n d o m ly g u esses o u t o f fo u r a ltern a tiv es, w h ile g ets a sco re o f 14 o n th e av era g e fo r g u essin g o u t o f tw o a ltern a tiv es a fter ru lin g o u t (co rrectly ) tw o a ltern a tiv es. T able 1 g iv es th e p ro b a b ility o f sco rin g 4 5 o r m o re u n d er th e th ree sch em es w ith ¡ 12 ,¡ 13 a n d ¡ 14 m a rk s fo r a w ro n g a n sw er. H ere, (a : b : c : d ) rep resen ts a stu d en t w h o h a s a n sw ered co rrectly a q u estio n s k n ow in g ly, h a s g u essed o u t o f tw o a ltern a tiv es in b q u estio n s, g u essed o u t o f th ree a ltern a tiv es in c q u estio n s a n d g u essed o u t o f fo u r a ltern a tiv es in d q u estio n s.
Table 1.
90
(a : b : c : d ) (3 5 :0 :6 5 :0 ) (3 5 :5 0 :1 0 :0 ) (3 5 :6 0 :5 :0 ) (3 8 :0 :0 :6 2 ) (3 8 :0 :6 0 :2 ) (3 8 :4 0 :4 :0 ) (3 8 :4 8 :4 :0 ) (3 8 :4 8 :4 :0 ) (3 8 :5 8 :4 :0 ) (4 0 :0 :0 :6 0 ) (4 0 :0 :5 0 :1 0 ) (4 0 :2 0 :1 0 :0 ) (4 0 :2 5 :1 0 :0 ) (4 0 :3 0 :0 :0 ) (5 0 :0 :0 :5 0 ) (5 0 :0 :4 0 :1 0 )
¡ 12 0 .0 3 8 0 .6 8 0 .7 8 0 .0 0 26 0 .0 9 0 .7 1 0 .8 6 0 .8 6 0 .8 9 0 .0 0 74 0 .1 2 0 .4 7 0 .6 8 0 .7 1 0 .3 6 0 .7 6
¡ 13 0 .3 1 0 .9 4 0 .9 8 0 .0 7 4 0 .4 9 0 .9 3 0 .9 7 0 .9 7 0 .9 9 0 .1 5 0 .5 7 0 .7 5 0 .8 7 0 .9 0 .9 1 0 .9 9
¡ 14 0 .6 1 0 .9 9 1 0 .2 7 0 .7 9 0 .9 8 1 1 1 0 .4 3 0 .8 5 0 .9 2 0 .9 7 0 .9 8 0 .9 9 1
RESONANCE March 2006
CLASSROOM
W e stro n g ly reco m m en d th e sch em e w h ere a w ro n g a n sw er g ets ¡ 12 . T h e ta b le g iv en a b ov e sh ow s th a t u n d er th is sch em e, a ca n d id a te ca n h o p e to g a in sig n i¯ ca n tly o n ly w h en h e/ sh e ca n ru le o u t tw o a ltern a tiv es o n a g iv en q u estio n a n d th en g u ess ra n d o m ly o u t o f th e rem a in in g tw o . H ere, w e ca n n o t elim in a te th e e® ect o f ra n d o m g u essin g , b u t th e e® ect is m in im ized . T h is b rin g s u s to a n o th er q u estio n . C o n sid er a situ a tio n w h ere m u ltip le-ch o ice test is b ein g u sed , a n d th e test is co n d u cted say tw ice o r th rice a y ea r a n d ca n d id a tes ca n a p p ea r a g a in a n d a g a in . T h ere a re m a n y su ch situ a tio n s. S o in a b o u t a y ea r, a ca n d id a te m ay h av e sev era l ch a n ces, say 3 . T h is d ra stica lly p u sh es u p th e ch a n ces o f a ca n d id a te g ettin g selected in o n e o f th e tests. S o let u s co n sid er a test, w h ich is to b e u sed a s a ¯ n a l selectio n test (a n d so th e cu t-o ® is h ig h , say 6 5 ) a n d a ca n d id a te ca n a p p ea r 3 tim es in th e ex a m . T h e ta b le b elow g iv es p ro b a b ility o f selectio n (sco rin g 6 5 o r m o re) in u p to 3 a ttem p ts. A s in th e ea rlier ca se, (a : b : c : d ) rep resen ts a stu d en t w h o h a s a n sw ered co rrectly a q u estio n s k n o w in g ly, h a s g u essed o u t o f tw o a ltern a tiv es in b q u estio n s, g u essed o u t o f th ree a ltern a tiv es in c q u estio n s a n d g u essed o u t o f fo u r a ltern a tiv es in d q u estio n s (in ea ch o f h is a ttem p ts). T able 2 g iv es th e p ro b a b ility o f a ca n d id a te sco rin g 6 5 o r m o re in a t lea st o n e o f th e th ree a ttem p ts. W e ca n see th a t w ith ¡ 13 a s m a rk s fo r a n in co rrect a n sw er, sh o rtfa ll o f 1 0 m a rk s ca n b e m a d e-u p w ith a h ig h p ro b a b ility, a n d if a ca n d id a te ca n ru le o u t tw o a ltern a tiv es in m a n y q u estio n s, a sh o rtfa ll o f 1 0 ca n b e m a d e u p w ith a h ig h p ro b a b ility ev en w ith ¡ 12 a s m a rk s fo r a n in co rrect a n sw er. L et u s ex a m in e a n a ltern a tiv e. R eca ll th a t th e m a in rea so n in fav o u r o f u sin g m u ltip le-ch o ice q u estio n s is th a t th e eva lu a tio n ca n b e d o n e b y sca n n in g th e a n sw er sh eet a n d th en co m p u terized m a rk in g . T h is en a b les th e
RESONANCE March 2006
91
CLASSROOM
Table 2.
(a : b : c : d ) ¡ 12 (5 0 :1 0 :4 0 :0 ) 0 .0 26 (5 0 :2 0 :3 0 :0 ) 0 .0 86 (5 0 :3 0 :2 0 :0 ) 0 .22 (5 0 :4 0 :1 0 :0 ) 0 .45 (5 0 :5 0 :0 :0 ) 0 .71 (5 5 :0 :4 0 :5 ) 0 .0 46 (5 5 :5 :4 0 :0 ) 0 .11 (5 5 :1 0 :3 5 :0 ) 0 .19 (5 5 :1 5 :3 0 :0 ) 0 .29 (5 5 :2 0 :2 5 :0 ) 0 .42 (5 5 :2 5 :2 0 :0 ) 0 .56 (5 5 :3 0 :1 5 :0 ) 0 .69 (6 0 :0 :4 0 :0 ) 0 .37 (6 0 :5 :3 5 :0 ) 0 .52 (6 0 :1 0 :3 0 :0 ) 0 .67 (6 0 :1 5 :2 5 :0 ) 0 .79
¡ 13 0 .1 8 0 .3 9 0 .6 5 0 .8 6 0 .9 6 0 .2 9 0 .4 9 0 .6 3 0 .7 6 0 .8 6 0 .9 2 0 .9 7 0 .8 5 0 .9 3 0 .9 7 0 .9 9
¡ 14 0 .4 3 0 .7 0 .8 9 0 .9 7 1 0 .6 1 0 .8 0 .8 9 0 .9 4 0 .9 8 0 .9 9 1 0 .9 8 0 .9 9 1 1
a g en cy co n d u ctin g ex a m in a tion to p ro cess la rg e n u m b er o f a n sw er scrip ts in sh o rt tim e. O n e a ltern a tiv e to red u ce th e e® ect o f ra n d o m ised g u essin g is to in crea se th e n u m b er o f a ltern a tiv es fo r ea ch q u estio n , b u t a s a rg u ed a b ov e, ¯ n d in g cred ib le a ltern ativ es is n o t ea sy. A n o th er o p tio n is to h av e q u estio n s fo r w h ich sev era l o p tio n s m ay b e co rrect a n d th e ca n d id a tes a re req u ired to m a rk a ll co rrect a ltern a tiv es. S o w h ile w e m ay still h av e 4 a ltern a tiv es, th e e® ectiv e a ltern a tiv es rises to 1 5 (w e sh o u ld k eep a t lea st o n e co rrect a n sw er so th a t w e ca n d i® eren tia te b etw een a ttem p ted a n d u n a n sw ered ). T h en w e ca n d o aw ay w ith n eg a tiv e m a rk in g . L et m e ex p la in th is b y a n ex a m p le. T h e ex a m p le is a q u estio n in m a th em a tics. S u b sets o f rea l n u m b ers ca n h av e m a n y p ro p erties su ch a s b ein g open a n d / o r con n ected a n d / o r bou n ded a n d / o r den se. C o n sid er th e q u estio n : T ick a ll co rrect a n sw ers: T h e set o f rea l n u m b ers f x 2 R : (sin (x ))2 < 1 g is (a ) O p en ; (b ) C o n n ected ; c) B o u n d ed ; (d ) D en se.
92
RESONANCE March 2006
CLASSROOM
H ere th e a n sw er is (a ) a n d (d ). T h e p o in t is if a ca n d id a te is clu eless a n d is g o in g to tick a t ra n d o m , h e h a s to co n sid er a ll p o ssib ilities (1 5 o f th em ) a n d ch a n ce o f tick in g th e co rrect o n e is o n ly 115 . C o n sid er a n o th er q u estio n : a certa in n u m erica l q u estio n in p h y sics o r ch em istry, w h o se a n sw er is say 3 .2 3 . T h e q u estio n co u ld b e: T ick a ll co rrect a n sw ers: (a ) b etw een 1 .1 a n d 2 .8 ; (b ) b etw een 1 .9 a n d 3 .4 ; c) b etw een 2 .5 a n d 3 .7 6 ; d ) b etw een 3 .1 a n d 3 .9 . H ere th e ca n d id a te sh o u ld tick th ree o p tio n s (b ), (c) a n d (d ). In a g iv en test, th e ex a m in ers sh o u ld ta k e ca re th a t th ere a re en o u g h q u estio n s w ith 1 , 2 , 3 a s w ell a s a ll 4 co rrect a n sw ers. T h is w ay, a g u essin g stra teg y w o u ld n o t g iv e a n a d va n ta g e to a ca n d id a te, a s e® ectiv ely th ere a re 1 5 o p tio n s. H ere, w e m ay a lso co n sid er a ltern a te sch em es, w h ere ea ch o p tio n m ay h av e a sso cia ted w ith it a n u m b er x { th e m a rk s a stu d en t g ets fo r tick in g th a t o p tio n (x p o sitiv e fo r co rrect ch o ices a n d n eg a tiv e fo r in co rrect ch o ices). T h e m a rk s fo r co rrect ch o ices sh o u ld a d d to 1 o n ea ch q u estio n . F o r ex a m p le, fo r th e q u estio n g iv en a b ov e w ith (a ), (d ) a s co rrect a n sw ers, th e m a rk s co u ld b e 12 fo r (a ), ¡ 12 fo r (b ), ¡ 12 fo r (c) a n d 12 fo r (d ). T h e q u estio n s co u ld b e su ch th a t th ey req u ire so m e a n a ly sis o r co m p u ta tio n ra th er th a n test m em o ry. It m ay b e n o ted th a t th e q u estio n s in th is n ew sty le w o u ld ty p ica lly req u ire m o re tim e fo r a ca n d id a te to u n d ersta n d a n d a n sw er. T h u s, th e ca n d id a tes sh o u ld b e g iv en m o re tim e to a n sw er. T h is w ay, th e test w ill b e a b le to g iv e b etter d iscrim in a tio n b a sed o n k n ow led g e b a se o f th e ca n d id a tes th a n th e p resen t sty le o f m u ltip le-ch o ice tests, w h ich o ften en d u p b ein g a test o f m em o ry a n d sp eed .
RESONANCE March 2006
Suggested Reading [1] Sudhakar Kunte, Technique of statistical simulation, Resonance, Vol.5, No.4, 2000.
93
