Online Error Detection in Multiprocessor Chips: A ...

2QOLQH(UURU'HWHFWLRQLQ0XOWLSURFHVVRU&KLSV $7HVW6FKHGXOLQJ6WXG\ 0DQROLV.DOLRUDNLV1LNRV)RXWULV'LPLWULV*L]RSRXORV0LKDOLV3VDUDNLV 

'HSWRI,QIRUPDWLFV 7HOHFRPP8QLYHUVLW\RI$WKHQV*UHHFH ^PDQROLVNDOQIRXWULVGJL]RS`#GLXRDJU 

$EVWUDFW²0XOWLFRUHDUFKLWHFWXUHVDUHHPSOR\HGLQWKHPDMRULW\ RIFRPSXWLQJGRPDLQVJHQHUDOSXUSRVHPLFURSURFHVVRUVDVZHOO DV VSHFLDOL]HG KLJKSHUIRUPDQFH DUFKLWHFWXUHV VXFK DV QHWZRUN SURFHVVRUV  2QOLQH HUURU GHWHFWLRQ LQ VXFK FKLSV FDQ HPSOR\ HIIHFWLYH WHFKQLTXHV IURP VLQJOH FRUH PLFURSURFHVVRUV KRZHYHU HIIHFWLYH WHVW VFKHGXOLQJ VKRXOG EH HPSOR\HG WR PLQLPL]H WKH RYHUDOO FKLS WHVW H[HFXWLRQ WLPH ZKLFK FDQ VLJQLILFDQWO\ LQFUHDVH GXH WR FRQJHVWLRQ RQ FRPPRQ KDUGZDUH UHVRXUFHV XVHG E\ WKH FRUHV ,Q WKLV SDSHU ZH DQDO\]H WKH PRVW LPSRUWDQW DVSHFWV RI RQOLQH HUURU GHWHFWLRQ DQG VFKHGXOLQJ LQ PXOWLSURFHVVRU FKLSV DQG HYDOXDWH WHVW H[HFXWLRQ WLPH LQ VHYHUDO GLIIHUHQW FRQILJXUDWLRQVRI,QWHO¶V6&&DUFKLWHFWXUH .H\ZRUGV²PXOWLSURFHVVRUFKLSVQHWZRUNSURFHVVLQJXQLWVRQOLQH HUURUGHWHFWLRQVRIWZDUHEDVHGWHVWLQJWHVWVFKHGXOLQJ

,

,1752'8&7,21

7KH DGYDQFHV LQ FLUFXLW PDQXIDFWXULQJ WHFKQRORJ\ GXULQJ WKH ODVWGHFDGHKDVXVKHUHGWKHSURFHVVRULQGXVWU\LQWKHPXOWLFRUH HUD0DQ\SURFHVVLQJFRUHVDUHQRZLQWHJUDWHGLQERWKJHQHUDO SXUSRVHPLFURSURFHVVRUV>@DQGVSHFLDOL]HG6\VWHPVRQFKLS 6R&V >@VXFKDV QHWZRUNSURFHVVLQJ XQLWV 138V >@>@ >@ +RZHYHU DV GHYLFH JHRPHWULHV DUH VKULQNLQJ DQG SURFHVVLQJ FRUHV DUH SUROLIHUDWLQJ LQ WKH XSFRPLQJ FRPSOH[ PXOWLSURFHVVRUFKLSVQHZUHOLDELOLW\LVVXHVDULVH $OWKRXJK HUURU GHWHFWLRQ KDV EHHQ H[WHQVLYHO\ VWXGLHG LQ WKHSDVWWKHSRUWLQJRIZHOOHVWDEOLVKHGWHFKQLTXHVIRUXQLFRUH SURFHVVRU WHVWLQJ WR FRPSOH[ PXOWLSURFHVVRU FKLSV LQWURGXFHV VLJQLILFDQW WHVW FKDOOHQJHV )LUVW WKH LQWHJUDWLRQ RI PXOWLSOH SURFHVVRUVLQD VLQJOH FKLS FRPHV DOVRZLWK WKHLQWHJUDWLRQRI VHYHUDO W\SHV RI PHPRULHV 5$0V ),)2V &$0V  DV ZHOO DV RQFKLS FRPPXQLFDWLRQ FRPSRQHQWV WKDW DOO QHHG WR EH FRQVLGHUHG LQ WKH V\VWHP¶V HUURU GHWHFWLRQ VFKHGXOH 6HFRQG WKH HUURU GHWHFWLRQ SURFHVV GXUDWLRQ PD\ H[FHVVLYHO\ HVFDODWH ZLWKWKHQXPEHURISURFHVVRUFRUHVGXHWRFRUHVFRQWHQWLRQIRU RQFKLS VKDUHG UHVRXUFHV 7KLUG SRZHU FRQVXPSWLRQ GXULQJ WHVWLQJPD\JREH\RQGWKHDYHUDJH SRZHUFRQVXPSWLRQRIWKH FKLS RU HYHQ H[FHVVLYH KHDW GLVVLSDWLRQ PD\ EH REVHUYHG ORFDOO\GXHWRWKHKLJKVZLWFKLQJDFWLYLW\FRQFXUUHGWRPXOWLSOH SURFHVVRUFRUHVEHLQJXQGHUWHVW :H VWXG\ RQOLQH HUURU GHWHFWLRQ LQ PXOWLSURFHVVRU FKLSV DSSO\LQJIXQFWLRQDOVRIWZDUHEDVHG WHVWVGXULQJQRUPDOPRGH RI RSHUDWLRQ 2XU VWXG\ LV DSSOLFDEOH WR JHQHUDOSXUSRVH PLFURSURFHVVRUV DV ZHOO DV WR VSHFLDOL]HG PDQ\FRUH DUFKLWHFWXUHV VXFK DV QHWZRUN SURFHVVLQJ XQLWV 138V  :H GHILQH WKH FKDOOHQJHV RI WKLV UHVHDUFK GLUHFWLRQ IRU WKH WHVW UHTXLUHPHQWV DQG SDUWLFXODU DUFKLWHFWXUDO IHDWXUHV RI VXFK FRPSRQHQWV 7KH HPSOR\PHQW RI IXQFWLRQDO PHWKRGV LQ PLFURSURFHVVRUWHVWLQJKDVEHHQUHYLWDOL]HGWKHODVWGHFDGH>@

c 2013 IEEE 978-1-4799-0662-8 





'HSWRI,QIRUPDWLFV8QLYHUVLW\RI3LUDHXV*UHHFH PSVDUDN#XQLSLJU

DQG WRGD\ WKH\ DUH XVHG WR GHWHFW HUURUV DW GLIIHUHQW VWDJHV RI WKH SURFHVVRU OLIHWLPH +HUH ZH DGGUHVV WKH SUREOHP RI LQFUHDVHG WHVW DSSOLFDWLRQ WLPH LQ FRPSOH[ PDQ\FRUH SURFHVVRUV VXFK DV 138V  DQG GLVFXVV IXQFWLRQDO WHVW VFKHGXOLQJ DV D PHDQV WR PLQLPL]H WHVW DSSOLFDWLRQ WLPH :H DQDO\]H WKH LPSRUWDQFH DQG WKH DLPV RI WHVW VFKHGXOLQJ LQ DQ RQOLQHHUURU GHWHFWLRQVFKHPH IRU PXOWLSURFHVVRUVDQG GLVFXVV DOOFKLSSDUDPHWHUVQHHGWREHFRQVLGHUHGLQWHVWVFKHGXOLQJ ,, 5(/$7(':25. 5HFHQWDSSURDFKHV>@>@>@KDYHVWXGLHGWKHDSSOLFDWLRQRI VRIWZDUHEDVHGWHVWLQJLQPXOWLSURFHVVRUFKLSVIRFXVLQJRQWHVW WLPH UHGXFWLRQ ,Q >@ $SRVWRODNLV HW DO FRQVLGHU WKH DSSOLFDWLRQ RI VRIWZDUHEDVHG WHVWLQJ RQ V\PPHWULF VKDUHG PHPRU\ PXOWLSURFHVVRUV 603  DQG SURSRVH D PHWKRGRORJ\ WKDW UHGXFHV EXV FRQWHQWLRQV DQG GDWD FDFKH LQYDOLGDWLRQV LQ RUGHUWRUHGXFHWHVWH[HFXWLRQWLPH7KHPHWKRGRORJ\KDVEHHQ DSSOLHGWRYDULRXV603EHQFKPDUNVZLWKXSWRFRUHV,Q>@ )RXWULV HW DO H[SORLW ERWK FRUH DQG WKUHDGOHYHO H[HFXWLRQ SDUDOOHOLVP DQG VFKHGXOH WKH WHVW WKUHDGV LQWR WKH SURFHVVRU KDUGZDUHWKUHDGVWRUHGXFHWKHWHVWH[HFXWLRQWLPHDQGLQFUHDVH WKH IDXOW FRYHUDJH 7KH\ KDYH GHPRQVWUDWHG WKH PHWKRGRORJ\ LQ D PXOWLWKUHDG SURFHVVRU DUFKLWHFWXUH FRQVLVWLQJ RI  FRUHV DQGWKUHDGV,Q>@6NLWVDVHWDOSURSRVHVHOHFWLYHVRIWZDUH EDVHG WHVWLQJ DV D PHDQV WR UHGXFH WHVW WLPH 7KH VHOHFWLYH WHVWLQJ PHWKRGRORJ\ PRQLWRUV V\VWHP DFWLYLW\ DQG WHVWV RQO\ WKH VWUDLQHG PRGXOHV ZKLOH XQGHUXWLOL]HG PRGXOHV DUH RQO\ VSRUDGLFDOO\WHVWHG7KHPHWKRGRORJ\KDVEHHQGHPRQVWUDWHGLQ DFRUHPXOWLSURFHVVRUFRQQHFWHGLQD['PHVK ,,, 7(676&+('8/,1*,668(6 :HILUVWGLVFXVVWKHFRQFHSWRIIXQFWLRQDORQOLQHHUURUGHWHFWLRQ LQPXOWLSURFHVVRUFKLSVDQGKLJKOLJKWWKHPDLQREMHFWLYHVRID WHVW VFKHGXOLQJ DSSURDFK 1H[W ZH H[DPLQH DOO V\VWHP SDUDPHWHUV PHPRU\ KLHUDUFK\ LQWHUFRQQHFWLRQ QHWZRUN IUHTXHQF\ GRPDLQV  WKDW DIIHFW WHVW H[HFXWLRQ WLPH DQG WKXV QHHGWREHFRQVLGHUHGE\DWHVWVFKHGXOLQJDSSURDFK )RU WKH DQDO\VLV RI WHVW VFKHGXOLQJ LVVXHV DQG WKH DVVHVVPHQW RI V\VWHP SHUIRUPDQFH GXULQJ WKH RQOLQH HUURU GHWHFWLRQSURFHVVZHXVHDVDGHPRQVWUDWLRQYHKLFOHDVWDWHRI WKHDUW H[SHULPHQWDO PDQ\FRUH SURFHVVRU ,QWHO¶V 6LQJOH&KLS &ORXG FRPSXWHU 6&&  >@ >@ 7KH 6&& FKLS LV D UHVHDUFK PLFURSURFHVVRUFUHDWHGE\,QWHO/DEVDQGFRQWDLQV3HQWLXP FRUHV7KH6&& DUFKLWHFWXUHLVVKRZQLQ )LJ,W FRQVLVWVRI  WLOHV WZR 3HQWLXP FRUHV SHU WLOH  FRQQHFWHG LQ D [ ' PHVK QHWZRUN VXSSRUWLQJ  *%V ELVHFWLRQ EDQGZLGWK  LQWHJUDWHG ''5 PHPRU\ FRQWUROOHUV 0&  DQG KDUGZDUH VXSSRUW IRU PHVVDJH SDVVLQJ 7KH 6&& SURFHVVLQJ FRUHV DUH DVVLJQHGLQPXOWLSOHIUHTXHQF\YROWDJHDQGPHPRU\GRPDLQV

169

/

3&

7*

/

0HVK 3&

03%

7LOH

7LOH

7LOH

7LOH





0&

'5$0

0&



0&

'5$0

'5$0

0&

'5$0



)UHTXHQF\ GRPDLQ 9ROWDJHGRPDLQ



0HPRU\GRPDLQ

)LJ ,QWHO¶VPDQ\FRUH6&&$UFKLWHFWXUH

$ )XQFWLRQDO2QOLQH(UURU'HWHFWLRQ )XQFWLRQDORQOLQH HUURU GHWHFWLRQ DSSURDFKHVDUHEDVHG RQ WKH DSSOLFDWLRQ RI WHVW SURJUDPV GXULQJ QRUPDO V\VWHP RSHUDWLRQ 7KHUHDUHWZRDOWHUQDWLYHVIRUWKHH[HFXWLRQRIWHVWSURJUDPVDV VKRZQLQ)LJD HDFKSURFHVVRUFRUHUXQVWKHWHVWSURJUDPV LQGLYLGXDOO\GXULQJLWVLGOHSHULRGVDQGE JURXSRISURFHVVRU FRUHVDUHLVRODWHGDQGVHWRXWRIVHUYLFHSHULRGLFDOO\WRUXQWKH WHVW SURJUDPV WKH FDUGLQDOLW\ RI WKH FRUH JURXSV FDQ UDQJH EHWZHHQDQGWKHWRWDOQXPEHURISURFHVVRUFRUHV7KHIRUPHU FDOOHG KHUHDIWHU ³QRQLQWUXVLYH´ DSSURDFK GRHV QRW LQWHUUXSW QRUPDOV\VWHPRSHUDWLRQDQGLVDSSOLHGVSRUDGLFDOO\DFFRUGLQJ WR WKH LGOH SHULRGV RI WKH FRUHV ,W LV HYLGHQW WKDW RYHUORDGHG FRUHVOLNH&RUHLQ)LJHJFRUHVRID138WKDWSURFHVVHV WUDIILFEXUVWV PD\UHPDLQXQWHVWHGIRUDORQJWLPHSHULRG2Q WKH FRQWUDU\ WKH ODWWHU DSSURDFK LQWUXGHV LQWR WKH QRUPDO RSHUDWLRQRIWKHSURFHVVRUWKXVDGYHUVHO\DIIHFWLQJWKHV\VWHP SHUIRUPDQFH EXW JXDUDQWHHV WKH SHULRGLFLW\ RI WKH WHVW H[HFXWLRQERXQGLQJWKHPD[LPXPHUURUGHWHFWLRQODWHQF\ ,QDKRPRJHQHRXVPXOWLSURFHVVRUDUFKLWHFWXUHLGHQWLFDOWHVW SURJUDPV PXVW EH H[HFXWHG E\ DOO SURFHVVRU FRUHV 7KH GHWHFWLRQFDSDELOLWLHVRIWKHWHVWSURJUDPVVKRXOGVSDQRYHUWKH HQWLUH VHW RI RQFKLS FRPSRQHQWV WKH SURFHVVRUV WKHPVHOYHV WKH PHPRULHV LQWHUFRQQHFWLRQ UHODWHG DQG RWKHU LQWHUIDFLQJ FRPSRQHQWV DQGVKRXOGJXDUDQWHHKLJKIDXOWFRYHUDJH &RUH &RUH &RUH « &RUH1

D

WLPH

&RUH &RUH &RUH « &RUH1

E 1RUPDO:RUNORDG

7HVW:RUNORDG

WLPH ,GOH



)LJ D 1RQLQWUXVLYHVSRUDGLFDSSURDFKE ,QWUXVLYHSHULRGLFDSSURDFK

170

% :K\WHVWVFKHGXOLQJLVLPSRUWDQW" )XQFWLRQDO HUURU GHWHFWLRQ DQG GLDJQRVLV LQ PXOWLSURFHVVRU FKLSV IDFHV D PDMRU FKDOOHQJH UHODWHG WR WHVW DSSOLFDWLRQ WLPH ZH DVVXPH WKDW WHVW DSSOLFDWLRQ WLPH LQFOXGHV RQFKLS WHVW JHQHUDWLRQSUHSDUDWLRQ DFWXDO WHVW H[HFXWLRQ DQG RQFKLS UHVSRQVH FROOHFWLRQ  ,Q D FRPSOH[ PXOWLSURFHVVRU FKLS WKH VWUDLJKWIRUZDUG DSSOLFDWLRQ RI NQRZQ WHVW SURJUDPV IRU WKH LQGLYLGXDO FRUHV  PD\ SURGXFH ORQJ GHOD\V GXH WR UHVRXUFH FRQIOLFWVDQG FRQVHTXHQWO\ LQFUHDVHH[FHVVLYHO\ WKHWHVW WLPH 7KLV PDLQO\ KDSSHQV LQ UDQGRPO\ RU QDLYHO\ VFKHGXOHG WHVW SURJUDP H[HFXWLRQ EHFDXVH WKH VDPH WHVW FRGH RU WHVW FRGHV ZLWK VLPLODU H[HFXWLRQ SURILOHV  PD\ UXQ VLPXOWDQHRXVO\ LQ PXOWLSOHFRUHVFDXVLQJDODUJHQXPEHURIFRQIOLFWVLQWKHVKDUHG UHVRXUFHV EXVHV PHPRULHV PHPRU\ FRQWURO XQLWV IXQFWLRQDO XQLWVHWF 7KLVSKHQRPHQRQPD\EHH[DFHUEDWHGLQPXOWLFRUH DUFKLWHFWXUHVVXFKDV138VZKHQWKH\SURFHVVWUDIILFEXUVWV $WHVWVFKHGXOLQJPHWKRGRORJ\DLPVWRUHGXFHWKHGXUDWLRQ RIWKHWHVWSURJUDPH[HFXWLRQ OLPLWLQJWKHFRQWHQWLRQRI FRUHV IRUVKDUHGUHVRXUFHV,QWKHFDVHRIWKHQRQLQWUXVLYHDSSURDFK )LJD WKLVWLPHUHGXFWLRQZLOOPDNH IHDVLEOHWKH H[HFXWLRQ RI WHVW SURJUDPV LQ VKRUWHU LGOH SHULRGV LQFUHDVLQJ WKH OLNHOLKRRGRIDQLGOHWLPHVORWWRH[HFXWHWKHWHVWSURJUDPVHYHQ LQ KLJKXWLOL]HG SURFHVVRU FRUHV 7KLV LV YHU\ LPSRUWDQW LQ WKH FDVHRI138VVLQFHRQOLQHHUURUGHWHFWLRQVKRXOGEHSHUIRUPHG HYHQ GXULQJ KHDY\ WUDIILF SHULRGV 2Q WKH RWKHU KDQG LQ WKH FDVHRIWKHLQWUXVLYHDSSURDFK)LJE VKRUWHUWHVWDSSOLFDWLRQ WLPHPHDQVHLWKHUOHVVSHUIRUPDQFHRYHUKHDGRUPRUHIUHTXHQW WHVWUXQVWKXVVKRUWHUHUURUGHWHFWLRQODWHQF\ 7KH WHVW DSSOLFDWLRQ WLPH UHGXFWLRQ ZLOO EH DFKLHYHG E\ SDUDOOHO H[HFXWLRQ RI SURJUDPV RQ GLIIHUHQW FRUHV JXLGHG E\ SURILOLQJ WKH H[HFXWLRQ RI WHVW SURJUDPV RQ VLQJOH SURFHVVRU FRUHVDQGHIIHFWLYHKHXULVWLFVWRLGHQWLI\WKHVFKHGXOHVWKDWZLOO OHDG WR WKH VPDOOHVW QXPEHU RI UHVRXUFH FRQIOLFWV )RU WKH LQWUXVLYH DSSURDFK WKH WHVW SURJUDP SURILOLQJ DQG WHVW VFKHGXOLQJ FDQ EH D DSSOLHG VWDWLFDOO\ VLQFH WKH WHVW SURJUDPV UXQQLQJLQHDFKJURXSRISURFHVVLQJFRUHVDUHNQRZQDSULRUL :KHUHDVWKHQRQLQVWUXVLYHDSSURDFKUHTXLUHVRQOLQHSURILOLQJ RIWKHXVHUSURJUDPVDQGG\QDPLFWHVWVFKHGXOLQJ 7KHPDLQREMHFWLYHVRIDWHVWVFKHGXOLQJDSSURDFKDUH ƒ 7RUHGXFHFRUHFRQWHQWLRQIRUVKDUHGUHVRXUFHVWDNLQJLQWR FRQVLGHUDWLRQ WKH WHVW SURJUDP SURILOLQJ DQG WKH V\VWHP DUFKLWHFWXUH6HH6HFWLRQ,,,&IRUUHODWHGGLVFXVVLRQ ƒ 7RJURXSWKHSURFHVVLQJFRUHVLQWKHLQWUXVLYHDSSURDFKLQ VXFKDZD\VRWKDWWRPLQLPL]HWKHWUDIILFFRQJHVWLRQGXULQJWKH WHVW H[HFXWLRQ RI WKH JURXS )RU H[DPSOH LQ WKH FDVH RI 6&& DUFKLWHFWXUHJLYHQWKDWWKHPHVKVXSSRUWVDVLPSOHSUHGLFWLYH [\  URXWLQJ VFKHPH WKH URXWHV EHWZHHQ WKH SURFHVVRU FRUHV DQG WKH IRXU PHPRU\ FRQWUROOHUV FDQ EH SUHFRPSXWHG )LJ  VKRZV VL[ FRUHV RI 6&& ZLWKLQ WKH VDPH PHPRU\ GRPDLQ VDPH PHPRU\ FRQWUROOHU  UHTXHVWLQJ VLPXOWDQHRXVO\ PHPRU\ WUDQVIHU DQG KLJKOLJKWV WKH FRUUHVSRQGLQJ [\  URXWHV ,W LV REYLRXVWKDWWKHPHVKLQWHUIDFHXQLWVRIWKHWKUHHWLOHVDQGWKH URXWHURIWLOH DUHFRQJHVWHGGXHWRVLPXOWDQHRXVPHPRU\ UHTXHVWV IURP PXOWLSOH FRUHV DQG PD\ LPSRVH ORQJ GHOD\V LQ WKHV\VWHPSHUIRUPDQFH7KXVJURXSLQJWKHFRUHVWRDYRLGWKH FURVVLQJRIWKHURXWHVPLQLPL]HVWKHWUDIILFFRQJHVWLRQ ƒ 7RRYHUORDGZLWKWHVWSURJUDPVWKHOHVVVWUHVVHGSURFHVVRU FRUHV KDYLQJ PRUH LGOH SHULRGV 7KLV FDQ EH GRQH IRU WKH WHVW SURJUDPVWKDWWDUJHWVKDUHGUHVRXUFHVHJPHPRU\FRQWUROOHUV FRPPRQ)38VURXWHUVTXHXHVHWF DQGPXVWEHDSSOLHGRQFH

2013 IEEE 19th International On-Line Testing Symposium (IOLTS)

7LOH  5

'5$0

0&

5

0HVK

7LOH  5

0HVK

&RUH &RUH

5 &RUH

0HVK

5

7LOH 

&RUH

&RUH &RUH

5

0HPRU\GRPDLQ

)LJ 7UDIILFFRQJHVWLRQLQDQ6&&PHPRU\GRPDLQ



ƒ 7R DOORFDWH WKH DYDLODEOH EDQGZLGWK RI WKH V\VWHP HJ PHPRU\ DQG FRPPXQLFDWLRQQHWZRUNEDQGZLGWK  WRWKH FRUHV DFFRUGLQJWRWKHLUEDQGZLGWKUHTXLUHPHQWV)RUH[DPSOHLQWKH 6&&DUFKLWHFWXUHWKHPDLQPHPRU\DFFHVVWLPHGHSHQGVRQWKH FRUH ORFDWLRQ WKH ORQJHU WKH GLVWDQFH EHWZHHQ WKH SURFHVVRU FRUH DQG WKH '5$0 FRQWUROOHU WKH KLJKHU LV WKH URXQG WULS GHOD\ IRU WKH GDWD WUDQVIHU 7KXV D WHVW VFKHGXOLQJ DOJRULWKP VKRXOGDOORFDWHPRUHEDQGZLGWKWRWKHPRUHGLVWDQWFRUHV & :KLFKV\VWHPSDUDPHWHUVDIIHFWWHVWVFKHGXOLQJ" 7KHH[HFXWLRQWLPHRIDSURJUDPLVJLYHQE\ ([HFXWLRQWLPH  &38WLPH0HPDFFHVVHVî&DFKHKLWWLPH &DFKHPLVVUDWHî0HPDFFHVVWLPH   ZKHUHWKH0HPDFFHVVWLPHRIDFRUHLQDPXOWLSURFHVVRUFKLS LV WKH WRWDO GHOD\ EHWZHHQ VHQGLQJ D PHPRU\ UHTXHVW DQG UHFHLYLQJ WKH UHTXHVWHG GDWD IURP PHPRU\ ,Q WKH 6&& DUFKLWHFWXUHWKH/FDFKHKLWWLPHLVFRUHF\FOHVZKLOHWKH PHPRU\DFFHVVWLPHLVJLYHQE\>@>@ 0HPDFFHVVWLPH î7FRUHîQîî7PHVKî7UDP  ZKHUH WKH ODWHQF\ RI D FRUH UHTXHVWLQJ GDWD IURP LWV PHPRU\ FRQWUROOHU LV  FRUH FORFN F\FOHV  WKH GHOD\ RI D URXWHU IRUZDUGLQJ UHTXHVW LV  PHVK FORFN F\FOHV DQG WKH '5$0 DFFHVVWLPHLVPHPRU\FORFNF\FOHVQGHQRWHVWKHQXPEHURI KRSVLQWKHURXWHEHWZHHQWKHFRUHDQGLWVPHPRU\FRQWUROOHU ,W LV HYLGHQW IURP IRUPXODV   DQG   WKDW WKH IROORZLQJ V\VWHP SDUDPHWHUV DIIHFW WKH H[HFXWLRQ WLPH RI WHVW SURJUDPV DQGVKRXOGEHWDNHQLQWRFRQVLGHUDWLRQIRUWHVWVFKHGXOLQJ ƒ /RFDO FDFKHV 7KH EHKDYLRU RI ORFDO FDFKHV SOD\V DQ LPSRUWDQW UROH LQ WKH RYHUDOO PHPRU\ SHUIRUPDQFH 3URILOLQJ WKH FDFKH EHKDYLRU IRU WKH WHVW SURJUDPV ZH FDQ FKDUDFWHUL]H WKH VXESURJUDPV DV ORZ RU KLJK FDFKH PLVV UDWH SURJUDPV 1RWH WKDW KLJK PLVV UDWH SURJUDPV UHTXLUH KLJKHU PHPRU\ EDQGZLGWK 7KHUHIRUH DQ HIIHFWLYH VFKHGXOLQJ DOJRULWKP VKRXOGDYRLGH[HFXWLQJKLJKPLVVUDWHSURJUDPVVLPXOWDQHRXVO\ LQ PXOWLSOH FRUHV EHFDXVH WKH FXPXODWLYH UHTXLUHG EDQGZLGWK PD\ H[FHHG WKH DYDLODEOH OLPLW FDXVLQJ FRQVLGHUDEOH GHOD\V 7KH 6&& DUFKLWHFWXUH VXSSRUWV .%  .% LQVWUXFWLRQ  GDWD / SHU FRUH DQG  .% / SHU FRUH 7KH H[SHULPHQWDO UHVXOWVVHH)LJ VKRZDSHUIRUPDQFHORVVRIDERXWZKHQ WKH 6&& / FDFKH LV GLVDEOHG DQ LQGLFDWLRQ RI WKH LPSDFW RI FDFKHVRQWHVWH[HFXWLRQWLPH ƒ ,QWHUFRQQHFWLRQ QHWZRUN :KHQ PXOWLSOH FRUHV LQ D PHPRU\ GRPDLQ DFFHVV WKH LQWHUFRQQHFWLRQ QHWZRUN FRQFXUUHQWO\ WKH\ H[KLELW KLJK WUDIILF FRQJHVWLRQ ,Q >@

*VFKZDQGWQHU HWDODVVHVVHGWKH PHPRU\ SHUIRUPDQFHRIWKH 6&& ZKHQPXOWLSOH FRUHVDFFHVV WKHVDPH PHPRU\ FRQWUROOHU LQSDUDOOHO7KHSHUIRUPDQFHGHJUDGDWLRQZKHQDOOFRUHVRID PHPRU\ GRPDLQ VLPXOWDQHRXVO\ LVVXH PHPRU\ UHTXHVWV UHDFKHVXSWRFRPSDUHGWRWKHVLQJOHFRUHSHUIRUPDQFH )RU WKH HYDOXDWLRQ RI WKH PHPRU\ SHUIRUPDQFH WKH\ XVHG D PHPRU\LQWHQVLYHEHQFKPDUN)RUWKHSXUSRVHRIRXUVWXG\ZH GHYHORSHG D WHVW SURJUDP WKDW VWUHVVHV WKH PHPRU\ V\VWHP RI WKH6&&SURFHVVRUVLQFHLWH[HFXWHVDODUJHQXPEHURIPHPRU\ ORDG DQG VWRUH RSHUDWLRQV RQ WRWDO .% RI GDWD DUH PRYHG VHYHUDO WLPHV EHWZHHQ FRUH DQG PHPRU\  RQ D ORRSEDVHG VWUXFWXUH 2XU H[SHULPHQWV VKRZHG D KLJK SHUIRUPDQFH GURS ZKHQDOOFRUHVRIDPHPRU\GRPDLQUXQLQSDUDOOHOWKHVDPH WHVW SURJUDPV )LJ  VKRZV WKH H[HFXWLRQ WLPH RI RXU WHVW SURJUDPZLWK/FDFKHHLWKHUHQDEOHGRUGLVDEOHG7KHOLQHVRI WKH GLDJUDP SUHVHQW UHVXOWV IRU WKH H[HFXWLRQ RI WKH WHVW SURJUDP LQ D VLQJOH FRUH RU LQ   DQG  FRUHV RI WKH VDPH PHPRU\GRPDLQLQSDUDOOHO1RWHWKDWWKHH[SHULPHQWDOUHVXOWV LQ>@SURYHGWKDWUXQQLQJWKHEHQFKPDUNLQDOOFRUHVRID VLQJOH PHPRU\ GRPDLQ SURGXFHV WKH VDPH PHPRU\ SHUIRUPDQFHUHVXOWVFRPSDUHGWRUXQQLQJLWLQDOOFRUHVGXH WR WKHFRPSOHWHO\ VHSDUDWHGPHPRU\ GRPDLQVWKLVLV ZK\ ZH UHVWULFWHGRXUH[SHULPHQWVWRWKHFRUHV RIDVLQJOHPHPRU\ GRPDLQ1RWHDOVRWKDWVXFFHVVLYHUXQVRIWKHWHVWSURJUDPJLYH QRQLGHQWLFDO H[HFXWLRQ WLPHV 7KXV )LJ  UHSRUWV ERWK WKH DYHUDJH FDOFXODWHG RYHU D QXPEHU RI  UXQV  DQG WKH PD[LPXPH[HFXWLRQWLPH7KHPD[LPXPH[HFXWLRQWLPHPXVW EHFRQVLGHUHGLQRXUVWXG\VLQFHDWHVWVFKHGXOLQJPHWKRGRORJ\ VKRXOG EH EDVHG RQ UHVXOWV RI WKH ZRUVWFDVH H[HFXWLRQ WLPH DQDO\VLV LQ RUGHU WR VFKHGXOH WKH WHVW SURJUDPV LQWR WKH LGOH SHULRGVRIWKHFRUHV$VVKRZQLQ )LJWKHXSVFDOLQJRIWKH WHVWH[HFXWLRQWLPHLQFRUHVUHDFKHVXSWR/HQDEOHG  DQG/GLVDEOHG IRUWKHDYHUDJHFDVHDQGXSWR/ HQDEOHG  DQG  / GLVDEOHG  IRU WKH PD[LPXP FDVH 7KLV KLJKXSVFDOLQJLVGXHWRWKHIDFWWKDWWKHWHVWSURJUDPVUXQQLQJ LQ DOO  FRUHVRIWKHVDPH PHPRU\ GRPDLQ H[HFXWH LGHQWLFDO FRSLHV RI WKH VDPH SURJUDP LQ SDUDOOHO LQFUHDVLQJ WKH WUDIILF FRQJHVWLRQLQWKHPHVK7KHUHIRUHDQHIIHFWLYHWHVWVFKHGXOLQJ DOJRULWKP VKRXOG DYRLG WKH KLJK PHPRU\ SHUIRUPDQFH GURS EDODQFLQJ WKH PHPRU\ LQWHQVLYH WHVW VXESURJUDPV LQWR WKH FRUHVRIWKHGRPDLQ)RUH[DPSOHLQDQ6&&OLNHDUFKLWHFWXUH PHPRU\LQWHQVLYHWHVWVXESURJUDPVVKRXOGQRWEHVFKHGXOHGLQ SDUDOOHO LQ WKH WZR FRUHV RI D WLOH RU RQ WLOHV DFFHVVLQJ WKH PHPRU\FRQWUROOHUWKURXJKWKHVDPHURXWH



)LJ$YHUDJH DQG PD[LPXP H[HFXWLRQ WLPHV RI WHVW SURJUDP ZLWK / HQDEOHG RU GLVDEOHG &ORFN IUHTXHQF\ VHWWLQJV WLOHPHVK''5 0+]0+]0+] 3DUHQWKHVHV LQ WKH GLDJUDP LQFOXGH WKH XSVFDOLQJRIWKHWHVWH[HFXWLRQWLPHFRPSDUHGWRVLQJOHFRUHH[HFXWLRQ

2013 IEEE 19th International On-Line Testing Symposium (IOLTS)

171

ƒ 0DLQ PHPRU\ 7KH XVH RI VKDUHG RU SULYDWH PHPRU\ IRU WKHVWRUDJHRI WHVWGDWD PD\ FRQVLGHUDEO\ DIIHFWWKH H[HFXWLRQ WLPH ,Q WKH RULJLQDO YHUVLRQ RI WKH WHVW SURJUDP HDFK FRUH RZQV D VHSDUDWH FRS\ RI WKH SURJUDP ZKLFK UHVLGHV LQ LWV SULYDWHVSDFH7RUHGXFHWKHPHPRU\VWRUDJHUHTXLUHPHQWVZH GHYHORSHGDQRWKHUYHUVLRQRIWKHWHVWSURJUDPZKLFKVWRUHVWKH WHVWGDWDLQDVKDUHGPHPRU\VSDFH7KXVDOOFRUHVDFFHVVWKH WHVWGDWDIURPWKHVDPHPHPRU\VSDFH7KLVDOORZVXVWRVWRUHD VLQJOHFRS\RIWHVWGDWDDQGSURFHVVRUVWRKDYHDFFHVVWRWKHWHVW UHVSRQVHVRIWKHRWKHUFRUHV1RWHWKDWWKHFRUHVVWRUHWKHLURZQ WHVW UHVSRQVHV LQ WKH VKDUHG PHPRU\ UHJLRQ EXW LQ GLIIHUHQW DUUD\V 7KLV WHVW FRQILJXUDWLRQ HQDEOHV WKH DSSOLFDWLRQ RI D PXWXDOWHVWLQJVWUDWHJ\ZKHUHWKHGLDJQRVLVRIDGHIHFWHGFRUH LV SHUIRUPHG E\ LWV QHLJKERUV )LJ  SUHVHQWV WKH H[HFXWLRQ WLPHRIWKHUHYLVHGWHVWSURJUDPUXQQLQJLQDQGFRUHV RIWKHVDPHGRPDLQ7KHH[HFXWLRQWLPHRIWKHUHYLVHGSURJUDP LVPXFKKLJKHUWKDQWKHH[HFXWLRQWLPHRIWKHRULJLQDOSURJUDP VHH )LJ   7KLV LV PDLQO\ GXH WR WKH IDFW WKDW WKH VKDUHG PHPRU\LQ6&&LVQRQFDFKHDEOH)XUWKHUPRUHWKHXSVFDOLQJ RIWKHWHVWH[HFXWLRQWLPHLQFRUHVUHDFKHVXSWRIRUWKH DYHUDJHFDVHDQGXSWRIRUWKHPD[LPXPFDVH7KXVDWHVW VFKHGXOLQJDOJRULWKPVKRXOGDGGUHVVWKHSUREOHPRIH[FHHGLQJ WKHDYDLODEOHEDQGZLGWKRIVKDUHGPHPRU\LQFRQMXQFWLRQZLWK WKHSUREOHPRIWUDIILFFRQJHVWLRQLQWKHQHWZRUN



)LJ0D[LPXP H[HFXWLRQ WLPHV RI WHVW SURJUDP IRU  DQG  FRUHV / GLVDEOHG  )RXU GLIIHUHQW PHVK''5 FORFN IUHTXHQF\ FRPELQDWLRQV 7LOH IUHTXHQF\ LV  0+] 7KH QXPEHUV LQ WKH ERWWRP RI WKH EDUV VKRZWKHXSVFDOLQJRIWHVWWLPHIURPVLQJOHFRUHWRFRUHH[HFXWLRQ

,9 &21&/86,21 ,Q WKLV SDSHU ZH LQYHVWLJDWH WKH GLIIHUHQW DVSHFWV RI DQ LPSRUWDQW WRSLF LQ SDUDOOHO PXOWLFRUH SURFHVVRUV DUFKLWHFWXUHV RQOLQH HUURU GHWHFWLRQ E\ VRIWZDUHEDVHG PHWKRGV DQG WKH LPSRUWDQFH RI HIIHFWLYH WHVW VFKHGXOLQJ DOJRULWKPV 8VLQJ WKH FRUH,QWHO¶V6&&DUFKLWHFWXUHZHHYDOXDWHWKHSHUIRUPDQFH RIWHVWSURJUDPVDQGWKHWHVWWLPHLQFUHDVHZKHQPXOWLSOHFRUHV UXQ WKHP &RQJHVWLRQ RQ VKDUHG KDUGZDUH UHVRXUFHV FDXVHV VHULRXV SHUIRUPDQFH ORVV RU WHVW WLPH LQIODWLRQV  2XU H[SHULPHQWDO VWXG\ GHPRQVWUDWHV WKH QHHG IRU HIIHFWLYH WHVW VFKHGXOLQJDOJRULWKPVLQWKHVHPDVVLYHO\SDUDOOHODUFKLWHFWXUHV $&.12:/('*(0(176



)LJ$YHUDJHDQGPD[LPXPH[HFXWLRQWLPHVRIWHVWSURJUDP/GLVDEOHG  7HVWGDWDLQVKDUHGPHPRU\&ORFNIUHTXHQF\VHWWLQJVWLOHPHVK''5 0+]0+]0+] 3DUHQWKHVHV LQ WKH GLDJUDP LQFOXGH WKH XSVFDOLQJRIWKHWHVWH[HFXWLRQWLPHFRPSDUHGWRVLQJOHFRUHH[HFXWLRQ

7RLGHQWLI\ZKLFKLVWKHSUHYDLOLQJERWWOHQHFNRIWKHV\VWHP HJ &38 QHWZRUN RU PDLQ PHPRU\  DQG SURSRVH HIIHFWLYH VFKHGXOHVDGHWDLOHGSURILOLQJRIWKHWHVWSURJUDPVLVQHHGHG,I PXOWLSOH IUHTXHQF\ VHWWLQJV DUH VXSSRUWHG WKH VFKHGXOLQJ DOJRULWKP FRXOG DGMXVW WKH FORFN IUHTXHQFLHV WR WKH WHVW SURJUDP FKDUDFWHULVWLFV 7KXV D WHVW VFKHGXOLQJ PHWKRGRORJ\ FRXOG DOORFDWH &38LQWHQVLYH SURJUDPV WR &38V UXQQLQJ DW KLJKHU IUHTXHQFLHV DQG PHPRU\LQWHQVLYH SURJUDPV WR &38V UXQQLQJ DW ORZHU IUHTXHQFLHV FRPSDUHG WKDQ WKH PHVK DQG PHPRU\FORFNV7KH6&&VXSSRUWVGLIIHUHQWIUHTXHQF\VHWWLQJV IRU WKH &38V WKH PHVK DQG WKH PHPRU\ ,W VXSSRUWV  IUHTXHQF\ GRPDLQV VHH )LJ   ZKLFK FDQ EH FORFNHG WR D UDQJH IURP  WR  0+] 7KH QHWZRUN DQG WKH PHPRU\ FRQWUROOHUVFDQDOVRRSHUDWHRQWKHLURZQFORFNVWKHQHWZRUNDW RU0+]DQGWKHFRQWUROOHUDWRU0+])LJ VKRZV H[HFXWLRQ WLPHRIWKHWHVWSURJUDP UXQQLQJRQ DQG  FRUHV IRU  GLIIHUHQW PHVK''5 IUHTXHQF\ FRPELQDWLRQV 7KHEHVWPHPRU\SHUIRUPDQFHLVDFKLHYHGIRUWKHFRPELQDWLRQ PHVK''5DW0+]2EVHUYHWKDWLQFUHDVLQJRQO\WKH PHVKIUHTXHQF\GRHVQRWUHGXFHH[HFXWLRQWLPH

172

7KLVZRUNLV IXQGHGLQ SDUWE\ D UHVHDUFK JLIW IURP WKH &LVFR 8QLYHUVLW\ 5HVHDUFK 3URJUDP )XQG D FRUSRUDWH DGYLVHG IXQG RI 6LOLFRQ 9DOOH\ &RPPXQLW\ )RXQGDWLRQ  RQ RQOLQH HUURU GHWHFWLRQIRUPXOWLFRUHPLFURSURFHVVRUDUFKLWHFWXUHV,WLV DOVR VXSSRUWHG E\ ,QWHO¶V 0$5& 3URJUDP SURYLGLQJ DFFHVV WR WKH 6LQJOHFKLS&ORXG&RPSXWHU6&& FKLS 5()(5(1&(6 >@ - +RZDUG HW DO $ &RUH ,$ PHVVDJHSDVVLQJ SURFHVVRU ZLWK '9)6LQQP&026,66&&SS)HE >@ 7 &KHQ HW DO &HOO %URDGEDQG (QJLQH $UFKLWHFWXUH DQG LWV ILUVW LPSOHPHQWDWLRQ²$ SHUIRUPDQFH YLHZ ,%0 -RXUQDO RI 5HVHDUFK DQG 'HYHORSPHQWYROQRSS6HSW >@ 7KH &LVFR 4XDQWXP)ORZ 3URFHVVRU &LVFR¶V 1H[W *HQHUDWLRQ 1HWZRUN 3URFHVVRU&LVFR6\VWHPV,QF >@ 2&7(21 3OXV &1;;  WR &RUH 0,36%DVHG 6R&V &DYLXP 1HWZRUNV0RXQWDLQ9LHZ&$ >@ 0$GLOHWWD05RVHQEOXWK'%HUQVWHLQ³7KHQH[WJHQHUDWLRQRILQWHO L[SQHWZRUNSURFHVVRUV´,QWHO7HFKQRORJ\-RXUQDO $XJ >@ 03VDUDNLV'*L]RSRXORV(6DQFKH]065HRUGD0LFURSURFHVVRU 6RIWZDUH%DVHG 6HOI7HVWLQJ ,((( 'HVLJQ  7HVW RI &RPSXWHUV YROQRSS0D\-XQH >@ $ $SRVWRODNLV ' *L]RSRXORV 0 3VDUDNLV $ 3DVFKDOLV ³6RIWZDUH %DVHG 6HOI7HVWLQJ RI 6\PPHWULF 6KDUHG0HPRU\ 0XOWLSURFHVVRUV´ ,(((7UDQVDFWLRQVRQ&RPSXWHUVYROQRSS >@ 1 )RXWULV 0 3VDUDNLV ' *L]RSRXORV $ $SRVWRODNLV ; 9HUD $ *RQ]DOH]$³0WVEVW6HOIWHVWRSWLPL]DWLRQLQPXOWLWKUHDGHGPXOWLFRUH DUFKLWHFWXUHV,7&SS >@ 0$6NLWVDV&$1LFRSRXORVDQG0.0LFKDHO7RZDUG6HOHFWLYH 6RIWZDUH%DVHG 6HOI7HVWLQJ LQ 0XOWL&RUH 0LFURSURFHVVRUV LQ 3URF VW0(',$1:RUNVKRSSS >@ 6&&3URJUDPPHU¶V*XLGHUHY-DQ >@ 3 *VFKZDQGWQHU 7 )DKULQJHU 5 3URGDQ 3HUIRUPDQFH $QDO\VLV DQG %HQFKPDUNLQJRIWKH,QWHO6&&&/867(5 SS

2013 IEEE 19th International On-Line Testing Symposium (IOLTS)