comparing the Robustness of POSIX Operating Systems

WK$QQXDO,QWHUQDWLRQDO6\PSRVLXPRQ)DXOW7ROHUDQW&RPSXWLQJ-XQH0DGLVRQ:LVFRQVLQ

&RPSDULQJWKH5REXVWQHVVRI326,;2SHUDWLQJ6\VWHPV 3KLOLS.RRSPDQ -RKQ'H9DOH 'HSDUWPHQWRI(OHFWULFDODQG&RPSXWHU(QJLQHHULQJ ,QVWLWXWHIRU&RPSOH[(QJLQHHUHG6\VWHPV &DUQHJLH0HOORQ8QLYHUVLW\3LWWVEXUJK3HQQV\OYDQLD86$ NRRSPDQ#FPXHGXMGHYDOH#HFHFPXHGX

$EVWUDFW

&ULWLFDO V\VWHP GHVLJQHUV DUH WXUQLQJ WR RIIWKHVKHOI RSHUDWLQJV\VWHP26  VRIWZDUH WRUHGXFHFRVWVDQGWLPH WRPDUNHW  8QIRUWXQDWHO\ JHQHUDOSXUSRVH 26HV GR QRW DOZD\V UHVSRQG WR H[FHSWLRQDO FRQGLWLRQV UREXVWO\ HLWKHU DFFHSWLQJ H[FHSWLRQDO YDOXHV ZLWKRXW FRPSODLQW RU VXIIHULQJ DEQRUPDO WDVN WHUPLQDWLRQ  (YHQ WKRXJK GLUHFW PHDVXUHPHQW LV LPSUDFWLFDO WKLV SDSHU XVHV D PXOWL YHUVLRQ FRPSDULVRQ WHFKQLTXH WR UHYHDO D  WR  QRUPDOL]HG UDWH DW ZKLFK H[FHSWLRQDO SDUDPHWHU YDOXHV FDXVH QR HUURU UHSRUW LQ FRPPHUFLDO 326,; 26 LPSOHPHQWDWLRQV  $GGLWLRQDOO\  IXQFWLRQV DFURVV  26HV DUH FRPSDUHG WR UHYHDO FRPPRQ PRGH UREXVWQHVV IDLOXUHV:KLOHWKHEHVWVLQJOH26KDVDUREXVWQHVV IDLOXUHUDWHIRUV\VWHPFDOOVRIIDLOXUHVDUHFRPPRQ DFURVV DOO  26HV H[DPLQHG  +RZHYHU FRPELQLQJ & OLEUDU\ FDOOV ZLWK V\VWHP FDOOV LQFUHDVHV WKHVH UDWHV WR IRUWKHEHVWVLQJOH26DQGIRUFRPPRQPRGH IDLOXUHV  7KHVH UHVXOWV VXJJHVW WKDW 26 LPSOHPHQWDWLRQV DUHQRWFRPSOHWHO\GLYHUVHDQGWKDW&OLEUDU\IXQFWLRQVDUH ERWKOHVVGLYHUVHDQGOHVVUREXVWWKDQV\VWHPFDOOV

,QWURGXFWLRQ 7KH UREXVWQHVV RI WKH RSHUDWLQJ V\VWHP 26 $SSOLFDWLRQ 3URJUDPPLQJ ,QWHUIDFH $3,  LV EHFRPLQJ LQFUHDVLQJO\ LPSRUWDQW  &RVW DQG WLPHWRPDUNHW FRQVWUDLQWV DUH SUHVVXULQJ HYHQ FULWLFDOV\VWHP GHYHORSHUV WRXVHRIIWKHVKHOIFRPPHUFLDO26VRIWZDUH:KLOHPDQ\ RIWKHVH26HVDUHJHQHUDOO\FRQVLGHUHGUREXVWWKH%DOOLVWD WHVWLQJ V\VWHP  KDV SURGXFHG UHVXOWV VKRZLQJ WKDW FRPPHUFLDO26HVKDYHDVLJQLILFDQWUREXVWQHVVIDLOXUHUDWH IRU VLQJOH SURFHGXUH FDOOV ZLWK H[FHSWLRQDO SDUDPHWHU YDOXHVDWWKHWDVNOHYHODQGHYHQDIHZUHDGLO\UHSURGXFLEOH FDWDVWURSKLFIDLOXUHPRGHV>@ 7KLV SDSHU FRPSDUHV WKH UHVXOWV RI PXOWLSOH 26 LPSOHPHQWDWLRQV WR LQFUHDVH WKH XQGHUVWDQGLQJ RI WKHLU UREXVWQHVV IDLOXUH PRGHV DQG LQ SDUWLFXODU LQVWDQFHV LQ ZKLFK WKH\ IDLO WR GHWHFW H[FHSWLRQDO LQSXW SDUDPHWHU YDOXHV  $GGLWLRQDOO\ D ODUJH VHW RI WHVW GDWD RQ PXOWLSOH

LPSOHPHQWLRQVRIWKHVDPH$3,RIIHUVDXQLTXHFKDQFHWR H[SORUH WKH LVVXH RI TXDQWLI\LQJ ODUJHVFDOH VRIWZDUH GLYHUVLW\DOEHLWRQO\RIH[FHSWLRQKDQGOLQJFKDUDFWHULVWLFV )LUVW D PXOWLYHUVLRQ FRPSDULVRQ RI %DOOLVWD WHVWLQJ UHVXOWV LV XVHG WR LGHQWLI\ VRFDOOHG 6LOHQW UREXVWQHVV IDLOXUHVLQILIWHHQ26HV$6LOHQWIDLOXUHLVRQHLQZKLFKD IXQFWLRQFDOOSURGXFHVQRLQGLFDWLRQRIDQHUURUZKHQIHG H[FHSWLRQDO SDUDPHWHU YDOXHV ZKHQ LQ IDFW VXFK DQ LQGLFDWLRQ VKRXOG EH SURGXFHG WR LPSOHPHQW UREXVW EHKDYLRU  7KHVH IDLOXUHV FDQQRW RWKHUZLVH EH PHDVXUHG ZLWKRXW D EHKDYLRUDO VSHFLILFDWLRQ  $GGLWLRQDO UHVXOWV SUHVHQWHGLQFOXGHXVLQJDVLPLODUWHFKQLTXHWR³GLVWLOO´QRQ H[FHSWLRQDO WHVW FDVHV IURP WKH %DOOLVWD FRPELQDWLRQDO WHVWLQJDSSURDFKDQGDOLVWLQJRIWKHGDWDW\SHVPRVWRIWHQ FRUUHODWHGZLWKUREXVWQHVVIDLOXUHV 1H[W 1ZD\ FRPSDULVRQV RI GLIIHUHQW 26 UREXVWQHVV WHVWLQJUHVXOWVDUHXVHGWRPHDVXUHVRIWZDUHGLYHUVLW\IURP WKH SRLQW RI YLHZ RI UREXVW H[FHSWLRQ KDQGOLQJ  :KLOH DFWXDOO\ LPSOHPHQWLQJ D V\VWHP ZLWK DQ 1YHUVLRQ 26 VHHPVLPSUDFWLFDOWKHUHVXOWVRIWKLVFRPSDULVRQJLYHWKH ILUVWODUJHVFDOHH[SHULPHQWDOUHVXOWVIRUXQGHUVWDQGLQJWKH OHYHO RI VRIWZDUH GLYHUVLW\ WKDW LV OLNHO\ WR EH IRXQG LQ FRPPHUFLDOSURGXFWVDJDLQOLPLWHGWRH[FHSWLRQKDQGOLQJ FKDUDFWHULVWLFV   7KH UHVXOWV RI FRPSDULQJ WKLUWHHQ 26 LPSOHPHQWDWLRQV WZR 26HV KDG WR EH HOLPLQDWHG WR PD[LPL]H XVDEOH FRPSDULVRQ GDWD  VKRZ WKDW 26 LPSOHPHQWDWLRQV DUH QRW HQWLUHO\ GLYHUVH DQG WKDW DQ DVVXPSWLRQ RI IDLOXUH LQGHSHQGHQFH VHHPV OHVV XQUHDVRQDEOH IRU 26 V\VWHP FDOOV WKDQ IRU & OLEUDU\ IXQFWLRQV

%DFNJURXQG 7KLV SDSHU WDNHV FRQFHSWV IURP WKH ILHOG RI 1YHUVLRQ VRIWZDUH IDXOW WROHUDQFH DQG FRPELQHV WKHP ZLWK UHVXOWV IURP D VFDODEOH UREXVWQHVV WHVWLQJ PHWKRGRORJ\  ,W WKHQ DVVHVVHV26UREXVWQHVVIDLOXUHVDQGWKHLQKHUHQWGLYHUVLW\ RI PXOWLSOH YHUVLRQ WHFKQLTXHV RQ RQH W\SH RI ODUJH VRIWZDUHV\VWHP

.RRSPDQ 'H9DOH)7&6

1YHUVLRQVRIWZDUH 1YHUVLRQVRIWZDUHLQYROYHVXVLQJ1GLIIHUHQWYHUVLRQV RI SURJUDPV LPSOHPHQWLQJ WKH VDPH VSHFLILFDWLRQ IRU EXLOGLQJ UREXVW IDXOWWROHUDQW VRIWZDUH >@>@  *HQHUDOO\ WKHVH V\VWHPV XVH DQ LGHD VLPLODU WR 1YHUVLRQ PRGXODU KDUGZDUHUHGXQGDQF\EDVLQJDFWLRQVXSRQDPDMRULW\YRWH WDNHQ RI VHYHUDO VRIWZDUH YHUVLRQV WR GHWHUPLQH ZKLFK RXWSXW YDOXHV DUH FRUUHFW  1XPHURXV H[SHULPHQWV KDYH EHHQSHUIRUPHGWRVWXG\1YHUVLRQVRIWZDUHHIIHFWLYHQHVV LQ VHYHUDO PLVVLRQ FULWLFDO DUHDV PRVW QRWDEO\ LQ WKH DHURVSDFH LQGXVWU\ >@>@>@  9DULDWLRQV RQ WKLV WKHPH KDYH EHHQ VXFFHVVIXOO\ LPSOHPHQWHG LQ VDIHW\ FULWLFDO LQGXVWULDOV\VWHPVVXFKDVWKH$LUEXVIOLJKWFRQWUROV\VWHP 1$6$¶V6SDFH6KXWWOHDQGUDLOZD\FRQWUROV\VWHPV $V DQ H[DPSOH D ODUJH UHVHDUFK HIIRUW LQYROYLQJ IDXOW WROHUDQW VRIWZDUH FRQWUROV IRU D 1$6$ DYLRQLFV V\VWHP LQYROYHG ILYH LQGHSHQGHQW WHDPV RI GHYHORSHUV FKRVHQ IURPYDULRXVXQLYHUVLWLHV>@7KH\FDUHIXOO\FRQVWUXFWHGD VSHFLILFDWLRQDQGSHUIRUPHGVHYHUDOOHYHOVRIWHVWLQJRQWKH GHYHORSLQJ VRIWZDUH ZKLFK XSRQ FRPSOHWLRQ DYHUDJHG DSSUR[LPDWHO\OLQHVRI3DVFDOFRGH'HWDLOHGDQDO\VLV ZDV SHUIRUPHG RQ WKH FROOHFWHG GDWD LQ D ODWHU VWXG\ DQG IRXQG WKDW OHVV WKDQ  RI WKH IDXOWV GHWHFWHG FRXOG EH FODVVLILHGDVVLPLODU>@)XUWKHUWKHVWXG\FRQFOXGHGWKDW DIWHUFHUWLILFDWLRQWHVWLQJWKLVQXPEHUGURSSHGVLJQLILFDQWO\ DQGKDGHOLPLQDWHGDOOVSHFLILFDWLRQIDXOWV $QRWKHU VWXG\ VXJJHVWV WKDW DQ DVVXPSWLRQ RI IDLOXUH LQGHSHQGHQFH EDVHG RQ GLYHUVLW\ PD\ QRW EH XQLYHUVDOO\ SRVVLEOH  >@>@>@ EXW WKHVH UHVXOWV DUH FRQWURYHUVLDO >@>@>@0RUHLPSRUWDQWO\VXFKVWXGLHVDUHLQKHUHQWO\ OLPLWHGE\WKHKLJKFRVWRIVRIWZDUHGHYHORSPHQWPDNLQJ LW SURKLELWLYHO\ H[SHQVLYH WR FRQGXFW IXOOVFDOH 1YHUVLRQ VRIWZDUHVWXGLHVLQJHQHUDO6RLWLVUHDVRQDEOHWRDVNDQ DVVXPSWLRQ RI PXOWLYHUVLRQ GLYHUVLW\ VFDOHV WR V\VWHPV KDYLQJDPLOOLRQOLQHVRIFRGHEXWLQJHQHUDOXQGHUWDNLQJD SDUDOOHO GHYHORSPHQW HIIRUW RI WKDW PDJQLWXGH LV LPSUDFWLFDO )RUWXQDWHO\WKHUHDUHDVRPHFRPPHUFLDOO\GHYHORSHG IXOOVL]H V\VWHPV EXLOW WR VWDQGDUG $SSOLFDWLRQ 3URJUDPPLQJ ,QWHUIDFHV $3,V   7KHVH V\VWHPV FDQ EH WHVWHG WR GHWHUPLQH WKH UHODWLYH LQGHSHQGHQFH RI IDLOXUHV 7KH SUREOHP RI HYDOXDWLQJ PXOWLSOH YHUVLRQ HIIHFWLYHQHVV FDQWKXVEHUHGXFHGWRRQHRIILQGLQJDWHVWVXLWHZKLFKLV HFRQRPLFDOWRGHYHORSDQG\HWKDVQRWDOUHDG\EHHQXVHG E\ VRIWZDUHYHQGRUV IRUGHIHFW UHPRYDO  7KLVSDSHU XVHV VXFK UHVXOWV IRU UREXVWQHVV IDLOXUHV IRXQG E\ WKH %DOOLVWD UREXVWQHVV WHVWLQJ V\VWHP DSSOLHG WR 326,; >@ 26 LPSOHPHQWDWLRQV

%DOOLVWDWHVWLQJPHWKRGRORJ\ ,Q EULHI WKH %DOOLVWD WHVWLQJ PHWKRGRORJ\ LQYROYHV

DXWRPDWLFDOO\JHQHUDWLQJFRPELQDWLRQVRIH[FHSWLRQDODQG YDOLG SDUDPHWHU YDOXHV WR EH XVHG LQ FDOOLQJ VRIWZDUH PRGXOHV  7KH UHVXOWV RI WKHVH FDOOV DUH H[DPLQHG WR GHWHUPLQH ZKHWKHU WKH PRGXOH GHWHFWHG DQG QRWLILHG WKH FDOOLQJ SURJUDP RI DQ HUURU WKH WDVN DEQRUPDOO\ WHUPLQDWHG WKH WDVN KXQJ RU ZKHWKHU WKH HQWLUH V\VWHP FUDVKHG$GHWDLOHGGLVFXVVLRQRIWHVWFDVHJHQHUDWLRQFDQ EH IRXQG LQ D SUHYLRXV SDSHU >@ EXW WKH JHQHUDO WHVW PHWKRGRORJ\LVVXPPDUL]HGEHORZ %DOOLVWDRSHUDWHVDWWKHOHYHORIVLQJOHIXQFWLRQFDOOVWR FUHDWH UHSHDWDEOH VLPSOH WHVWV WKDW QRQHWKHOHVV XQFRYHU UREXVWQHVV IDLOXUHV  ,Q HDFK WHVW FDVH D VLQJOH VRIWZDUH 0RGXOH XQGHU 7HVW RU 0X7  LV FDOOHG D VLQJOH WLPH WR GHWHUPLQHZKHWKHULWLVUREXVWZKHQFDOOHGZLWKDSDUWLFXODU VHW RI SDUDPHWHU YDOXHV  7KHVH SDUDPHWHU YDOXHV RU WHVW YDOXHV DUH GUDZQ IURP SRROV RI QRUPDO DQG H[FHSWLRQDO YDOXHVEDVHGRQWKHGDWDW\SHRIHDFKDUJXPHQWSDVVHGWR WKH0X7$WHVWFDVHWKHUHIRUHFRQVLVWVRIWKHQDPHRIWKH 0X7 DQG D WXSOH RI WHVW YDOXHV WKDW DUH SDVVHG DV SDUDPHWHUVLHDWHVWFDVHLVDSURFHGXUHFDOORIWKHIRUP 0X7BQDPHWHVWBYDOXH WHVWBYDOXH      7KXV WKH JHQHUDODSSURDFKWR%DOOLVWDWHVWLQJLVWRWHVWWKHUREXVWQHVV RIDVLQJOHFDOOWRD0X7IRUDVLQJOHWXSOHRIWHVWYDOXHV DQGWKHQUHSHDWWKLVSURFHVVIRUPXOWLSOHWHVWFDVHVWKDWHDFK KDYHGLIIHUHQWFRPELQDWLRQVRIERWKYDOLGDQGLQYDOLGWHVW YDOXHV  :KLOH DFWXDO WHVWV DUH SHUIRUPHG LQ EDWFKHV IRU HIILFLHQF\ LQ SUDFWLFH YLUWXDOO\ DOO WHVW UHVXOWV KDYH EHHQ IRXQGWREHUHSURGXFLEOHLQLVRODWLRQ 7KH %DOOLVWD WHVW KDUQHVV FDWHJRUL]HV WKH WHVW UHVXOWV DFFRUGLQJWRWKH&5$6+VHYHULW\VFDOH>@ ‡ &DWDVWURSKLF IDLOXUHV RFFXU ZKHQ WKH HQWLUH 26 EHFRPHVFRUUXSWHG RU WKHPDFKLQHFUDVKHV RU UHERRWV ,QRWKHUZRUGVWKLVLVDFRPSOHWHV\VWHPFUDVK ‡ 5HVWDUW IDLOXUHV RFFXU ZKHQ D IXQFWLRQ FDOO WR DQ 26 IXQFWLRQ QHYHU UHWXUQV UHVXOWLQJ LQ D WDVN WKDW KDV KXQJDQGPXVWEHWHUPLQDWHGE\IRUFH ‡ $ERUWIDLOXUHVWHQGWREHWKHPRVWSUHYDOHQWDQGUHVXOW LQ DEQRUPDO WHUPLQDWLRQ D FRUH GXPS  RI D WDVN FDXVHGE\DVLJQDOJHQHUDWHGZLWKLQWKH0X7 ‡ 6LOHQWIDLOXUHVRFFXUZKHQDQ26UHWXUQVQRLQGLFDWLRQ RIHUURURQDQH[FHSWLRQDORSHUDWLRQZKLFKFOHDUO\FDQ QRWEHSHUIRUPHGIRUH[DPSOHZULWLQJWRDUHDGRQO\ ILOH DQGZKLFKVKRXOGLQIDFWSURGXFHDQHUURUUHSRUW LQDUREXVWV\VWHP7KLVLVQRWWREHFRQIXVHGZLWKWKH SUREOHPRIQRQGLDJQRVDEOHH[SHULPHQWVGXHWROLPLWHG REVHUYDELOLW\EHFDXVHWKHHUURUUHSRUWLQJPHFKDQLVPLV IXOO\REVHUYDEOHDQGLVREVHUYHGWRIDOVHO\LQGLFDWH³QR HUURU´ WKLV LV DQ DSSOLFDWLRQFHQWULF UDWKHU WKDQ 26 FHQWULFYLHZ  ‡ +LQGHULQJIDLOXUHVRFFXUZKHQDQLQFRUUHFWHUURUFRGH LVUHWXUQHGIURPD0X7ZKLFKFRXOGPDNHLWPRUHGLI ILFXOWWRH[HFXWHDSSURSULDWHHUURUUHFRYHU\7KHVHIDLO XUHVKDYHEHHQREVHUYHGLQSUDFWLFHEXWDUHEH\RQGWKH

.RRSPDQ 'H9DOH)7&6

 VFRSHRIWKLVSDSHU 7KHUH DUH WZR DGGLWLRQDO SRVVLEOH RXWFRPHVRIH[HFXWLQJD WHVW FDVH $ WHVW FDVH PLJKW UHWXUQ ZLWK DQ HUURU FRGH WKDW LV DSSURSULDWH IRU LQYDOLG SDUDPHWHUVIRUPLQJWKHWHVWFDVH7KLV LVDFDVHLQZKLFKWKHWHVWFDVHSDVVHV± LQ RWKHU ZRUGV JHQHUDWLQJ DQ HUURU FRGH LV WKH FRUUHFW UHVSRQVH $GGLWLRQDOO\ LQ VRPH WHVWV WKH 0X7 OHJLWLPDWHO\UHWXUQVQRHUURUFRGHDQG VXFFHVVIXOO\ FRPSOHWHV WKH GHVLUHG RSHUDWLRQ  7KLV KDSSHQV ZKHQ WKH SDUDPHWHUV LQ WKH WHVW FDVH KDSSHQ WR EH DOO YDOLG D QRQH[FHSWLRQDO WHVW FDVH  RU ZKHQ LW LV XQUHDVRQDEOH WR H[SHFWWKH26WRGHWHFWDQH[FHSWLRQDO VLWXDWLRQ VXFK DV SRLQWLQJ WR DQ LQYDOLG DGGUHVV LQ WKH VDPH PHPRU\ SDJHDVDYDOLGDGGUHVV 

$,;  '81,;  '81,;  )UHH %6'  +38;  +38;  ,UL[  ,UL[  /LQX[  /\Q[26  1HW%6'  41;  41;  6XQ26  6XQ26 



1RUPDOL]HG5DZ)DLOXUH5DWHVIRU326,;&DOOV &DWDVWURSKLF

&DWDVWURSKLF

&DWDVWURSKLF

&DWDVWURSKLF

$ERUW )DLOXUHV 5HVWDUW )DLOXUH &DWDVWURSKLFV













1RUPDOL]HG)DLOXUH5DWH

)LJXUH   5DZ UREXVWQHVV WHVWV RQ  326,; RSHUDWLQJ V\VWHPV UHYHDO D VLJQLILFDQW$ERUWIDLOXUHUDWHDQGVHYHUDO&DWDVWURSKLFIDLOXUHV

26WHVWGDWD )LIWHHQ 26 LPSOHPHQWDWLRQV IURP WHQ YHQGRUV ZHUH WHVWHG ZLWK %DOOLVWD \LHOGLQJ D WRWDO RI  GDWD SRLQWVRQXSWRVHOHFWHG326,;IXQFWLRQVDQGV\VWHP FDOOVPRVWV\VWHPVGLGQRWVXSSRUWDOOWHVWHGFDOOVDQGWKXV QRW DOO WHVWV ZHUH UXQ RQ HYHU\ 26   7KH FRPSLOHUV DQG OLEUDULHVXVHGWRJHQHUDWHWKHWHVWVXLWHZHUHWKRVHSURYLGHG E\ WKH 26 YHQGRU  ,Q WKH FDVH RI )UHH%6' 1HW%6' /LQX[DQG/\Q[26WKLVPHDQWWKDW*18&YHUVLRQ DQGWKH*18&OLEUDULHVZHUHXVHGWREXLOGWKHWHVWVXLWH $VXPPDU\RIUREXVWQHVVIDLOXUHUDWHVLVVKRZQLQ)LJXUH LQ ZKLFK DQ DYHUDJH IDLOXUH UDWH LV FRPSXWHG E\ QRUPDOL]LQJ WKH IDLOXUH UDWH DV D SURSRUWLRQ RI IDLOHG WHVW FDVHV IRU HDFK IXQFWLRQ DQG WKHQ WDNLQJ D XQLIRUPO\ ZHLJKWHG DULWKPHWLF PHDQ DFURVV DOO VXSSRUWHG IXQFWLRQV )RU D SDUWLFXODU DSSOLFDWLRQ D ZHLJKWHG PHDQ SHU DQ RSHUDWLRQDO SURILOH PLJKW EH GHVLUDEOH EXW WKH UHVXOWV SUHVHQWHG KHUH DUH JHQHULF IRU WKH $3, UDWKHU WKDQ DQ\ SDUWLFXODU V\VWHP   7HVW H[HFXWLRQ WRRN DSSUR[LPDWHO\ WKUHHWRHLJKWKRXUVSHUV\VWHP 7KHUHZHUHVL[IXQFWLRQ26SDLUVWKDWUHVXOWHGLQHQWLUH RSHUDWLQJ V\VWHP FUDVKHV HLWKHU DXWRPDWLF UHERRWV RU V\VWHP KDQJV   $V DQ H[DPSOH RI WKHVH &DWDVWURSKLF IDLOXUHV ,UL[  FUDVKHV DQG UHTXLUHV D PDQXDO KDUGZDUH UHVHWZKHQH[HFXWLQJWKHFDOO PXQPDSPDOORF 0$;,17  5HVWDUWIDLOXUHVZHUHUHODWLYHO\VFDUFHEXWSUHVHQWLQDOO EXWWZRRSHUDWLQJV\VWHPV$ERUWIDLOXUHVZHUHFRPPRQ LQGLFDWLQJ WKDW LQ DOO RSHUDWLQJ V\VWHPV LW LV UHODWLYHO\ VWUDLJKWIRUZDUG WR HOLFLW D FRUH GXPS IURP DQ LQVWUXFWLRQ ZLWKLQ D IXQFWLRQ RU V\VWHP FDOO   $ FKHFN ZDV PDGH WR

HQVXUH WKDW $ERUW IDLOXUHV ZHUH QRW GXH WR FRUUXSWLRQ RI VWDFN YDOXHV DQG VXEVHTXHQW FRUUXSWLRQPLVGLUHFWLRQ RI FDOOLQJSURJUDPRSHUDWLRQ $ SDUWLFXODU SRLQW RI LQWHUHVW ZDV WKDW SUHYLRXVO\ UHSRUWHG UHVXOWV ZHUH VXEMHFW WR FULWLFLVP E\ 26 YHQGRUV EHFDXVHLQVHYHUDOFDVHVWKHODWHVWDYDLODEOH26YHUVLRQKDG QRW EHHQ WHVWHG  +RZHYHU ZKHQ D QHZHU YHUVLRQ ZDV WHVWHG WKHUH ZDV QRW QHFHVVDULO\ DQ LPSURYHPHQW LQ UREXVWQHVV HJ IRU +38; DQG 41;   $GGLWLRQDOO\ )LJXUH  UHSRUWV UHVXOWV IRU '81,; % ZLWK QR &DWDVWURSKLF IDLOXUHV  +RZHYHU RQ '81,; ' D FDWDVWURSKLF IDLOXUH VSUDQJ LQWR H[LVWHQFH GXH WR DQ DSSDUHQWFKDQJHLQWKHDLRBUDZOLEUDU\IRUWKHFDOO PSURWHFWPDOORF    7KHUHDUHKRZHYHUVRPHTXHVWLRQVOHIWXQDQVZHUHGE\ WKH GDWD LQ )LJXUH   )RU H[DPSOH WKH GHVLJQHUV RI )UHH%6' KDYH WROG XV WKDW WKH\ LQWHQWLRQDOO\ JHQHUDWH $ERUWIDLOXUHVDVWKHLUSUHIHUUHGHUURUUHSRUWLQJPHFKDQLVP DQGVXJJHVWHGLWZDVSRVVLEOHWKH\ORRNEDGE\FRPSDULVRQ EHFDXVH RWKHU 26 LPSOHPHQWDWLRQV PLJKW LQVWHDG VXIIHU IURPHOHYDWHGOHYHOVRI6LOHQWIDLOXUHV7KHPHULWVRIWKH )UHH%6' VWUDWHJ\ LQYROYH D GHEDWH DERXW WKH GHVLUH WR PDNHHUURUVKLJKO\YLVLEOHGXULQJGHYHORSPHQWYVDGHVLUH WRSHUIRUPILQHJUDLQHUURUUHFRYHU\RQFHDQDSSOLFDWLRQLV ILHOGHG EXW WKDW LV EH\RQG WKH VFRSH RI WKLV GLVFXVVLRQ )RUH[DPSOHWKLVPLJKWPHDQWKDW$,;ORRNVEHWWHUWKDQLW UHDOO\LVEHFDXVHDGGUHVVLVUHDGDEOHZLWKRXWH[FHSWLRQ OHDGLQJWRWKHSRVVLELOLW\RIHOHYDWHG6LOHQWIDLOXUHUDWHVIRU 18//SRLQWHUGHUHIHUHQFLQJ7KHUHIRUHLWZDVLPSRUWDQW WRGHWHUPLQH6LOHQWIDLOXUHUDWHVHYHQWKRXJKWKH\FRXOGQRW EHGLUHFWO\PHDVXUHG

.RRSPDQ 'H9DOH)7&6

'DWDDQDO\VLVYLDPXOWLYHUVLRQFRPSDULVRQ

IURP PDVNLQJ H[FHSWLRQV WKDW PLJKW RWKHUZLVH JR XQ QRWLFHGRQRWKHUSDUDPHWHUV%XWDOVRSDUWO\WKLVLVDVLGH HIIHFW RI VFDODEOH WHVWLQJLQ ZKLFK DOO IXQFWLRQVDUH WHVWHG ZLWK SDUDPHWHU YDOXHV WKDW PD\ EH H[FHSWLRQDO IRU RQO\ VRPHIXQFWLRQVIRUH[DPSOHDUHDGRQO\ILOHLVH[FHSWLRQDO IRUDZULWHIXQFWLRQEXWQRWIRUDUHDGIXQFWLRQWHVWHGZLWK WKHVDPHSDUDPHWHUYDOXHV  1ZD\ FRPSDULVRQV ZHUH XVHG WR LGHQWLI\ DQG SUXQH QRQH[FHSWLRQDO WHVW FDVHV IURP WKH GDWD VHW  7KH FRPSDULVRQV DVVXPHG WKDW DQ\ WHVW FDVH LQ ZKLFK DOO RSHUDWLQJV\VWHPVUHWXUQHGZLWKQRLQGLFDWLRQRIHUURUZHUH LQ IDFW QRQH[FHSWLRQDO WHVWV RU ZHUH H[FHSWLRQDO WHVWV ZKLFKFRXOGQRWUHDVRQDEO\EHH[SHFWHGWREHGHWHFWHGRQ FXUUHQW FRPSXWHU V\VWHPV   ,Q DOO  QRQ H[FHSWLRQDO WHVWV ZHUH UHPRYHG DFURVV DOO  RSHUDWLQJ V\VWHPV  )LJXUH  VKRZV WKH DGMXVWHG IDLOXUH UDWHV DIWHU UHPRYLQJQRQH[FHSWLRQDOWHVWV +DQG VDPSOLQJ RI VHYHUDO GR]HQ UHPRYHG WHVW FDVHV LQGLFDWHG WKDW DOO RI WKHP ZHUH LQGHHG QRQH[FHSWLRQDO WKXVVXJJHVWLQJDORZUDWHRIIDOVHVFUHHQLQJV:KLOHWKHUH LVWKHSRVVLELOLW\WKDWH[FHSWLRQDOWHVWFDVHVVOLSSHGSDVVHG WKLVVFUHHQLQJLWVHHPVXQOLNHO\WKDWWKHQXPEHULQYROYHG ZRXOGPDWHULDOO\DIIHFWWKHUHVXOWV

7KH VFDODELOLW\ RI WKH %DOOLVWD WHVWLQJ DSSURDFK KLQJHV RQ QRW QHHGLQJ WR NQRZ WKH IXQFWLRQDO VSHFLILFDWLRQ RI D 0X7 VR WKDW WKH VDPH FRPELQDWLRQV RI SDUDPHWHU YDOXHV FDQEHXVHGWRWHVWDQ\IXQFWLRQWDNLQJDJLYHQWXSOHRIGDWD W\SHV,QWKHJHQHUDOFDVHWKLVUHVXOWVLQKDYLQJQRZD\WR GHDOZLWKWHVWVWKDWSDVVZLWKQRLQGLFDWLRQRIHUURU±WKH\ FRXOG HLWKHU EH QRQH[FHSWLRQDO WHVW FDVHV LQ ZKLFK DOO SDUDPHWHU YDOXHV IDOO ZLWKLQ QRUPDO H[SHFWHG UDQJHV  RU 6LOHQWIDLOXUHVGHSHQGLQJRQWKHDFWXDOIXQFWLRQDOLW\RIWKH 0X7+RZHYHUWKHDYDLODELOLW\RIDQXPEHURIRSHUDWLQJ V\VWHPV ZLWK D VWDQGDUGL]HG $3, SHUPLWV HVWLPDWLQJ DQG UHILQLQJ UREXVWQHVV IDLOXUH UDWHV XVLQJ D YDULDWLRQ RI PXOWLSOHYHUVLRQVRIWZDUHYRWLQJ 7KHUH KDYH EHHQ PDQ\ HIIRUWV WR GHILQH DQG HYDOXDWH YDULRXV PXOWLSOH YHUVLRQ YRWLQJ DOJRULWKPV DQG VFKHPHV HJ  >@>@>@   0RVW RI WKHVH IRFXV RQ UHVROYLQJ SUREOHPVZLWKWKHH[LVWLQJVWDQGDUGYRWLQJWHFKQLTXHV2I SDUWLFXODUFRQFHUQLVKRZVHSDUDWHDOJRULWKPLFDSSURDFKHV PLJKWLQGXFHGLIIHUHQWURXQGRIIHUURUVOHDGLQJWRVHSDUDWH FRUUHFWEXWGLIIHUHQWDQVZHUV :KHQ DSSO\LQJ YRWLQJ WHFKQLTXHV WR WKH GRPDLQ RI VRIWZDUH UREXVWQHVV ZH EHQHILW IURP RXU VWXGLHG GLVLQWHUHVW LQ WKH IXQFWLRQDO FRUUHFWQHVV RI WKH RXWSXW $QHVWLPDWLRQRI6LOHQWIDLOXUHUDWHV 7KXV UREXVWQHVV WHVWLQJ UHVXOWV DUH RQO\ FRQFHUQHG ZLWK ZKHWKHUSDUDPHWHUYDOXHVZHUHLQIDFWH[FHSWLRQDOZKHWKHU 2QFH WKH QRQH[FHSWLRQDO WHVWV ZHUH UHPRYHG D H[FHSWLRQDOYDOXHVZHUHGHWHFWHGDQGZKHWKHUDWOHDVWRQH GLIIHUHQW YDULDWLRQ RQ PXOWLYHUVLRQ VRIWZDUH FRPSDULVRQ RI 1 26 YHUVLRQV UHVSRQGHG WR H[FHSWLRQDO YDOXHV ZDVXVHGWRGHWHFW6LOHQW)DLOXUHV7KHKHXULVWLFXVHGZDV JUDFHIXOO\ EXW QRW WR ZKHWKHU WKH IXQFWLRQ SHUIRUPHG DV WKDWLIDWOHDVWRQH26UHWXUQVDQHUURUFRGHWKHQDOORWKHU VSHFLILHG ZKLFK LV REYLRXVO\ LPSRUWDQW EXW LV PRUH RSHUDWLQJ V\VWHPV VKRXOG HLWKHU UHWXUQ DQ HUURU FRGH RU SURSHUO\ WKH VXEMHFW RI WUDGLWLRQDO VRIWZDUH WHVWLQJ   7KXV IRU RXU 1RUPDOL]HG)DLOXUH5DWHVIRU326,;&DOOV SXUSRVHVLWVXIILFHVLIDWOHDVWRQHRI $,; 1 YHUVLRQV SHUIRUPV H[FHSWLRQ '81,;

GHWHFWLRQ DQG UHWXUQV JUDFHIXOO\ VR '81,;'

ZH XVH D RQHRI1 FRPSDULVRQ )UHH%6' VWUDWHJ\ UDWKHU WKDQ UHTXLULQJ DQ 0 +38; $ERUW RI1 PDMRULW\ YRWLQJ VFKHPH  )RU +38; 6LOHQW

H[DPSOHLIDQ\26YHUVLRQUHSRUWVDQ 5HVWDUW ,UL[ H[FHSWLRQ FRQGLWLRQ LW LV DVVXPHG &DWDVWURSKLF

,UL[

WKDWDOOYHUVLRQVVKRXOGKDYHGHWHFWHG /LQX[ WKDW H[FHSWLRQ IDOVH DODUP UDWHV DUH /\Q[

GLVFXVVHGVKRUWO\  1HW%6'

(OLPLQDWLRQRIQRQH[FHS WLRQDOWHVWV %DOOLVWD XVHV FRPELQDWLRQV RI H[FHSWLRQDO DQG QRQH[FHSWLRQDO SDUDPHWHUYDOXHVWRGRWHVWLQJ3DUWO\ WKLV LV WR DYRLG FRUUHFW GHWHFWLRQ RI H[FHSWLRQDOYDOXHVIRURQH SDUDPHWHU



41; 41; 6XQ26 6XQ26













1RUPDOL]HG)DLOXUH5DWHDIWHUDQDO\VLV

)LJXUH 0XOWLYHUVLRQFRPSDULVRQVHOLPLQDWHWKHHIIHFWVIURPQRQH[FHSWLRQDO WHVWVDQGSHUPLWHVWLPDWLQJ6LOHQWIDLOXUHUDWHV

.RRSPDQ 'H9DOH)7&6

VXIIHUVRPHIRUPRIUREXVWQHVVIDLOXUHW\SLFDOO\DQ$ERUW IDLOXUH $VDQH[DPSOHZKHQDWWHPSWLQJWRFRPSXWHWKH ORJDULWKP RI ]HUR $,; +38; DQG ERWK YHUVLRQV RI 41; IDLOHG WR UHWXUQ DQ HUURU FRGH ZKHUHDV RWKHU RSHUDWLQJ V\VWHPV WHVWHG GLG UHSRUW DQ HUURU FRGH  7KLV LQGLFDWHG WKDW $,; +38; DQG 41; KDG VXIIHUHG 6LOHQWUREXVWQHVVIDLOXUHV 2I FRXUVH WKH KHXULVWLF RI GHWHFWLRQ EDVHG RQ D VLQJOH 26 UHSRUWLQJ DQ HUURU FRGH LV QRW D FRPSOHWHO\ DFFXUDWH PHFKDQLVP  0DQXDO UDQGRP VDPSOLQJ RI VHYHUDO GR]HQ UHVXOWV LQGLFDWHG WKDW DSSUR[LPDWHO\  RI GHWHFWHG WHVW FDVHV ZHUH DFWXDOO\ 6LOHQW IDLOXUHV  2I WKH  RI WHVW FDVHVWKDWZHUHIDOVHDODUPV ‡ ZHUHGXHWR326,;SHUPLWWLQJGLVFUHWLRQLQKRZ WR KDQGOH DQ H[FHSWLRQDO VLWXDWLRQ  )RU H[DPSOH PSURWHFW LVSHUPLWWHGEXWQRWUHTXLUHGWRUHWXUQ DQHUURULIWKHDGGUHVVRIPHPRU\VSDFHGRHVQRWIDOORQ DSDJHERXQGDU\ ‡ ZHUHGXHWREXJVLQ&OLEUDU\IORDWLQJSRLQWURX WLQHVUHWXUQLQJIDOVHHUURUFRGHV)RUH[DPSOH,UL[ UHWXUQVDQHUURUIRUWDQ LQVWHDGRIWKHFRUUHFW UHVXOWRI7ZRLQVWDQFHVZHUHIRXQGWKDWDUH OLNHO\GXHWRRYHUIORZRILQWHUPHGLDWHUHVXOWV+38; UHWXUQVDQHUURUFRGHIRUIPRG'%/B0$;3, DQG 41;UHWXUQVDQHUURUIRUOGH[SH  ‡ ZHUHGXHWRDILOHV\VWHPEXJLQ41;ZKLFK LQFRUUHFWO\UHWXUQHGHUURUVIRUILOHQDPHVKDYLQJHPEHG GHGVSDFHV ‡ 7KHUHPDLQLQJZHUHLQVWDQFHVLQZKLFKLWZDVQRW REYLRXV ZKHWKHU DQ HUURU FRGH FRXOG UHDVRQDEO\ EH UHTXLUHG WKLV ZDV PDLQO\ D FRQFHUQ ZKHQ SDVVLQJ D SRLQWHU WR D VWUXFWXUH FRQWDLQLQJ JDUEDJH GDWD ZKHUH VRPHRSHUDWLQJ V\VWHPV VXFKDV6XQ26 DSSDU HQWO\FKHFNHGWKHGDWDIRUYDOLGLW\ZKLOHRWKHUVGLGQRW &ODVVLI\LQJ WKH 6LOHQW IDLOXUHV VDPSOHG UHYHDOHG VRPH DGGLWLRQDOVRIWZDUHGHIHFWVWKDWPDQLIHVWHGLQXQXVXDOEXW VSHFLILHG VLWXDWLRQV  )RU LQVWDQFH 326,; UHTXLUHV LQW IGDWDV\QFKLQW ILOHGHV  WR UHWXUQ WKH (%$') HUURULIILOHGHVLVQRWYDOLGDQGLIWKHILOHLVQRWRSHQ IRUZULWH>@@ KRZHYHU RQO\ )UHH%6' /LQX[ DQG 6XQ26  UHWXUQHG VXFFHVVIXOO\DIWHUDQDWWHPSWWRZULWHGDWDWRDILOHSDVWLWV (2) ZKLOH HYHU\ RWKHU LPSOHPHQWDWLRQ UHWXUQHG (%$') 0DQXDO FKHFNLQJ RI UDQGRP VDPSOHVRI RSHUDWLQJ V\VWHP FDOOV LQGLFDWHG WKH IDLOXUH UDWHV FDXVHG E\ WKHVH SUREOHPV UDQJHGIURPWRRYHUDOO $ VHFRQG DSSURDFK ZDV DWWHPSWHG IRU GHWHFWLQJ 6LOHQW IDLOXUHV EDVHG RQ YRWLQJ VXFFHVVIXO UHWXUQV DJDLQVW $ERUW

IDLOXUHV LQ IXQFWLRQV IRU ZKLFK QR HUURU FRGHV ZHUH UHWXUQHG7RRXUVXUSULVHWKLVZDVRQO\VRPHZKDWHIIHFWLYH DWLGHQWLI\LQJ6LOHQWIDLOXUHVEXWGLGWXUQRXWWREHDIUXLWIXO ZD\WRUHYHDOVRIWZDUHGHIHFWV$UHODWLYHO\VPDOOQXPEHU  RIWHVWFDVHVJHQHUDWHGDQ$ERUWIDLOXUHIRUVRPH RSHUDWLQJV\VWHPVEXWVXFFHVVIXOO\FRPSOHWHGIRUDOORWKHU RSHUDWLQJ V\VWHPV  $ UDQGRPO\ VDPSOHG KDQG DQDO\VLV LQGLFDWHG WKDW WKLV GHWHFWLRQ PHFKDQLVP ZDV LQFRUUHFW DSSUR[LPDWHO\RIWKHWLPH 3DUWRIWKHKLJKIDOVHDODUPUDWHIRUWKLVVHFRQGDSSURDFK ZDVGXHWRGLIIHULQJRUGHUVIRUFKHFNLQJDUJXPHQWVDPRQJ WKHYDULRXVRSHUDWLQJV\VWHPV)RUH[DPSOHZULWLQJ]HUR E\WHVIURPD18//SRLQWHUPHPRU\ORFDWLRQPLJKW$ERUW LI D E\WH LV IHWFKHG IURP PHPRU\ EHIRUH FKHFNLQJ UHPDLQLQJOHQJWKWRWUDQVIHURUUHWXUQVXFFHVVIXOO\LIOHQJWK LVFKHFNHGEHIRUHWRXFKLQJPHPRU\ 7KHRWKHUSDUWRIWKHIDOVHDODUPUDWHZDVDSSDUHQWO\GXH WR SURJUDPPLQJ HUURUV LQ IORDWLQJ SRLQW OLEUDULHV  )RU LQVWDQFH )UHH%6' VXIIHUHG DQ $ERUW IDLOXUH RQ ERWK IDEV'%/B0$; DQGIDEV'%/B0$;  )LJXUH  VKRZV WKH DJJUHJDWH UHVXOWV RI 6LOHQW IDLOXUHV IURP PXOWLSOH YHUVLRQ FRPSDULVRQV ZLWK HUURU FRGHV ZHLJKWHG DW   SOXV 6LOHQW IDLOXUHV IURP PXOWLSOH YHUVLRQFRPSDULVRQVZLWKRQO\$ERUWIDLOXUHVZHLJKWHGDW    7KHVH UHVXOWV DUH REYLRXVO\ DSSUR[LPDWHG EXW GR LQGLFDWHWKDW6LOHQWHUURUVFDQEHSUHYDOHQW:LWKUHVSHFWWR WKHHDUOLHUGLVFXVVLRQRISRWHQWLDO$,;6LOHQWHUURUVLWZDV IRXQG WKDW HUURU FKHFNLQJ LQ RWKHU DUHDV DSSDUHQWO\ PDGH XS IRU ODFN RI HUURU GHWHFWLRQ RQ 18// SRLQWHU UHDGV JLYLQJ LW D 6LOHQW IDLOXUH UDWH FRPSDUDEOH WR VHYHUDO RWKHU V\VWHPVLQFOXGLQJ)UHH%6'

0XOWLYHUVLRQFRPSDULVRQRI26GLYHUVLW\ $Q DGGLWLRQDO XVH IRU PXOWLYHUVLRQ FRPSDULVRQ WHFKQLTXHVLVLQDWWHPSWLQJWRTXDQWLI\WKHGLYHUVLW\DPRQJ 26 LPSOHPHQWDWLRQV ZLWK UHVSHFW WR H[FHSWLRQ KDQGOLQJ UREXVWQHVV  $QDO\VLV LQ WKLV VHFWLRQ ZDV SHUIRUPHG RQ D VXEVHWRIWKHUREXVWQHVVWHVWLQJGDWDWHVWVFDVHVRQ HDFK RI  RSHUDWLQJ V\VWHPV IRU D WRWDO RI  WHVW UHVXOWV LQ DOO  7ZR 26HV 41;  DQG ,UL[   ZHUH HOLPLQDWHG EHFDXVH WKH\ GLG QRW VXSSRUW PDQ\ RI WKH IXQFWLRQV VXSSRUWHG E\ RWKHU 26HV DQG VLPLODUO\ PDQ\ IXQFWLRQVZHUHHOLPLQDWHGEHFDXVHWKH\ZHUHVXSSRUWHGE\ IHZ26HV7KHVHOHFWHG26HVDQGIXQFWLRQVPD[LPL]HWKH QXPEHURIXVDEOH WHVW FDVHVFRQVWUDLQHG E\ DUHTXLUHPHQW RIHYHU\WHVWFDVHEHLQJVXSSRUWHGE\DOO26HVXVHGLQWKH FRPSDULVRQV ,Q)LJXUHWKHKRUL]RQWDOD[LVLQGLFDWHVWKHYDOXHRI1 IRU 1ZD\ FRPSDULVRQV IURP 1  GDWD SRLQWV IRU  SRVVLELOLWLHVRQHIRUHDFKVLQJOH26 WKURXJKLQWHUPHGLDWH YDOXHV VXFK DV  GHSLFWLQJ DOO FRPELQDWLRQV RI  26HV WDNHQ  DW D WLPH  WR D VLQJOH GDWD SRLQW IRU ZD\ 26

.RRSPDQ 'H9DOH)7&6

UDWH  HVSHFLDOO\ WDNLQJ LQWR DFFRXQWWKDWWKHWULDQJOHSRLQWVDUH IRU DJJUHJDWH EHKDYLRU RI  IXQFWLRQV WKDW DOVR LQFOXGHV WKH GDWDIRUWKHV\VWHPFDOOV )RU ERWK & OLEUDU\ IXQFWLRQV DQG V\VWHP FDOOV WKHUH LV D VLJQLILFDQW VSDQ RI UREXVWQHVV IDLOXUH UDWHV XS WKURXJK PRGHUDWHO\ KLJK YDOXHV RI 1 DV ZHOO DV D QRWLFHDEOH GHFUHDVH LQ WKH ORZHVW SRVVLEOH UHVLGXDO IDLOXUH UDWH IRU LQFUHDVLQJ 1 7KHVH WUHQGV LQGLFDWH WKDW DQ LQFUHDVLQJ DPRXQW RI GLYHUVLW\ LQ H[FHSWLRQ KDQGOLQJ UREXVWQHVV LV DYDLODEOHIRULQFUHDVLQJ12USXW DQRWKHU ZD\ )LJXUH  VXJJHVWV WKDW FDUHIXO VHOHFWLRQ RI 1 26 LPSOHPHQWDWLRQV UHGXFHV WKH FRPPRQPRGHUREXVWQHVVIDLOXUHV REVHUYHG DV 1 LQFUHDVHV )LJXUH   0XOWLYHUVLRQ FRPSDULVRQV RI DOO FRPELQDWLRQV RI 1 26 LPSOHPHQWDWLRQV )XUWKHUPRUH ZKLFK 1 26 UHYHDOZLGHYDULDWLRQVLQUREXVWQHVVIDLOXUHUDWHVDQGH[FHSWLRQKDQGOLQJGLYHUVLW\ LPSOHPHQWDWLRQV RQH VHOHFWV GUDPDWLFDOO\DIIHFWV WKHGHJUHH RI FRPSDULVRQV  )RU HDFK YDOXH RI 1 WKHUH DUH WZR VHWV RI FRPPRQPRGHIDLOXUHREVHUYHG GDWDWULDQJOHVIRUDOOFDOOVDQGFLUFOHVIRUWKHVXEVHWRI 7RDGHJUHHRQHPLJKWH[SHFWDZLGHVSUHDGRIUHVLGXDO V\VWHPFDOOVH[FOXGLQJ&OLEUDU\FDOOV 7KHGDWDSRLQWV IDLOXUHUDWHVGXHWRWKHIDFWWKDW%6'DQG6\VWHP98QL[ IRU PRVWYDOXHV RI 1EOHQG WRJHWKHU EXW WKHUHVXOWLVRQH LPSOHPHQWDWLRQV KDYH VLJQLILFDQWO\ GLYHUJHG DQG WKDW WKDW VKRZV WKH VSDQ RI HIIHFWLYHQHVV IRU YDULRXV /LQX[ ZDV VSHFLILFDOO\ GHYHORSHG IURP VFUDWFK WR DYRLG FRPELQDWLRQV ZLWK WKH ORZHVW SRLQW EHLQJ WKH ORZHVW OLFHQVLQJHQWDQJOHPHQWV,QIDFWWKHGDWDIRUV\VWHPFDOOV SRVVLEOH UREXVWQHVV IDLOXUH UDWH IRU WKH EHVW SRVVLEOH VXJJHVW WKDW D KDOIGR]HQ LPSOHPHQWDWLRQV KDYH HYROYHG K\SRWKHWLFDOFRPELQDWLRQRI126YHUVLRQVH[HFXWHGEDFN VLJQLILFDQWGLYHUVLW\IURPHDFKRWKHUZLWKUHVSHFWWRV\VWHP WREDFN PD[LPXP DYDLODEOH GLYHUVLW\  DQG WKH KLJKHVW FDOOV WKH ERWWRPPRVW FLUFOH GHFUHDVHV LQ UHVLGXDO IDLOXUH SRLQW EHLQJ WKH ZRUVW RQH FRXOG GR E\ K\SRWKHWLFDOO\ UDWH VLJQLILFDQWO\ WKURXJK DERXW 1    6LPLODUO\ WKH IDFW SLFNLQJWKHVHWRI126YHUVLRQVZLWKWKHKLJKHVWGHJUHHRI WKDW WKH *18 & OLEUDULHV DQG %6'EDVHG OLEUDULHV ZHUH UREXVWQHVVIDLOXUHRYHUODSPLQLPXPDYDLODEOHGLYHUVLW\  LQGHSHQGHQWO\GHYHORSHGKHOSHGSURYLGHGLYHUVLW\ 7KXVLQ)LJXUHWKHKHLJKWRIDQ\SDUWLFXODUGDWDSRLQW +RZHYHUWKHSRWHQWLDOEDGQHZVIURPDGLYHUVLW\SRLQW PHDVXUHVWKHUREXVWQHVVIDLOXUHUDWHQRWGHWHFWHGE\DQ\RI RI YLHZ LV WKDW QR WZR 26 LPSOHPHQWDWLRQV DSSURDFK WKH WKH 1 26 LPSOHPHQWDWLRQV IRU WKDW SRLQW  7KH VSDQ RI V\VWHP FDOO GLYHUVLW\ RI DOO  26 LPSOHPHQWDWLRQV SRLQWV IRU D SDUWLFXODU 1 YDOXH LQGLFDWHV WKH GLIIHUHQFH LQ 6XSSRVHWKDWRQHZHUHWRLQWKHEHVWFDVHVHOHFWWKHWZR GLYHUVLW\ DPRQJ FRPELQDWLRQV OHDVW GLYHUVH EHLQJ WKH PRVW GLYHUVH 26 LPSOHPHQWDWLRQV $,; DQG )UHH%6' KLJKHVW   7KH JHQHUDO GLIIHUHQFH EHWZHHQ WULDQJOHV DQG ZKLFK DOVR DSSHDUHG LQ SHUVRQDO FRUUHVSRQGHQFH WR KDYH FLUFOHV LQGLFDWHV GLIIHUHQFHV EHWZHHQ DJJUHJDWH 326,; WKHPRVWH[WUHPHYLHZVDVWRZKDWFRQVWLWXWHGUREXVWQHVV  FDOOV DQG D V\VWHPFDOORQO\ VXEVHW RI WKH 326,; $3, ,I RQH ZHUH WR PDNH WKDW VHOHFWLRQ ZLWK WKH LQWHQW RI )LQDOO\WKHGRZQZDUGWUHQGLQIDLOXUHUDWHVDV1LQFUHDVHV DYRLGLQJDOOFRPPRQPRGH26H[FHSWLRQKDQGOLQJIDLOXUHV LQGLFDWHVWKHDGGLWLRQDOGLYHUVLW\DGGHGE\HDFKDGGLWLRQDO DPRQJ PXOWLSOH PDFKLQHV UXQQLQJ WKH VDPH DSSOLFDWLRQ 26 FRQVLGHUHG LQ D SRRO RI 1 26 LPSOHPHQWDWLRQV LH VRIWZDUH RQH ZRXOG RQO\ DFKLHYH D  FRPPRQ PRGH UHPRYLQJ DGGLWLRQDO FRPPRQ PRGH IDLOXUHV IRU D IDLOXUHH[SRVXUHIRUV\VWHPFDOOVDQGDH[SRVXUHIRU K\SRWKHWLFDOFRPSRVLWHH[FHSWLRQGHWHFWLRQV\VWHP  DOO FDOOV WHVWHG DVVXPLQJ HTXDO ZHLJKWLQJ DPRQJ DOO )URP )LJXUH  RQH FDQ VHH WKDW V\VWHP FDOOV DUH IXQFWLRQV  $QG LI RQH ZHUH WR SLFN DQ\ WZR RWKHU 26 JHQHUDOO\ IDU PRUH UREXVW WKDQ & OLEUDU\ FDOOV H[FHSW IRU LPSOHPHQWDWLRQV WKH FRPPRQ PRGH IDLOXUH H[SRVXUH WKH 1  SRLQW IRU 41;  DW  UREXVWQHVV IDLOXUH ZRXOGEHKLJKHU6HOHFWLQJWKHEHVWIRXURUILYHRUVL[26

.RRSPDQ 'H9DOH)7&6

LPSOHPHQWDWLRQV LPSURYHV WKH VLWXDWLRQ WR D GHJUHH EXW EHFRPHVLQFUHDVLQJO\LPSUDFWLFDOLQDUHDOLQVWDOODWLRQ $Q DGGLWLRQDO FRQFOXVLRQ IURP )LJXUH  DQG RWKHU DQDO\VLVZHKDYHSHUIRUPHGLVWKDWDVLJQLILFDQWIUDFWLRQRI $ERUW IDLOXUHV FRPH IURP & OLEUDU\ FDOOV DQG WHQG WR EH FRUUHODWHG DPRQJ 26 YHUVLRQV  7KLV LV SHUKDSV QRW D VXUSULVH FRQVLGHULQJ WKDW WKHVH FDOOV PRVW QRWDEO\ WKH VWULQJ KDQGOLQJ URXWLQHV DUH QRWRULRXVO\ VXVFHSWLEOH WR PHPRU\RYHUUXQDQGSRLQWHUYDOXHSUREOHPVDQGKDYHDQ $3, GHILQLWLRQ WKDW GRHV QRW VSHFLILFDOO\ HQFRXUDJH UREXVWQHVV6LPLODUO\LWLVQRWXQUHDVRQDEOHWRFRQMHFWXUH WKDW & OLEUDU\ FRGH KDV EHHQ VKDUHG DQG UHGLVWULEXWHG ODUJHO\XQFKDQJHGVLQFHPRVW26YHQGRUVFRQFHQWUDWHRQ SRUWLQJ DQ 26 WR WKHLU KDUGZDUH DQG RSWLPL]LQJ SHUIRUPDQFH RI V\VWHP FDOOV UDWKHU WKDQ WKH & OLEUDU\ IXQFWLRQV  1RQHWKHOHVV LI RQH ZHUH WR SUHVXPH D KLJK GHJUHH RI LPSOHPHQWDWLRQ GLYHUVLW\ VLPSO\ EHFDXVH & OLEUDULHV ZHUH SUHVXPHG WR EH  LQGHSHQGHQWO\ GHYHORSHG DQG FDPH IURP LQGHSHQGHQW YHQGRUV RQH ZRXOG EH PLVWDNHQIURPDQH[FHSWLRQKDQGOLQJSHUVSHFWLYH

)UHTXHQWVRXUFHVRIUREXVWQHVVIDLOXUH *LYHQWKDWUREXVWQHVVIDLOXUHVDUHSUHYDOHQWZKDWPLJKW EH IL[HG WR LPSURYH WKHP"  6RXUFH FRGH WR PRVW RI WKH RSHUDWLQJ V\VWHPV WHVWHG ZDV QRW DYDLODEOH DQG PDQXDO H[DPLQDWLRQ RI DYDLODEOH VRXUFH FRGH WR VHDUFK IRU URRW FDXVHV RI UREXVWQHVV IDLOXUHV LV LPSUDFWLFDO ZLWK VXFK D ODUJHVHWRIH[SHULPHQWDOGDWD7KHUHIRUHWKHEHVWWKDWFDQ EH SUHVHQWHG LV D OLVW RI GDWD YDOXHV WKDW DUH KLJKO\ FRUUHODWHGZLWKUREXVWQHVVIDLOXUHV7DEOH DQGIXQFWLRQV WKDW KDYH KLJK UREXVWQHVV IDLOXUH UDWHV HYHQ DIWHU ZD\ IDLOXUHGHWHFWLRQFRPSDULVRQVDUHDSSOLHG7DEOH  7DEOH'DWDW\SHVPRVWFRPPRQO\DVVRFLDWHGZLWK DERUWUREXVWQHVVIDLOXUHVIRURSHUDWLQJV\VWHPV 'DWD9DOXH

3HUFHQWDVVRFLDWHGZLWK UREXVWQHVVIDLOXUHV

,QYDOLGILOHSRLQWHUV H[FOXGLQJ18//



18//ILOHSRLQWHUV



,QYDOLGEXIIHUSRLQWHUV H[FOXGLQJ18//



18//EXIIHUSRLQWHUV



0,1,17LQWHJHUV



0$;,17LQWHJHUV



7DEOHVKRZVWKDW18//DQGLQYDOLGSRLQWHUYDOXHVDUH WKH PRVW FRPPRQ FDXVHV RI $ERUW IDLOXUHV ZKLFK LV SUREDEO\ QR JUHDW VXUSULVH  +RZHYHU PLQLPXP DQG PD[LPXP LQWHJHU YDOXHV DOVR VHHP WR EH FRUUHODWHG ZLWK

UREXVWQHVVIDLOXUHVZKLFKZDVQRWDQREYLRXVRXWFRPH 7DEOH  VKRZV WKH VLJMPSORQJMPS SDLU KLJK RQ WKH UREXVWQHVV IDLOXUH UDWH OLVW IRU ERWK V\VWHP FDOOV DQG & OLEUDU\FDOOV1RWHWKDW7DEOHGHDOVZLWKUHVLGXDOIDLOXUH UDWHVDIWHUFRPSDULVRQVDQGVRUHSUHVHQWVIDLOXUHVWKDWQR 26GHDOWZLWKJUDFHIXOO\%H\RQGWKDWVWULQJIXQFWLRQVDQG PDWKIXQFWLRQVDSSHDURQWKH&OLEUDU\OLVWEXWDUHQRWWKH RQO\FXOSULWV 7DEOH)XQFWLRQVZLWKKLJKHVWUHVLGXDOIDLOXUHUDWHV DIWHUZD\IDLOXUHGHWHFWLRQYRWLQJ 6\VWHP&DOOV VLJORQJMPS

&/LEUDU\&DOOV



ORQJMPS



VLJVHWMPS



VWUFS\



FWHUPLG



DWDQ



FORVHGLU



IUH[S



UHDGGLU



PRGI



JHWHQY



VHWMPS



JHWJUQDP



VSULQWI



JHWSZQDP



VWUIWLPH



JHWJUQDP



VWUQFDW



JHWFZG



VWUFDW



FUHDW



SULQWI



H[HFOS



VWUQFS\



H[HFYS



IDEV



VLJDGGVHW



WDQ



&RQFOXVLRQV 7KLVSDSHUGRFXPHQWVWKHXVHRIPXOWLYHUVLRQVRIWZDUH FRPSDULVRQWHFKQLTXHVLQDQDO\]LQJDVHWRIODUJHPDWXUH FRPPHUFLDOO\ DYDLODEOH VRIWZDUH V\VWHPV  ,Q WHUPV RI UREXVWQHVV DVVHVVPHQW WKH PXOWLYHUVLRQ DSSURDFK SHUPLWWHGLGHQWLI\LQJ6LOHQWIDLOXUHVIDLOXUHWRLQGLFDWHWKH RFFXUUDQFH RI DQ H[FHSWLRQDO FRQGLWLRQ ZKHQ RQH FRXOG KDYHEHHQLQGLFDWHG ZLWKDUHDVRQDEOHGHJUHHRIDFFXUDF\ EXWZLWKRXWQHHGWRFUHDWHIXQFWLRQDOVSHFLILFDWLRQVIRUDOO  IXQFWLRQV WHVWHG  7KH UHVXOW ZDV WKH GLVFRYHU\ RI D QRUPDOL]HG6LOHQWUREXVWQHVVIDLOXUHUDWHRIWRIRU VLQJOH26WHVWV$GGLWLRQDOO\DPXOWLYHUVLRQFRPSDULVRQ DSSURDFKSHUPLWWHGVFUHHQLQJRXWWKHQRQH[FHSWLRQDOWHVWV JHQHUDWHG E\ %DOOLVWD UREXVWQHVV WHVWLQJ  $Q DGGLWLRQDO XQH[SHFWHGUHVXOWZDVILQGLQJVRPHEXJVVRIWZDUHGHIHFWV ZLWKUHVSHFWWRUHTXLUHG326,;IXQFWLRQDOLW\ RQ326,; FHUWLILHGFRPPHUFLDORSHUDWLQJV\VWHPV $ VHFRQG DSSURDFK WR XVLQJ PXOWLYHUVLRQ VRIWZDUH FRPSDULVRQV SURGXFHG PHDVXUHPHQWV RI WKH GLYHUVLW\ RI 326,; RSHUDWLQJ V\VWHPV  ,Q SDUWLFXODU PHDVXUHPHQWV ZHUH PDGH IRU WKH UHVLGXDO UREXVWQHVV IDLOXUH UDWH WKDW ZRXOGUHPDLQLIRQHZHUHWRK\SRWKHWLFDOO\ FRPELQHWKH

.RRSPDQ 'H9DOH)7&6

PRVWJUDFHIXOH[FHSWLRQKDQGOLQJDELOLWLHVRIHYHU\RQHRI 1 GLIIHUHQW 26 LPSOHPHQWDWLRQV ZLWK WKH YDOXH RI 1 UDQJLQJIURPRQHWRWKLUWHHQ7KHUHVXOWVZHUHWKDWWKHFRUH VHW RI  V\VWHP FDOOV WHVWHG ZHUH PRGHUDWHO\ UREXVW DQG GLYHUVH EXW QRW SHUIHFWO\ VR 7KLV UHVXOW VXJJHVWV WKDW V\VWHPFDOOVKDYHDUHDVRQDEOHOHYHORIVRIWZDUHGLYHUVLW\ IRU PXOWLSOH YHUVLRQ SXUSRVHV RU DOWHUQDWHO\ WKDW D GHYHORSHU FRPELQLQJ WHFKQLTXHV DOUHDG\ LQ H[LVWLQJ 26 LPSOHPHQWDWLRQVLQWRDVLQJOH26PLJKWSRVVLEO\EDUULQJ WHFKQLFDO KXUGOHV  LPSURYH WKH UREXVWQHVV WKDW VLQJOH 26 +RZHYHU VHOHFWLQJ DQ\ SDLU RI 26 LPSOHPHQWDWLRQV GLG QRW VHHP WR UHVXOW LQ WKH KLJKHVW SRVVLEOH GHJUHH RI LPSOHPHQWDWLRQGLYHUVLW\ 7KH UHVXOWV IRU PXOWLYHUVLRQ DVVHVVPHQW RI & OLEUDU\ FDOOV ZHUH OHVV SURPLVLQJ  & OLEUDU\ FDOOV DSSHDU WR EH VLJQLILFDQWO\OHVVUREXVWRQDYHUDJHWKDQ26V\VWHPFDOOV DQG DGGLWLRQDOO\ GR QRW DSSHDU WR EH KLJKO\ GLYHUVH LQ LPSOHPHQWDWLRQ7KLVILQGLQJPHDQVWKDWSUHVXPLQJWKDW& OLEUDU\ LPSOHPHQWDWLRQV DUH GLYHUVH VLPSO\ EHFDXVH WKH\ FRPH IURP GLIIHUHQW YHQGRUV RU KDYH EHHQ LQGHSHQGHQWO\ GHYHORSHGLVSUREDEO\QRWDJRRGLGHD $OWKRXJKWKHGDWDSUHVHQWHGKHUHGRQRWSXUSRUWWRDSSO\ WRIXQFWLRQDOFRUUHFWQHVVWKH\GRVXJJHVWWKDWFRPPHUFLDO RIIWKHVKHOIVRIWZDUHGHYHORSHGWRDSDUWLFXODU$3,PLJKW SRVVLEO\ ODFN GLYHUVLW\ DW OHDVW ZLWK UHVSHFW WR H[FHSWLRQ KDQGOLQJ  ,W VHHPV YHU\ OLNHO\ WKDW VRPH RI WKLV ODFN RI GLYHUVLW\ LV GXH WR WKH GHVLJQ RI WKH $3, LWVHOI EXW LW LV GLIILFXOWWREHOLHYHWKDWWKLVLVWKHRQO\IDFWRUDWZRUN 7KHGHWDLOHGVRXUFHGDWDXVHGLQSUHSDULQJWKLVSDSHUDUH DYDLODEOHDWKWWSZZZLFHVFPXHGXEDOOLVWD

$FNQRZOHGJPHQW 7KLV UHVHDUFK ZDV VSRQVRUHG E\ '$53$ FRQWUDFW '$%7&WKH%DOOLVWDSURMHFW

5HIHUHQFHV >@$YL]LHQLV$7KH1YHUVLRQDSSURDFKWRIDXOWWROHUDQWVRIW ZDUH,(((7UDQVRQ6RIWZDUH(QJLQHHULQJYRO6(QR S >@$YL]LHQLV$*XQQLQJEHUJ3.HOO\-3-6WULQJLQL/ 7UDYHUVH3-7VR.69RJHV8³7KH8&/$'(',;6\VWHP $'LVWULEXWHG7HVWEHGIRU0XOWLSOH9HUVLRQ6RIWZDUH´WK)DXOW 7ROHUDQW&RPSXWLQJ6\PSS >@$YL]LHQLV$/\X06FKXW]:³,Q6HDUFKRI(IIHFWLYH 'LYHUVLW\$6L[/DQJXDJH6WXG\RI)DXOW7ROHUDQW)OLJKW&RQWURO 6RIWZDUH´WK,QWO6\PSRQ)DXOW7ROHUDQW&RPSXWLQJ >@%ULOOLDQW66.QLJKW-&/HYHVRQ1*³$QDO\VLVRI )DXOWVLQDQ19HUVLRQ6RIWZDUH([SHULPHQW´,(((7UDQVRQ 6RIWZDUH(QJLQHHULQJYROQRSS

>@&KHQ/$YL]LHQLV$³19HUVLRQ3URJUDPPLQJ$)DXOW 7ROHUDQFH$SSURDFKWR5HOLDELOLW\RI6RIWZDUH2SHUDWLRQ´7KH (LJKWK,QWO6\PSRQ)DXOW7ROHUDQW&RPSXWLQJ >@*HUVWLQJ-/1LVW5/5REHUWV'%9DQ9DONHQEXUJ 5/³$&RPSDULVRQRI9RWLQJ$OJRULWKPVIRU19HUVLRQ3UR JUDPPLQJ´3URFHHGLQJVRIWKHWZHQW\IRXUWK$QQXDO+DZDLL,QWO &RQIHUHQFHRQ6\VWHP6FLHQFHVYROSS >@,(((6WDQGDUGIRU,QIRUPDWLRQ7HFKQRORJ\3RUWDEOH2SHUDW LQJ6\VWHP,QWHUIDFH326,; 3DUW6\VWHP$SSOLFDWLRQ3UR JUDP,QWHUIDFH$3, $PHQGPHQW5HDOWLPH([WHQVLRQ>& /DQJXDJH@,(((6WGE,(((&RPSXWHU6RFLHW\  >@.HOO\-3(FNKDUGW'(9RXN0$0F$OOLVWHU')&DJ OD\DQ$.³$/DUJHVFDOH6HFRQG*HQHUDWLRQ([SHULPHQWLQ 0XOWL9HUVLRQ6RIWZDUH'HVFULSWLRQDQG(DUO\5HVXOWV´(LJK WHHQWK,QWO6\SRVLXPRQ)DXOW7ROHUDQW&RPSXWLQJ >@.QLJKW-&/HYHVRQ1*6W-HDQ/'$ODUJHVFDOH H[SHULPHQWLQ1YHUVLRQSURJUDPPLQJWK)DXOW7ROHUDQW &RPSXWLQJ6\PS >@.QLJKW/HYHVRQ$QHPSULFLDOVXW\RIIDLOXUHSUREDELOLWLHV LQPXOWLYHUVLRQVRIWZDUHWK,QWO)DXOW7ROHUDQW&RPSXWLQJ 6\PS >@.QLJKW-&/HYHVRQ1*³$UHSO\WRWKHFULWLFLVPVRIWKH .QLJKWDQG/HYHVRQH[SHULPHQW´6,*62)76RIWZDUH(QJLQHHU LQJ1RWHVYROQRS >@.RRSPDQ36XQJ-'LQJPDQ&6LHZLRUHN' 0DU] 7&RPSDULQJ2SHUDWLQJ6\VWHPV8VLQJ5REXVWQHVV%HQFK PDUNV3URFHHGLQJV6\PSRQ5HOLDEOHDQG'LVWULEXWHG6\VWHPV 'XUKDP1&2FWSS >@.URSS1.RRSPDQ3 6LHZLRUHN'$XWRPDWHG 5REXVWQHVV7HVWLQJRI2IIWKH6KHOI6RIWZDUH&RPSRQHQWVWK )DXOW7ROHUDQW&RPSXWLQJ6\PS-XQH >@/RUF]DN35&DJOD\DQ$.(FNKDUGW'(³$7KHRUHWL FDO,QYHVWLJDWLRQRI*HQHUDOL]HG9RWHUVIRU5HGXQGDQW6\VWHPV´ WK,QWO6\PSRQ)DXOW7ROHUDQW&RPSXWLQJ >@/\X05³6RIWZDUH5HOLDELOLW\0HDVXUHPHQWVLQ19HU VLRQ6RIWZDUH([HFXWLRQ(QYLURQPHQW´3URFHHGLQJVRIWKH7KLUG ,QWO6\PSRQ6RIWZDUH5HOLDELOLW\DQG(QJLQHHULQJ >@0F$OOLVWHU')6XQ&9RXN0$³5HOLDELOLW\RI9RWLQJ LQ)DXOW7ROHUDQW6RIWZDUH6\VWHPVIRU6PDOO2XWSXW6SDFHV´ ,(((7UDQVRQ5HOLDELOLW\YROQRSS >@9RXN0$0F$OOLVWHU')&DJOD\DQ$.:DONHU-/ (FNKDUGW'(³$QDO\VLVRI)DXOWV'HWHFWHGLQD/DUJH6FDOH 0XOWL9HUVLRQ6RIWZDUH'HYHORSPHQW([SHULPHQW´3URFHHGLQJV RIWKH1LQWK'LJLWDO$YLRQLFV6\VWHPV&RQIHUHQFH

.RRSPDQ 'H9DOH)7&6