Micro Macro Neural Network to Recognize Slow ...

/KETQ/CETQ0GWTCN0GVYQTMVQ4GEQIPK\G5NQY/QXGOGPV '/)DCUGF#EEWTCVGCPF3WKEM4QNNQXGT4GEQIPKVKQP



 :

 /KETQ/CETQ0GWTCN0GVYQTMVQ4GEQIPK\G5NQY /QXGOGPV'/)DCUGF#EEWTCVGCPF3WKEM 4QNNQXGT4GEQIPKVKQP 

7DNHVKL$QGR-XQ2NDPRWRDQG0DVDNDWVX*)XMLH )DFXOW\RI6FLHQFHDQG(QJLQHHULQJ:DVHGD8QLYHUVLW\ -DSDQ



+PVTQFWEVKQP 

7KH ZHDUDEOH URERWV WR VXSSRUW PDQ\ NLQGV RI PRYHPHQWV KDYH EHHQ GHYHORSHG IRU WKH HOGHU DQG GLVDEOHG SHRSOH DOO RYHU WKH ZRUOG +D\DVKL HW DO  )XUXVKR HW DO  .DZDPXUD HW DO   EHFDXVH ZH DUH IDFLQJ WKH HOGHU GRPLQDWHG VRFLHW\ $ VXUIDFH (OHFWUR0\R*UDP(0* VLJQDOZKLFKLVPHDVXUHGDOLWWOHEHIRUHWKHVWDUWRIWKHPRYHPHQW LVH[SHFWHGDVWKHWULJJHUVLJQDORIPRYHPHQWVXSSRUW :HKDYHEHHQDOVRGHYHORSLQJDQ(0*FRQWUROOHGLQWHOOLJHQWWUXQNFRUVHWVKRZQLQ)LJ WRVXSSRUWUROORYHUPRYHPHQWVLQFHLWLVRQHRIWKHPRVWLPSRUWDQWDFWLYLWLHVRIGDLO\OLYLQJ $'/  (VSHFLDOO\ WKH UROORYHU PRYHPHQW RI ERQH FDQFHU PHWDVWDVLV SDWLHQWV LV IRFXVHG DV WKHWDUJHWPRYHPHQW7KHERQHFDQFHUPHWDVWDVLVSDWLHQWVIHHOVHYHUSDLQZKHQWKH\FRQGXFW WKHUROORYHUPRYHPHQW7KHFRUHRIWKHLQWHOOLJHQWWUXQNFRUVHWV\VWHPLVDSQHXPDWLFUXEEHU PXVFOHWKDWLVRSHUDWHGE\WKH(0*VLJQDOVIURPWKHWUXQNPXVFOH$VVKRZQLQ)LJLQ RXU VWXG\ ZH ILUVW DQDO\]HG WKH (0* VLJQDO $QGR HW DO   WKDW LV XVHG DV WKH LQSXW VLJQDOIRUWKHLQWHOOLJHQWFRUVHWWRUHFRJQL]HDUROORYHUPRYHPHQW6HFRQGZHSURSRVHGDQ RULJLQDOQHXUDOQHWZRUNDOJRULWKPWRUHFRJQL]HWKHUROORYHUTXLFNO\DQGZLWKKLJKDFFXUDF\ $QGRHWDOD )LQDOO\ZHGHYHORSHGWKHPHFKDQLVPVRIWKHLQWHOOLJHQWFRUVHWWRDVVLVW UROORYHUPRYHPHQWXVLQJWKHSQHXPDWLFUXEEHUDFWXDWRU$QGRHWDOE  ,Q WKLV FKDSWHU WKH SURSRVHG RULJLQDO QHXUDO QHWZRUN FDOOHG WKH 0LFUR0DFUR 1HXUDO 1HWZRUN 0011  LV LQWURGXFHG ,Q DGGLWLRQ WKH PHWKRGRORJ\ WR GHWHUPLQH WKH RSWLPDO VWUXFWXUH RI WKH 0011 WR UHFRJQL]H WKH UROORYHU PRYHPHQW LV HVWDEOLVKHG 7KLV SDSHU LV RUJDQL]HG DV IROORZV 6HFWLRQ  VXPPDUL]HV WKH UHODWHG QHXUDO QHWZRUN WR UHFRJQL]H VRPH PRYHPHQWV EDVHG RQ WKH (0* VLJQDO 6HFWLRQ  GLVFXVVHV WKH WUDGLWLRQDO QHXUDO QHWZRUN NQRZQDV7LPH'HOD\1HXUDO1HWZRUN7'11 DQG0011VWUXFWXUHV6HFWLRQHVWDEOLVK WKHPHWKRGRORJ\WRGHWHUPLQHWKHRSWLPDOVWUXFWXUHRI0011DQGWKHUROORYHUUHFRJQLWLRQ UHVXOW XVLQJ WKH RSWLPDO0011LVFRPSDUHG ZLWK WKDW XVLQJ WUDGLWLRQDO 7'11 6HFWLRQ  SUHVHQWVDVXPPDU\DQGIXWXUHZRUN





4GEGPV#FXCPEGUKP$KQOGFKECN'PIKPGGTKPI

4GNCVGFPGWTCNPGVYQTMVQTGEQIPK\GOQXGOGPVWUKPI'/)UKIPCN 

6LQFH WKH UHFRJQLWLRQ RI UROORYHU LV EDVHG RQ QRLV\ DQG FRPSOH[ (0* VLJQDOV D KLJKO\ UREXVW V\VWHP WKDW LV XQDIIHFWHG E\ WKH SRVVLEOH PLVDOLJQPHQW RI HOHFWURGHV LQGLYLGXDO GLIIHUHQFHV RU VXUURXQGLQJ HOHFWULFDO FRQGLWLRQV LV QHFHVVDU\ WR UHFRJQL]H (0* VLJQDOV DFFXUDWHO\$1HXUDO1HWZRUN11 LVRQHRIWKHOHDUQLQJPDFKLQHVWKDWXVH(0*VLJQDOVWR UHFRJQL]HPRYHPHQW.XULED\DVKLHWDO)XNXGDHWDO:DQJHWDO.LJXFKL HW DO  +RX HW DO  =HFFD HW DO   11 LV FDSDEOH RI QRQOLQHDU PDSSLQJ JHQHUDOL]DWLRQDQGDGDSWLYHOHDUQLQJ7KHUHDUHJHQHUDOO\WZRNLQGVRI11WKDWUHFRJQL]HD WLPHVHULHVVLJQDO2QHLVWKH7LPH'HOD\1HXUDO1HWZRUN7'11 :DLEHO LQZKLFK D GHOD\ LV LQWURGXFHG LQ WKH QHWZRUN DQG SDVW GDWD WKH GDWD FROOHFWHG EHIRUH WKH FXUUHQW PHDVXUHPHQW SRLQW  LV VHW DV WKH LQSXW VLJQDO RI WKH QHWZRUN 7KH RWKHU LV WKH 5HFXUUHQW 1HXUDO1HWZRUN511 .HOO\HWDO7VXMLHWDO ZKLFKXVHVIHHGEDFNIURPWKH RXWSXW VLJQDO RI WKH RXWSXW OD\HU DV WKH LQSXW VLJQDO RI WKH LQSXW OD\HU 7R DYRLG QHHGOHVV WLPHVWUHWFKSURSHUWLHVDQGWRUHGXFHFDOFXODWLRQDPRXQWVDQGFRVWVZHVHOHFWHG7'11DV WKHEDVHQHXUDOQHWZRUNIRUWKHZRUNUHSRUWHGKHUH 0DQ\UHVHDUFKHUVKDYHXVHG7'11WRUHFRJQL]HPRYHPHQWVIURP(0*VLJQDOV)RUH[DPSOH +LQFDSLHHWDO+LQFDSLHHWDO HVWLPDWHGWKHPRYHPHQWRIWKHDIIHFWHGVLGHRIDSDWLHQW E\ XVLQJ (0* GDWD RI WKH XQDIIHFWHG VLGH LQ WKHLU GHYHORSPHQW RI D SURVWKHWLF XSSHU OLPE +LUDNDZD HW DO +LUDNDZD HW DO   DQG )DUU\ HW DO )DUU\ HW DO   UHFRJQL]HG PRYHPHQWXVLQJIUHTXHQF\GRPDLQLQIRUPDWLRQRIWKH(0*VLJQDO+XDQJHWDO+XDQJHWDO   SURSRVHG WKH IHDWXUH YHFWRU FRPSRVHG RI DQ LQWHJUDWHG (0* =HUR &URVVLQJ DQG YDULDQFHWRUHFRJQL]HHLJKWILQJHUPRYHPHQW)LQDOO\1LVKLNDZDHWDO1LVKL]DZDHWDO  UHFRJQL]HGWHQNLQGVRIPRYHPHQWVXVLQJD*DERUWUDQVIRUPHG(0*VLJQDO

)LJ,QWHOOLJHQWWUXQNFRUVHWWRVXSSRUWUROORYHUPRYHPHQW





/KETQ/CETQ0GWTCN0GVYQTMVQ4GEQIPK\G5NQY/QXGOGPV '/)DCUGF#EEWTCVGCPF3WKEM4QNNQXGT4GEQIPKVKQP



)LJ  &RQFHSW RI WKH LQWHOOLJHQW FRUVHW XVLQJ (0* VLJQDO RULJLQDO QHXUDO QHWZRUN DQG SQHXPDWLFDFWXDWRU  +RZHYHUDOORIWKHVHUHODWHGUHVHDUFKHIIRUWVVKDUHWZRFRPPRQSUREOHPVZKLFKDUHVORZ UHVSRQVH WLPH DQG LQFRUUHFW UHFRJQLWLRQ RI WKH PRYHPHQW &RQVHTXHQWO\ ZH SUHYLRXVO\ SURSRVHG WKH RULJLQDO DOJRULWKP FDOOHG WKH 0LFUR 0DFUR 1HXUDO 1HWZRUN 0011  FRPSRVHGRIWKH0LFUR3DUWZKLFKGHWHFWVDUDSLGFKDQJHLQWKHVWUHQJWKRIWKH(0*VLJQDO DQG WKH 0DFUR 3DUW ZKLFK GHWHFWV WKH WHQGHQF\ RI WKH (0* VLJQDO WRZDUG D FRQWLQXLQJ LQFUHDVH RU FRQWLQXLQJ GHFUHDVH WR LPSURYH WKH UHVSRQVH WLPH DQG DFFXUDWH UHFRJQLWLRQ RI WKH UROORYHU PRYHPHQW EDVHG RQ WKH (0* VLJQDO DV LQSXW +RZHYHU WKH PHWKRGRORJ\ WR GHVLJQ RU RSWLPL]H WKH VWUXFWXUH RI WKH 0011LV QRW HVWDEOLVKHG EHFDXVH WKHUH DUH PDQ\ SDUDPHWHUVWRGHWHUPLQHWKHVWUXFWXUHRIWKH0011



/KETQ/CETQ0GWTCN0GVYQTM //00 

6TCFKVKQPCN6KOG&GNC[0GWTCN0GVYQTM )RUWKHOHDUQLQJPDFKLQHLQWKLVUHVHDUFKZH VHOHFWHGWKHWKUHHOD\HUIHHGIRUZDUGW\SHRI 7LPH 'HOD\ 1HXUDO 1HWZRUN 7'11  DV WKH VWUXFWXUH RI WKH QHWZRUN DQG WKH EDFN SURSDJDWLRQ %3  PHWKRG ZLWK D PRPHQWXP WHUP DV WKH OHDQLQJ DOJRULWKP ZKLFK LV D VWDQGDUG QHXUDO QHWZRUN WR UHFRJQL]H WLPHVHULHV VLJQDOV ,Q DGGLWLRQ ZH VHOHFWHG WKH VHTXHQWLDO DGMXVWPHQW PHWKRG WR PRGLI\ WKH ZHLJKW DQG WKUHVKROG RI HDFK XQLW 7KH UHODWLRQVEHWZHHQHDFKSDLURIXQLWVLQWKH7'11DUHVKRZQLQ  DQG   QP 

¦Ȧ

QHWLP

M 

P LM

[ PM  șLP 

 



[LP 

I QHW LP 

I QHW    H[SX  QHW 



   



4GEGPV#FXCPEGUKP$KQOGFKECN'PIKPGGTKPI

ZKHUH P   DQG  L  «QP QP LV WKH QXPEHU RI WKH PWK OD\HU XQLW ZPLM LV WKH ZHLJKW EHWZHHQWKHP WKOD\HU·VLWKXQLWDQGWKHPWKOD\HU·VMWKXQLW[PLLLVWKHRXWSXWRIWKHPWK OD\HU·VLWKXQLW TPLLVWKHWKUHVKROGLQWKHPWKOD\HU·VLWKXQLWDQGXLVWKHFRQVWDQWWRGHFLGH WKHJUDGLHQWRIWKHVLJPRLGIXQFWLRQ  ,QWKLVVWXG\WKHQXPEHURILQSXWOD\HUXQLWVZDVW\SLFDOO\DQGWKDWLVWKHLQSXWRIWKH LQSXW OD\HU ZDV (0* VLJQDOV VHPJ WL  L   ,Q RWKHU ZRUGV WKH WLPH LW WRRN WKH 7'11V\VWHPWRUHFRJQL]HWKHUROORYHUPRYHPHQWIURPWKHLQSXWWHG(0*GDWDZDV PVHF =HFFDHWDO 



%QPEGRVQH/KETQ/CETQ0GWTCN0GVYQTM 8VLQJ7'11SUHYLRXVUHVHDUFKHUVIRFXVHGRQXSSHUOLPEPRYHPHQWZKLFKLVDUHODWLYHO\ IDVW PRYHPHQW 6LQFH WKH PRYHPHQW WDNHV RQO\ D VKRUW WLPH OHVV WLPHVHULHV (0* GDWD LV LQSXWWHG LQWR WKH V\VWHP 7KH DGYDQWDJH RI WKLV VKRUW GDWD OHQJWK LV WKDW WKHUH DUH IHZHU FDOFXODWLRQV WR EH GRQH DQG WKHUHIRUH OHVV FRVW WKH GLVDGYDQWDJH LV WKDW OHVV LQSXW GDWD PHDQVPRUHIDOVHUHFRJQLWLRQV :H IRFXVHG RQ WKH UROORYHU PRYHPHQW ZKLFK LV D UHODWLYHO\ VORZ PRYHPHQW 6LQFH WKH PRYHPHQW WDNHV D UHODWLYHO\ ORQJ WLPH LW LV SRVVLEOH WR KDYH PRUH WLPHVHULHV (0* GDWD LQSXWWHGLQWRWKHV\VWHP :HFKHFNHGWKHLPSDFWRISDVWWLPHVHULHV(0*GDWDXVLQJ7'11RQWKHUHFRJQLWLRQUHVXOW 7KH VWUXFWXUH RI 7'11 ZDV DV IROORZV WKH QXPEHU RI LQSXW OD\HU XQLWV ZDV  WKH QXPEHU RI KLGGHQ OD\HU XQLWV ZDV  DQG WKH QXPEHU RI RXWSXW OD\HU XQLWV ZDV  :H GHWHUPLQHGWKHQXPEHURILQSXWXQLWVDVEDVHGRQRXU(0*H[SHULPHQW$QGRHWDO   ZKLFK VKRZHG WKDW WKH VKRUWHVW WLPH VSHQW RQ UROORYHU ZDV  VHF  ZLWKRXW WDNLQJ LQWRDFFRXQWWKHWLPHIRUDQ\SUHYLRXVUROORYHUPRYHPHQW 7RFKHFNWKHLPSRUWDQFHRIHDFKXQLWLQ7'11WKHFRQWULEXWLRQUDWHRIWKHZHLJKWRIHDFK LQSXWXQLWZDVFDOFXODWHGE\   1 

5FRQWULEXWLRQ L

¦Ȧ M 

1 1 

P LM

¦¦ Ȧ

P LM

u 

 

L  M    ZKHUH 5FRQWULEXWLRQL  LV WKH FRQWULEXWLRQ UDWH RI WKH ZHLJKW RI LQSXW XQLW L ZKRVH GDWD LV WKH (0*GDWDRILPVHF EHIRUHWKHFXUUHQWPHDVXUHGSRLQW1 P   $VDUHVXOWLWZDVIRXQGWKDWWKHZHLJKWVRIXQLWVLQWKHUDQJHRIWRPVHF ZHUHKLJKHU WKDQWKRVHRIWKHRWKHUXQLWVLQ7'116HH)LJ ,WLVQDWXUDOWKDWWKH(0*GDWDQHDUHVWWR WKHWLPHRIPHDVXUHPHQWKDVDODUJHLPSDFWRQWKHUHFRJQLWLRQUHVXOW+RZHYHULWLVZRUWK QRWLQJ WKDW WKH FRQWULEXWLRQ UDWH RI WKH LQSXWWHG (0* GDWD EHIRUH  PVHF  LV DOPRVW FRQVWDQW(YHQWKRXJKWKHLPSRUWDQFHRIGDWDIURPPVHF EHIRUHLVWKHVDPHDVWKDWRI GDWD IURP  PVHF  EHIRUH WKH ODWWHU GDWD ZDV QRW XVHG WR UHFRJQL]H WKH UROORYHU PRYHPHQW LQ WKH WUDGLWLRQDO 7'11 6HH 6HFWLRQ   7KHUHIRUH LQ WKH WUDGLWLRQDO 7'11 



/KETQ/CETQ0GWTCN0GVYQTMVQ4GEQIPK\G5NQY/QXGOGPV '/)DCUGF#EEWTCVGCPF3WKEM4QNNQXGT4GEQIPKVKQP



 )LJ&RQWULEXWLRQUDWHDVDIXQFWLRQRILQSXWXQLW  ZKRVH LQSXW XQLW QXPEHU ZDV  PVHF  D ODWHU UHVSRQVH DQG D KLJKHU LQFLGHQFH RI IDOVH UHFRJQLWLRQZHUHHYLGHQW :KHQ ORQJ SDVW WLPHVHULHV (0* GDWD LV XVHG LQ 7'11 WKH DGYDQWDJH RI WKLV ORQJ GDWD OHQJWK LV WKDW PRUH LQSXW GDWD PHDQV IDVWHU UHVSRQVH DQG OHVV IDOVH UHFRJQLWLRQ 7KH GLVDGYDQWDJHLVWKHODUJHDPRXQWRIFDOFXODWLRQVDQGLWVFRVW ,QWKHSURSRVHG0LFUR0DFUR1HXUDO1HWZRUN0011 VRPHRIWKHORQJSDVWWLPHVHULHV GDWDZDVFRPSUHVVHG7KHUHIRUHWKHDPRXQWDQGFRVWRIWKHFDOFXODWLRQVGRQRWLQFUHDVH 7KHEDVLFFRQFHSWRIWKH0LFUR0DFUR1HXUDO1HWZRUN0011 LVWRXVHWKHORQJSDVWWLPH VHULHV (0* GDWD WR GLVFULPLQDWH WKH PRYHPHQW DFFXUDWHO\ DQG TXLFNO\ ZLWKRXW LQFUHDVLQJ WKHFDOFXODWLRQFRVWE\FRPSUHVVLQJVRPHRIWKHORQJSDVWGDWD



5VTWEVWTGQHVJG/KETQ/CETQ0GWTCN0GVYQTM %DVLFDOO\ ZH XSJUDGHG WKH WUDGLWLRQDO 7'11 WR 0011 )LJ   7KH PRVW LPSRUWDQW IHDWXUH RI 0011 LV WKDW LW FDQ KDQGOH DQ LQFUHDVHG DPRXQW RI LQSXW GDWD WR WKH QHXUDO QHWZRUNZLWKRXWLQFUHDVLQJWKHQXPEHURIFDOFXODWLRQV7UDGLWLRQDO7'11LVGHILQHGLQRXU QHWZRUNDVWKH0LFUR3DUW7KHLQSXWGDWD PLFUR [Q LQWKH0LFUR3DUWLVGHILQHGDVIROORZLQJ  PLFUR

[Q  VHPJW²Q 

 

 ZKHUHQ 1PLFURDQG1PLFURLVWKHQXPEHURILQSXWXQLWLQ0LFURSDUW  $VFDQEHVHHQLQ)LJWKHGDWDIRU7PLFURWLVWKH0LFUR3DUWDQGWKHGDWDIRU²7PDFUR 7PLFUR W7PLFURLVWKH0DFUR3DUW,QDGGLWLRQWKHLQSXWGDWD PDFUR [Q LQWKH0DFUR3DUWLV GLYLGHGLQWRVHYHUDO7$59PVHF DQGWKHDYHUDJHUHFWLILHGYDOXH$59 RIWKH(0*VLJQDO DPRQJWKH7$59YDOXHVFDOFXODWHGE\ LVGHILQHGDVWKHLQSXWYDOXHRIWKH0DFUR3DUW





4GEGPV#FXCPEGUKP$KQOGFKECN'PIKPGGTKPI W  Q7$59

PDFUR

[

 Q

¦ VHPJ L

L W  Q  7$59

7$59

 



 ZKHUHQ 1PDFUR 7KHUHIRUH WKH QXPEHU RI LQSXW XQLWV RI WKH 0DFUR 3DUW LV H[SUHVVHG E\ WKH IROORZLQJ HTXDWLRQ  1 PDFUR 7PDFUR 7 $59     ZKHUH1PDFURLVWKHQXPEHURILQSXWXQLWVRIWKH0DFUR3DUW 7KH UHODWLRQV EHWZHHQ HDFK SDLU RI XQLWV LQ ERWK WKH 0DFUR 3DUW DQG WKH 0LFUR 3DUW DUH VKRZQLQ  DQG DERYH 7KH RXWSXW GDWD RI WKH 0LFUR SDUW DQG 0DFUR SDUW LV GHILQHG DV WKH LQSXW GDWD RI WKH ,QWHJUDWHG /D\HU ,Q WKH ,QWHJUDWHG OD\HU WKH RXWSXW VLJQDO LV FDOFXODWHG XVLQJ DOVR     DQG  

)LJ'HYHORSPHQWRI0011DOJRULWKPIURP7'11DOJRULWKP 





/KETQ/CETQ0GWTCN0GVYQTMVQ4GEQIPK\G5NQY/QXGOGPV '/)DCUGF#EEWTCVGCPF3WKEM4QNNQXGT4GEQIPKVKQP



 )LJ  0LFUR 0DFUR QHXUDO QHWZRUN 1RWH WKDW 0011 LV GLYLGHG LQWRWKH 0LFUR 3DUW DQG WKH 0DFUR 3DUW 7KH 0LFUR 3DUW LV 7'11 XVLQJ WKH GDWD IRU 7PLFUR DV WKH LQSXW VLJQDO 7KH LQSXW GDWD RI WKH 0DFUR 3DUWXVHV WKH GDWD IRU 7PDFUR ZKLFK LV WKH $59 RI WKH (0* VLJQDO DPRQJDOO7$59YDOXHV



1RVKOCNUVTWEVWTGQHRTQRQUGF//00CPFTQNNQXGTOQXGOGPVTGEQIPKVKQP 

1DLGEVKXG 7KHVWUXFWXUHRIWKH0011LVFRPSOH[EHFDXVHPDQ\SDUDPHWHUVGHWHUPLQHWKHVWUXFWXUH RI WKH 0011 ,Q WKLV VHFWLRQ EDVHG RQ WKH FRQWULEXWLRQ UDWH VKRZQ LQ )LJ  DQG DQ H[SHULPHQWDERXWUROORYHUUHFRJQLWLRQXVLQJ0011WKHRSWLPDOSDUDPHWHUVLQ0011DUH GHWHUPLQHG



/GVJQFQNQI[QHTQNNQXGTTGEQIPKVKQPGZRGTKOGPV :HGHILQHGWKHUROORYHUPRYHPHQWDVDFRQWLQXRXVPRYHPHQWLQYROYLQJDGHOLEHUDWHFKDQJH RI SRVWXUH IURP D VXSLQH SRVLWLRQ WR D ODWHUDO RU SURQH SRVLWLRQ ,Q WKLV UHVHDUFK UROORYHU PRYHPHQWV ZHUH SHUIRUPHG WKLUW\ WLPHV LQ DGYDQFH E\ HDFK RI WKUHH \RXQJ KHDOWK\ PDOH VXEMHFWV (0* VLJQDOV REWDLQHG IURP WKH LQWHUQDO REOLTXH ,2  PXVFOH ZHUH VHOHFWHG DV WKH LQSXW VLJQDOV EDVHG RQ RXU SUHYLRXV VWXG\ $QGR HW DO   7KH (0* VLJQDOV ZHUH VDPSOHG DW D UDWH RI  +]  UHFWLILHG ZLWKD VHFRQGRUGHUORZSDVV ILOWHU ZLWK DFXWRII IUHTXHQF\ RI  +]  DQG QRUPDOL]HG E\ WKH  PD[LPDO YROXQWDU\ FRQWUDFWLRQ 09&  PHWKRG=DPDQHWDO.XPDUHWDO ZKLFKVKRZVWKHUDWLRRIPXVFOHDFWLYLW\LQ WKH09&RIWKH,2PXVFOHWRWKHPHDVXUHG(0*VLJQDO+HOHQHWDO  $VWKHOHDUQLQJGDWDIRUHYHU\UROORYHUW\SHRIWKHGDWDRXWRIUROORYHUV²IRU HDFK RI WKH WKUHH VXEMHFWV  ZDV UDQGRPO\ VHOHFWHG .XULED\DVKL HW DO  )XNXGD HW DO  7KHRWKHURIWKHGDWDZDVXVHGDVWHVWGDWD%HFDXVHWKHQXPEHUVRIOHDUQLQJDQG





4GEGPV#FXCPEGUKP$KQOGFKECN'PIKPGGTKPI

WHVW GDWD ZHUH VPDOO WKH NIROG FURVV YDOLGDWLRQ HVWLPDWLRQ N    ZDV XVHG WR SUHYHQW GHJUDGDWLRQRIWKHDFFXUDF\EDVHGRQWKHVHOHFWLRQRIOHDUQLQJGDWD 7KH WLPH UHTXLUHG WR UHFRJQL]H WKH UROORYHU ZDV PHDVXUHG XVLQJ 7'11 )XUWKHUPRUH E\ V\QFKURQL]LQJ WKH (0* GDWD ZLWK WKH GDWD RI D ' PRWLRQFDSWXUH V\VWHP 9,&21 VDPSOLQJ IUHTXHQF\  +]  DQG PHDVXUHPHQW DFFXUDF\  PP   WKH VWDUW RI UROORYHU PRYHPHQWZDVUHFRJQL]HG



'XCNWCVKQPKPFGZ 7KH UHFRJQLWLRQ UHVXOWV RI WKH WHVW GDWD ZHUH HYDOXDWHG DFFRUGLQJ WR WKH UHVSRQVH E\ WKH LQGH[HVSUHVHQWHGEHORZ   7KHUHVSRQVHWLPHWUHVSRQVHLVWKHWLPHIURPWKHVWDUWRIWKHUROORYHUPRYHPHQWWRWKH UHFRJQLWLRQRIWKHUROORYHUPRYHPHQWE\WKHQHXUDOQHWZRUN  W UHVSRQVH W UHFRJQLWLRQ  W PRYHPHQW     ZKHUHWUHFRJQLWLRQLVWKHWLPHZKHQWKHUROORYHULVUHFRJQL]HGDQGWPRYHPHQWLVWKHWLPHZKHQWKH UROORYHUVWDUWV    0RYHPHQWUHFRJQLWLRQUDWHEHIRUHVWDUWLQJPRYHPHQW3VWDUW  3VWDUW 1EHIRUH1WRWDO    ZKHUH 3VWDUW LV WKH UDWLR RI 1EHIRUH WKH QXPEHU RI WLPHV UROORYHU ZDV UHFRJQL]HG EHIRUH WKH PRYHPHQWVWDUWHGWR1WRWDOWKHWRWDOQXPEHURIUROORYHUPRYHPHQWV    1XPEHURIIDOVHUHFRJQLWLRQUDWH1IDOVH 1IDOVH LV WKH QXPEHU RI WLPHV ZKHQ IDOVH UHFRJQLWLRQ RFFXUUHG WKDW LV WKH WLPHV WKDW 11 UHFRJQL]HGDUROORYHUPRYHPHQWHYHQWKRXJKQRUROORYHUZDVDFWXDOO\FRQGXFWHG



5VTWEVWTGQH6&00CPFTGEQIPKVKQPTGUWNV $VVWDWHGDERYHIRUWKHOHDUQLQJPDFKLQHLQWKLVUHVHDUFKZHVHOHFWHGWKHWKUHHOD\HUIHHG IRUZDUG W\SH 11 DQG WKH EDFN SURSDJDWLRQ PHWKRG ZLWK PRPHQWXP WHUP ZKLFK LV D VWDQGDUG QHXUDO QHWZRUN IRU UHFRJQL]LQJ WLPHVHULHV VLJQDOV 7KH QXPEHU RI LQSXW OD\HU XQLWV ZDV  7KH XQLW QXPEHUV RI WKH KLGGHQ OD\HU DQG WKH RXWSXW OD\HU ZHUH  DQG  UHVSHFWLYHO\ $VVKRZQLQ)LJE ZKHQ7'11ZDVXVHGWKHUHFRJQLWLRQUHVXOWVZHUHDVIROORZVWUHVSRQVH ZDV6' PVHF 3VWDUWZDVRXWRIWULDOV DQG1IDOVHZDVRXWRI WULDOV



1RVKOCNUVTWEVWTGQH//00CPFTGEQIPKVKQPTGUWNV 7KHVWUXFWXUHRI0011ZDVUHVROYHGEDVHGRQPDQ\SDUDPHWHUV )LUVWLQWKH0LFUR3DUWZKLFKLVWKHWUDGLWLRQDO7'11WKHQXPEHURILQSXWOD\HUXQLWVZDV IL[HGDW7PLFUR PVHF LQ)LJ EHFDXVHWKHFRQWULEXWLRQUDWHVLQaPVHF DUH KLJKHUWKDQWKRVHDWRWKHULQSXWWLPHVDVVKRZQLQ)LJ7KHQXPEHURIKLGGHQOD\HUXQLWV



/KETQ/CETQ0GWTCN0GVYQTMVQ4GEQIPK\G5NQY/QXGOGPV '/)DCUGF#EEWTCVGCPF3WKEM4QNNQXGT4GEQIPKVKQP



ZDVIL[HGDWDQGWKHQXPEHURIRXWSXWOD\HUXQLWVZDVIL[HGDW7KHQXPEHURIKLGGHQ OD\HUXQLWVZDVGHWHUPLQHGEDVHGRQWKH´UXOHRIWKXPEµDVIROORZ  1KLGGHQ 1LQSXW1RXWSXW     ZKHUH1KLGGHQ1LQSXWDQG1RXWSXWDUHWKHQXPEHUVRIKLGGHQOD\HULQSXWOD\HUDQGRXWSXWOD\HU XQLWV  6HFRQG WKH RSWLPDO VWUXFWXUH RI WKH 0DFUR 3DUW ZDV GHWHUPLQHG DV IROORZV 7KH YDOXH RI 7$59 ZDV FKDQJHG IURP  WR  7$59    «   DQG WKH YDOXH RI 1PDFUR WKH QXPEHURILQSXWOD\HUXQLWVZDVFKDQJHGIURPWR1PDFUR « $GGLWLRQDOO\ DFFRUGLQJ WR WKH UXOH RI WKXPE WKH QXPEHU RI KLGGHQ OD\HU XQLWV ZDV VHW DW 1PDFUR  LI 1PDFURZDVHYHQ RU1PDFUR LI1PDFURZDVRGG %DVHGRQRXU(0*H[SHULPHQW$QGRHW DO ZKLFKVKRZHGWKDWWKHVKRUWHVWWLPHVSHQWRQUROORYHUZDVPVHF ZHDSSOLHG   ZKHQ ZH FDOFXODWHG WKH UHVSRQVH WLPH IRU HDFK UROORYHU PRYHPHQW XVLQJ 0011 ZLWKRXWWDNLQJLQWRDFFRXQWWKHWLPHIRUDQ\SUHYLRXVUROORYHUPRYHPHQW  7$59 u 1 PDFUR d   1 PLFUR          :H REWDLQHG WKH EHVW UHVXOWV IRU UHVSRQVH ZLWK FKDQJLQJ YDOXHV RI 7$59 DQG 1PDFUR  ZKHQ 7$59 VHF DQG1PDFUR :LWKWKHVHFRQGLWLRQVWKHDYHUDJHWUHVSRQVHIRU0011ZDV 6'   PVHF  7KH DYHUDJH WUHVSRQVH IRU 7'11 ZDV  6'   PVHF  1HJDWLYH YDOXHV PHDQWKHUROORYHUZDVUHFRJQL]HGEHIRUHWKHPRYHPHQWVWDUWHG7KHUHIRUHWKHUHFRJQLWLRQ WLPHRI0011ZDV6' PVHF IDVWHUWKDWWKDWRI7'11 )XUWKHUPRUH DV VKRZQ LQ 7DEOH  WKH 3VWDUW ZDV   RXW RI  WLPHV  DQG 1IDOVH ZDV RQO\LQWULDOV )LJXUH  VKRZV DQH[DPSOHRI 0011 7$59   PVHF  1PDFUR   :KHQ WKH UHVXOWVRI 7'11LQ)LJE DQGWKH0011LQ)LJF DUHFRPSDUHGWKHIROORZLQJREVHUYDWLRQVDUH FOHDU 7'11 UHJLVWHUV D IDOVH UHFRJQLWLRQ IRXU WLPHV DQG PRVW LPSRUWDQWO\ WKH UHVSRQVH VSHHG LQ UHFRJQL]LQJ UROORYHU LV IDVWHU VWHDGLHU DQG PRUH DFFXUDWH ZKHQ 0011 LV XVHG WKDQZKHQ7'11LVXVHG          





4GEGPV#FXCPEGUKP$KQOGFKECN'PIKPGGTKPI



D ,QSXWVLJQDOWR11



E 2XWSXWVLJQDOIURPWKH7'11

 F 2XWSXWVLJQDOIURPWKH0011  )LJ&RPSDULVRQEHWZHHQUHFRJQLWLRQRIUROORYHUE\7'11DQG00111RWHWKDW7'11 IDLOV WR UHFRJQL]H WKH UROORYHU DW  PVHF  DQG  PVHF  KRZHYHU LW GRHV UHFRJQL]H WKH UROORYHU DIWHU WKH PRYHPHQW VWDUWV ,Q FRQWUDVW 0011 UHFRJQL]HV WKH UROORYHU EHIRUH WKH PRYHPHQWVWDUWV(0*VLJQDOGDWDLVLQFOXGHGIRUUHIHUHQFH  1HXUDO1HWZRUN 7'11 0011

3VWDUW  

7DEOH3VWDUWDQG1IDOVHRI7'11DQG0011

1IDOVH  



&KUEWUUKQP ,Q 6HFWLRQ  WKH HIIHFWLYHQHVV RI 0011 LQ UHFRJQL]LQJ WKH UROORYHU LV VKRZQ LQ FRPSDULVRQZLWKWKHHIIHFWLYHQHVVRI7'11 ,Q WKLV VHFWLRQ WKH RXWSXW VLJQDO RI QRW RQO\ RSWLPDO 0011 EXW DOVR WKH RXWSXW RI WKH 0LFURSDUWDQG0DFURSDUWLQWKHUHFRJQLWLRQRIWKHUROORYHUPRYHPHQWLVGLVFXVVHGWRVKRZ WKHFKDUDFWHULVWLFVRI0011,QRWKHUZRUGVILUVWWKHQXPEHURILQSXWOD\HUXQLWVLQ7'11 ZDVDQGWKHLQSXWRIWKHLQSXWOD\HUZDVGHILQHGWRVKRZWKHFKDUDFWHULVWLFVRIWKH0LFUR SDUWDV(0*VLJQDOVVHPJWL L  6HFRQGWKHQXPEHURILQSXWOD\HUXQLWVLQ7'11



/KETQ/CETQ0GWTCN0GVYQTMVQ4GEQIPK\G5NQY/QXGOGPV '/)DCUGF#EEWTCVGCPF3WKEM4QNNQXGT4GEQIPKVKQP



ZDVDQGWKHLQSXWRIWKHLQSXWOD\HUZDVGHILQHGWRVKRZWKHFKDUDFWHULVWLFVRIWKH0DFUR SDUWDVWKHDYHUDJHUHIOHFWHGYDOXHVDPRQJPVHF  7DEOHVKRZVWKHUHVXOWRIUHFRJQLWLRQWLPHDQGWKHQXPEHURIWKHIDOVHUHFRJQLWLRQXVLQJ WKH7'11,QSXWPVHF UDZ(0*VLJQDO 0011,QSXWPVHF UDZ(0*DQG $59(0*DPRQJPVHF RQO\0LFUR3DUW,QSXWPVHF UDZ(0* LQ0011DQG RQO\ 0DFUR 3DUW ,QSXW  $59 (0* DPRQJ  PVHF  LQ 0011 7KH UHVSRQVH WLPHV XVLQJRQO\WKH0LFURSDUWDQGWKH0DFURSDUWZHUHWUHVSRQVH 6' PVHF DQGWUHVSRQVH  6' PVHF 7KHQXPEHURIIDOVHUHFRJQLWLRQVXVLQJRQO\WKH0LFURSDUWDQGRQO\WKH 0DFURSDUWZDV1IDOVH LQWLPHV DQG1IDOVH LQWLPHV  :KHQ WKH LQSXW GDWD ZDV VKRUW SDVW WLPHVHULHV GDWD WKH UHVSRQVH WLPH ZDV VKRUW DQG WKH VWDELOLW\ RI WKH UHFRJQLWLRQ ZDV ORZ +RZHYHU ZKHQ WKH LQSXW GDWD ZDV WKH $59 RI  PVHF WKHUHVSRQVHWLPHEHFDPHORQJHUDQGWKHVWDELOLW\LQFUHDVHG 7KH UHVSRQVH WLPH WUHVSRQV GLG QRW VKRZ D VLJQLILFDQW GLIIHUHQFH S    EHWZHHQ WKH RSWLPL]HG0011DQG7'11XVLQJPVHF WLPHVHULHVGDWDDVGLGWKHLQSXWGDWDWKDWLV ZKHQ XVLQJ WKH RQO\ 0LFUR SDUW LQ 0011 ,Q DGGLWLRQ WKH QXPEHU RI IDOVH UHFRJQLWLRQV ZDVDOPRVWWKHVDPHZKHQWKHQXPEHULQWKHRSWLPL]HG0011ZDVFRPSDUHGZLWKWKDWLQ 7'11 XVLQJ WKH $59 RI  PVHF  WKDW LV ZKHQ XVLQJ WKH RQO\ 0DFUR SDUW LQ 0011 7KHUHIRUHWKHDGYDQWDJHVRITXLFNUHVSRQVHLQWKH0LFURSDUW6HH)LJE DQGWKHVWDEOH UHFRJQLWLRQ RI WKH 0DFUR SDUW 6HH )LJ  F  DUH FRPELQHG LQ WKH GHYHORSHG RSWLPDO 0011 $V D UHVXOW WKH 0011 LV DQ 11 WKDW IHDWXUHV TXLFN UHVSRQVH DQG OLWWOH IDOVH UHFRJQLWLRQ6HH)LJG 



1IDOVH WUHVSRQVHPVHF 1HXUDO1HWZRUN 7'11 6'   ,QSXWPVHF UDZ(0*  0011 6'   ,QSXWPVHF UDZ(0*DQG$59 (0*DPRQJPVHF  2QO\0LFURSDUWLQ0011 6'   ,QSXWPVHF UDZ(0*  2QO\0DFURSDUWLQ0011 6'   ,QSXW$59(0*DPRQJPVHF  7DEOH)HDWXUHVRI7'110011DQG0LFURDQG0DFURSDUWVRI0011

      





D ,QSXWVLJQDOWRWKH11

E 2XWSXWIURPWKH0LFURSDUWLQ0011

F 2XWSXWIURPWKH0DFURSDUWLQ0011

4GEGPV#FXCPEGUKP$KQOGFKECN'PIKPGGTKPI







G 2XWSXWIURPWKH0011  )LJ&RPSDULVRQZLWKWKHRXWSXWRI7'110LFURSDUWLQ00110DFUR3DUWLQ0011 DQG00111RWHWKDWWKH(0*VLJQDOLVLQFOXGHGIRUUHIHUHQFHDVD E LVWKHRXWSXWRI 0LFURSDUWDQGVKRZVWKHTXLFNDQGXQVWDEOHUHVSRQVHF LVWKHRXWSXWRI0DFURSDUWDQG VKRZVWKHVORZDQGVWDEOHUHVSRQVHDQGG LVRXWSXWRI0011DQGVKRZTXLFNDQGVWDEOH UHVSRQVH



/KETQ/CETQ0GWTCN0GVYQTMVQ4GEQIPK\G5NQY/QXGOGPV '/)DCUGF#EEWTCVGCPF3WKEM4QNNQXGT4GEQIPKVKQP



5WOOCT[CPFHWVWTGYQTM 

:HKDYHEHHQVWXG\LQJSDWLHQWVZLWKFDQFHUERQHPHWDVWDVLVZKRKDYHDYHU\VKRUWWLPHWR OLYH 6SHFLILFDOO\ ZH KDYH GHYHORSHG WKH (0* FRQWUROOHG LQWHOOLJHQW FRUVHW WR VXSSRUW WKH UROORYHUPRYHPHQW ,QWKLVSDSHUZHGHVFULEHGDQRULJLQDOQHXUDOQHWZRUNWKDWZHGHYHORSHGFDOOHGWKH0LFUR 0DFUR 1HXUDO 1HWZRUN 0011  IRU WKH SXUSRVH RI UHFRJQL]LQJ DQG UHVSRQGLQJ WR WKH UROORYHUPRYHPHQWEDVHGRQLQSXWWHG(0*VLJQDOV )LUVWWKHVWUXFWXUHRIWKH0011ZDVRSWLPL]HGZLWK1PLFUR LQWKH0LFURSDUWDQG1PDFUR   DQG 7$59   LQ WKH 0DFUR SDUW DQG WKHQ WKH UHVSRQVH DQG DFFXUDF\ RI WKH 0011 ZHUH DQDO\]HG $IWHU WKDW WKH UHVSRQVH DQG DFFXUDF\ RI WKH RSWLPL]HG 0011 LQ UHFRJQL]LQJ WKH UROORYHU PRYHPHQW ZHUH FRPSDUHG ZLWK WKRVH RI WKH WUDGLWLRQDO 7'11 7HVW UHVXOWV VKRZHG WKDW UHFRJQLWLRQ LQ 0011 ZDV  6'   PVHF  ZKLFK LV TXLFNHU WKDQ WKH UHFRJQLWLRQ LQ 7'11 $GGLWLRQDOO\ WKH QXPEHU RI IDOVH UHFRJQLWLRQV LQ 0011 ZDVRQO\RQHWKLUGRIWKRVHLQ7'11+HQFHZHFDQYHULI\WKDWRXU0011LVHIIHFWLYHDQG XVHIXO LQ UHFRJQL]LQJ UROORYHU EDVHG RQ LQSXWWHG (0* VLJQDOV ZKLFK DUH QRLV\ DQG YDU\ FRQVLGHUDEO\ IURP LQGLYLGXDO WR LQGLYLGXDO ,Q DGGLWLRQ E\ FRPSDULQJ WKH UHFRJQLWLRQ UHVXOWV RI RQO\ WKH 0LFUR SDUW DQG RQO\ WKH 0DFUR SDUW ZH IRXQG WKDW WKH DGYDQWDJHV RI TXLFN UHVSRQVH LQ WKH 0LFUR SDUW DQG VWDEOH UHFRJQLWLRQ LQ WKH 0DFUR SDUW DUH IHDWXUHV RI 0011 ,Q WKH IXWXUH ZH ZLOO LQFRUSRUDWH 0011 LQWR RXU UROORYHU VXSSRUW V\VWHP WKDW XVHV SQHXPDWLF UXEEHU PXVFOHV DQG WKHQ ZH ZLOO WHVW WKH HIIHFWLYHQHVV RI WKH WRWDO V\VWHP LQ FOLQLFDOWHVWVZLWKFDQFHUSDWLHQWVLQWHUPLQDOFDUH



#EMPQYNGFIGOGPV 

7KLVZRUNZDVVXSSRUWHGLQSDUWE\WKH*OREDO&HQWHURI([FHOOHQFH3URJUDP´*OREDO5RERW $FDGHPLDµ:DVHGD8QLYHUVLW\7RN\R-DSDQDQG*UDQWLQ$LGIRU6FLHQWLILF5HVHDUFK$   DQG*UDQWLQ$LGIRU6FLHQWLILF5HVHDUFK -DSDQ,QDGGLWLRQWKLVZRUN ZDVDGYLVHGE\'U7DNDKDVKL0' 6KL]XRND&DQFHU&HQWHU



4GHGTGPEGU 

$QGR 7DNHVKL 2NDPRWR -XQ  0DVDNDWVX * )XMLH   7KH GHYHORSPHQW RI UROORYHU VXSSRUWV\VWHPZLWK(0*FRQWUROIRUERQHPHWDVWDVLVSDWLHQWV 3URFHHGLQJVRI ,(((,QWHUQDWLRQDO&RQIHUHQFHRQ5RERWLFVDQG$XWRPDWLRQSS5RPD,WDO\ $SULO $QGR 7DNHVKL2NDPRWR -XQ  0DVDNDWVX * )XMLH D  7KH GHYHORSPHQW RI UROORYHU VXSSRUW V\VWHP ZLWK (0* FRQWURO IRU ERQH PHWDVWDVLV SDWLHQWV3URFHHGLQJVRIWK $QQXDO,QWHUQDWLRQDO,((((0%6&RQIHUHQFH9DQFRXYHU&DQDGD$XJXVW  $QGR 7DNHVKL 2NDPRWR -XQ  0DVDNDWVX * )XMLH E  ´,QWHOOLJHQW FRUVHW WR VXSSRUW UROORYHU RI FDQFHU ERQH PHWDVWDVLV SDWLHQWVµ 3URFHHGLQJV RI WKH  ,(((56- ,QWHUQDWLRQDO &RQIHUHQFH RQ ,QWHOOLJHQW 5RERWV DQG 6\VWHPV SS 1LFH )UDQFH 6HSWHPEHU





4GEGPV#FXCPEGUKP$KQOGFKECN'PIKPGGTKPI

.$)DUU\,':DONHU 5*%DUDQLXN 0\RHOHFWULF7HOHRSHUDULRQRID&RPSOH[ 5RERWLF +DQG ,((( 7UDQVDFWLRQV RQ 5RERWLFV DQG $XWRPDWLRQ 9RO 1R SS ; )XNXGD 2 7VXML 7 6KLJH\RVKL +  .DQHNR 0   ´$Q (0* FRQWUROOHG KXPDQ VXSSRUWLQJ URERW XVLQJ QHXUDO QHWZRUNµ 3URFHHGLQJVRI,(((56-,QWHUQDWLRQDO &RQIHUHQFH RQ ,QWHOOLJHQW 5RERWV DQG 6\VWHPV YRO SS    .\RQJMX6RXWK.RUHD2FWRUEHU -XQML)XUXVKR&KHQJTLX/L6KRXML0RULPRWR0LZD7RNXGD7DNHKLWR.LNXFKL