Multiclass Classification Problem of Large-Scale Biomedical ... - Core

Available online at www.sciencedirect.com

ScienceDirect Procedia Technology 22 (2016) 938 – 945

WK,QWHUQDWLRQDO&RQIHUHQFH,QWHUGLVFLSOLQDULW\LQ(QJLQHHULQJ,17(5(1*2FWREHU 7LUJX0XUHV5RPDQLD

0XOWLFODVV&ODVVL¿FDWLRQ3UREOHPRI/DUJH6FDOH%LRPHGLFDO0HWD 'DWD 6HEDVWLDQ6WXGHQWD -XVW\QD3LHWHUD.U]\V]WRI)XMDUHZLF]D D

,QVWLWXWHRI$XWRPDWLF&RQWURO6LOHVLDQ8QLYHULW\RI7HFKQRORJ\XO$NDGHPLFND*OLZLFH3RODQG

$EVWUDFW 2QHRIWKHLPSRUWDQWGDWDPLQLQJPHWKRGLQELRPHGLFDOUHVHDUFKLVFODVVL¿FDWLRQWDVN5HFHQWDGYDQFHVLQELRPHGLFLQHSURYLGH RSSRUWXQLWLHVIRUPROHFXODUELRORJ\VXFKDVPHDVXUHPHQWRIDFWLYLW\RIWKRXVDQGVRIPROHFXODUWLVVXHELRPDUNHUV)RUH[DPSOH ZHFDQXVHGDWDRIJHQHH[SUHVVLRQPHDVXUHGE\'1$PLFURDUUD\VRU51$6HTWHFKQLTXH'1$PHWK\ODWLRQOHYHOVPHDVXUHGE\ '1$PHWK\ODWLRQPLFURDUUD\VRUSURWHLQDQGSKRVSKRSURWHLQOHYHOVPHDVXUHGE\UHYHUVHSKDVHSURWHLQDUUD\$ELJSUREOHPLQ DSSO\LQJ ODUJHVFDOH JHQRPLF DQG SURWHRPLF GDWD IRU FODVVL¿FDWLRQ SUREOHP LV WKH GLPHQVLRQ RI WKHVH GDWD ,Q WKLV ZRUN ZH SURSRVH QRYHO PXOWLFODVV IHDWXUH VHOHFWLRQ DQG FODVVL¿FDWLRQ V\VWHP IRU PHUJHG GDWD IURP GLơHUHQW PROHFXODU ELRPHGLFDO WHFKQLTXHV+RZHYHUZKHQZHPHUJHWKHVHGDWDWKHELJJHVWSUREOHPLVWKHKXJHQXPEHURIIHDWXUHVZLWKDOLPLWHGQXPEHURI VDPSOHV )RU WKDW UHDVRQ WKH IHDWXUH VHOHFWLRQ VWHS LV FUXFLDO LQ KLJK GLPHQVLRQ GDWD FODVVL¿FDWLRQ SUREOHP 2XU UHVXOWV KDYH VKRZQ WKDW LQWHJUDWHG DQDO\VLV ZLWK SURSHU IHDWXUH VHOHFWLRQ DQG FODVVL¿FDWLRQ WHFKQLTXHV XVHG IRU ODUJHVFDOH PHWDGDWD FDQ LPSURYH WKH FODVVL¿FDWLRQ DFFXUDF\ DQG IHDWXUH VHOHFWLRQ VWDELOLW\ LQGH[ :H KDYH SURRIHG WKDW IRU PHUJHG GDWD ZH REVHUYH VLJQL¿FDQWO\KLJKHUFODVVL¿FDWLRQDFFXUDF\IRUWKHVDPHQXPEHURIVHOHFWHGIHDWXUHVDVIRUVLQJOHWHFKQLTXHGDWDVHW © 2016 Published by Elsevier Ltd. This is an open access article under the CC BY-NC-ND license ‹7KH$XWKRUV3XEOLVKHGE\(OVHYLHU/WG (http://creativecommons.org/licenses/by-nc-nd/4.0/). 3HHUUHYLHZXQGHUUHVSRQVLELOLW\RIWKH³3HWUX0DLRU´8QLYHUVLW\RI7LUJX0XUHV)DFXOW\RI(QJLQHHULQJ Peer-review under responsibility of the “Petru Maior” University of Tirgu Mures, Faculty of Engineering .H\ZRUGV0XOWLFODVVFODVVL¿FDWLRQ690IHDWXUHVHOHFWLRQPHWDGDWDDQDO\VLV%LRPHGLFDOGDWDDQDO\VLV





&RUUHVSRQGLQJDXWKRU7HO (PDLODGGUHVVVHEDVWLDQVWXGHQW#SROVOSO

2212-0173 © 2016 Published by Elsevier Ltd. This is an open access article under the CC BY-NC-ND license (http://creativecommons.org/licenses/by-nc-nd/4.0/). Peer-review under responsibility of the “Petru Maior” University of Tirgu Mures, Faculty of Engineering doi:10.1016/j.protcy.2016.01.093

939

Sebastian Student et al. / Procedia Technology 22 (2016) 938 – 945

,QWURGXFWLRQ 2QHRIWKHLPSRUWDQWGDWDPLQLQJPHWKRGLQELRPHGLFDOUHVHDUFKLVFODVVL¿FDWLRQWDVN2YHUWKHSDVWGHFDGHVD ZLGH UDQJH RI FODVVL¿FDWLRQ DOJRULWKPV KDYH EHHQ SURSRVHG LQ OLWHUDWXUH WR WDFNOH YDULRXV FODVVL¿FDWLRQ SUREOHPV &ODVVL¿FDWLRQDOJRULWKPVFDQEHGLYLGHGLQWRWZRFDWHJRULHVELQDU\DQG PXOWLFODVVFODVVL¿HUV7KHLQWHUHVWRIWKLV ZRUN LV IRFXVHG RQ PXOWLFODVV SUREOHPV WKXV PDQ\ FODVVL¿FDWLRQ SUREOHPV LQ ELRPHGLFLQH KDYH WKH PXOWLFODVV QDWXUH $GGLWLRQDOO\ UHFHQW DGYDQFHV LQ ELRPHGLFLQH SURYLGH DQ RSSRUWXQLWLHV IRU PROHFXODU ELRORJ\ VXFK DV PHDVXUHPHQW RI DFWLYLW\ RI WKRXVDQGV RI PROHFXODU WLVVXH ELRPDUNHUV )RU H[DPSOH ZH FDQ XVH GDWD RI JHQH H[SUHVVLRQ PHDVXUHG E\ '1$ PLFURDUUD\V RU 51$6HT WHFKQLTXH '1$ PHWK\ODWLRQ OHYHOV PHDVXUHG E\ '1$ PHWK\ODWLRQ PLFURDUUD\V RU SURWHLQ DQG SKRVSKRSURWHLQ OHYHOV PHDVXUHG E\ UHYHUVH SKDVH SURWHLQ DUUD\ +RZHYHU WKHFRPELQDWLRQRISURWHRPLFVJHQRPLFVDQGRWKHUPRGHUQPROHFXODUWHFKQLTXHVLQWKH¿HOGRIFOLQLFDODSSOLFDWLRQV VWXGLHVSURYLGHVDSODWIRUPIRUDFKLHYLQJDQHZGHSWKLQPROHFXODUSUR¿OLQJ>@1RZDGD\VFRPELQLQJPXOWLSOH VRXUFHVRIGDWDWRLPSURYHWKHFODVVL¿FDWLRQDQDO\VHVLVDFKDOOHQJLQJWDVNLQELRLQIRUPDWLFV7KHPHWDDQDO\VLVRI VXFKLQWHJUDWHGGDWDPHDVXUHGLQWKHVDPHVDPSOHVFDQLPSURYHFODVVL¿FDWLRQSRZHU:HKDYHWRXQGHUOLQHWKDWQRW RQO\ WKH GLơHUHQW W\SH RI WKH GDWD ELQDU\ GDWD HWF  EXW DOVR WKH GLơHUHQW GDWD VFDOH FDQ OHDG WR SUREOHPV $ ELJ SUREOHPLQDSSO\LQJODUJHVFDOHJHQRPLFDQGSURWHRPLFGDWDIRUFODVVL¿FDWLRQSUREOHPLVWKHGLPHQVLRQRIWKLVGDWD >@,QPRVWFDVHVVWDQGDUGVWDWLVWLFDOPHWKRGRORJ\GRHVQRWZRUNZHOOZKHQLQWKHFODVVL¿HGGDWDDUHPRUHYDULDEOHV WKDQ VDPSOHV &ODVVL¿FDWLRQ WDVN RI VXFK GDWD LV KDUG WR SHUIRUP HVSHFLDOO\ LI ZH FODVVLI\ FRPELQHG GDWD 6RPH VWXGLHVKDYHFRQVLGHUHGWKLVSUREOHP\HWWKH\PHUHO\IRFXVRQWZRFODVVSUREOHP /DUJHVFDOHPHWDGDWDVHWGHVFULSWLRQ ,QWKLVVWXG\ZHXVHSXEOLFO\DYDLODEOHPXOWLFODVVJHQRPLFDQGSURWHRPLFGDWDVHWVRIKXPDQEUHDVWWXPRUV,WLV LPSRUWDQWWRVWUHVVWKDWWKHGDWDVHWVRIDOOW\SHVLHPLFURDUUD\&KLS6HTDQGSURWHRPLFGDWDZHUHREWDLQHGIURP WKH VDPH  SDWLHQWV ZLWK EDVDOOLNH FDQFHU  FDVHV  /XPLQDO $  FDVHV  DQG /XPLQDO %  FDVHV  7KH GDWDVHWFDQEHIUHHO\GRZQORDGHGIURP7KH&DQFHU*HQRPH$WODV7KHUHVXOWVVKRZQKHUHDUHLQZKROHEDVHGXSRQ GDWDJHQHUDWHGE\WKH7&*$5HVHDUFK1HWZRUNKWWSFDQFHUJHQRPHQLKJRY 0HWKRGRORJ\ &ODVVL¿HUGHVLJQ 2SWLPDOGHVLJQRIWKHFODVVL¿FDWLRQV\VWHPIRUPXOWLFODVVODUJHVFDOHPHWDGDWDGHVFULEHGLQWKHSUHYLRXVVHFWLRQ FRQVWLWXWHV FKDOOHQJLQJ FRPSXWDWLRQDO DQG FRQFHSWLRQDO SUREOHP 7R WKH PRVW LPSRUWDQW DQG FUXFLDO VWHSV RI FODVVL¿HUGHVLJQ ZHLQFOXGH GDWDSUHSURFHVVLQJFKRLFHRISURSHUFODVVL¿FDWLRQDQGIHDWXUHVHOHFWLRQ PHWKRGV)RU WKDWUHDVRQZHFRPSDUHUHVXOWVIRUGLơHUHQWFODVVL¿HUVIHDWXUHVHOHFWLRQPHWKRGVDQGGLơHUHQWQXPEHURIVHOHFWHG IHDWXUHV2XUFODVVL¿HULVEDVHGRQWKHFODVVL¿FDWLRQV\VWHPSURSRVHGLQ>@DQGLVVKRZQLQ)LJ,WLVLPSRUWDQWWKDW VXFKDFODVVL¿FDWLRQV\VWHPFDQDOVREHXVHGIRURWKHUGDWDVXFKDVWKHELRPHGLFDOLPDJLQJGDWD>@

 )LJ&ODVVLILFDWLRQV\VWHPVFKHPH

940

Sebastian Student et al. / Procedia Technology 22 (2016) 938 – 945

:H KDYH FRPSDUHG WKH IROORZLQJ W\SHV RI FODVVL¿FDWLRQ 'LDJRQDO /LQHDU 'LVFULPLQDQW $QDO\VLV FODVVL¿HU '/'$ >@DQG6XSSRUW9HFWRU0DFKLQHVPHWKRG690 >@$GGLWLRQDOO\ZHXVHGIROORZLQJIHDWXUHVHOHFWLRQ DOJRULWKPV WKH *6 PHWKRG SURSRVHG LQ >@ %66:66VWDWLVWLFV ZKHUH %66 GHQRWHV WKH EHWZHHQJURXS VXP RI VTXDUHVDQG:66WKHZLWKLQJURXSVXPRIVTXDUHVDQG3/6PHWKRG>@,QDOOFDVHVWKHRQHYHUVXVRQHDSSURDFK ZDVXVHGEDVHGRQWKHVFKHPHRQ)LJ

 )LJ)HDWXUHVHOHFWLRQV\VWHPVFKHPH

690PHWKRGV ,Q RXU DQDO\VHV ZH KDYH XVHG ELQDU\ 690 :H GHQRWH =  ]  ]  ]O DV RXU GDWDVHW ZKHUH ]L  [L  \L DQG L   O DQG \L  ^  . `  7KH UHVSRQVH \L LV WKH FODVV RI SUHGLFWRU YHFWRU [L  7KH 690 VROYHV WKH IROORZLQJSUREOHP O & &  . Z Z  & [L  PLQ   ¦ & Z E [  L 



 

& & \L  .  Z [  E t   [L  [L t  L   O  

 

VXEMHFWWR

 & & .  Z Z  ZH PD[LPL]H WKH   & & PDUJLQEHWZHHQWZRJURXSVRIGDWD & $IWHUVROYLQJWKLVSUREOHPZHKDYHGHFLVLRQIXQFWLRQ Z ˜ [  E IRUOLQHDU __ Z __ NHUQHOIXQFWLRQ .    ZKHUH .  LV WKH NHUQHO IXQFWLRQ DQG & LV SHQDOW\ SDUDPHWHU :KHQ ZH PLQLPL]H

941

Sebastian Student et al. / Procedia Technology 22 (2016) 938 – 945

:HGRQ WGLVFXVVDERXWNHUQHOIXQFWLRQEHFDXVHLQRXUFDVHWKHEHVWUHVXOWV ZHUHREWDLQHGIRUWKHOLQHDUNHUQHO & & & & IXQFWLRQ .  Z [ Z7 [  /'$DQG'/'$PHWKRGV 7KH VHFRQG FODVVL¿FDWLRQ PHWKRG WKDW ZH KDYH XVHG LV /LQHDU 'LVFULPLQDQW $QDO\VLV 7KH SULPDU\ SXUSRVH RI /'$LVWRVHSDUDWHVDPSOHVRIGLVWLQFWJURXSV(DFKVDPSOHJURXS S L KDVDFODVVPHDQ [L

1M

 1L

¦[



L M



 



 



 



 

M 

DQGZHKDYH 1L REVHUYDWLRQVLQ S L FODVV /HWXVGHILQHDVDPSOHJURXSFRYDULDQFHPDWUL[

VL

 1L ¦  [L  M  [L  [L  M  [L 7  1L   M 

DVFDWWHUPDWUL[EHWZHHQFODVV J

6E

¦ 1 [ L

L

 [  [L  [ 7 

L 

DQGDZLWKLQFODVVFRYDULDQFHPDWUL[ J

6Z

¦ 1

L

  V L 

L 

FRPSXWHGE\SRROLQJWKHHVWLPDWHVRIWKHFRYDULDQFHPDWULFHVRIHDFKFODVV 7KHPDLQREMHFWLYHRI/'$LVWRILQGDSURMHFWLRQPDWUL[ ) /'$ WKDWPD[LPL]HVWKHUDWLRRIWKHGHWHUPLQDQWRI 6E WRWKHGHWHUPLQDQWRI 6 Z )LVKHUFULWHULRQ 

) /'$

DUJ PD[ )

__ ) 7 6E ) __  __ ) 7 6 Z ) __



 

ZKLFKPD\EHIRXQGE\VROYLQJWKHIROORZLQJHLJHQYDOXHSUREOHP

6E )  6Z )/





 

,Q'/'$WKHFRYDULDQFHPDWUL[ V L LVHVWLPDWHGE\WKHGLDJRQDOFRPPRQVDPSOHFRYDULDQFHPDWUL[&RYDULDQFH PDWUL[LVGLDJRQDOZLWKHDFKGLDJRQDOHOHPHQWEHLQJWKHSRROHGVDPSOHYDULDQFHRIWKHFRUUHVSRQGLQJSUHGLFWRU

5HVXOWV ,Q WKLV VHFWLRQ ZH FRPSDUH GLơHUHQW ZD\V RI GDWD LQWHJUDWLRQ GLơHUHQW PHWKRGV RI FODVVL¿FDWLRQ DQG IHDWXUH VHOHFWLRQ7DEOHSUHVHQWVFODVVL¿FDWLRQDFFXUDF\RI'/'$DQG690FODVVL¿HU

942

Sebastian Student et al. / Procedia Technology 22 (2016) 938 – 945

7DEOH7DEOHRIERRWVWUDSEDVHGDFFXUDF\UDWHIRUGL൵HUHQWGDWDW\SHDQGPHWKRGV 

&ODVVL¿FDWLRQPHWKRG

6HOHFWLRQPHWKRG

0HDQDFFXUDF\UDWH

FRQ¿GHQFHLQWHUYDO

&KLS6HT

'/'$

3/6



 

0LFURDUUD\

'/'$

%66:66



 

3URWHLQV

'/'$

%66:66



 

$OOGDWD

'/'$

*6



 

0LFURDUUD\&KLS6HT

'/'$

*6



 

0LFURDUUD\3URWHLQ

'/'$

*6



 

&KLS6HT3URWHLQ

'/'$

3/6



 

&KLS6HT

690

*6



 

0LFURDUUD\

690

*6



 

3URWHLQV

690

%66:66



 

$OOGDWD

690

*6



 

0LFURDUUD\&KLS6HT

690

*6



 

0LFURDUUD\3URWHLQ

690

*6



 

 ,Q DOO FDVHV WKH EHVW UHVXOWV ZDV REVHUYHG IRU WKH '/'$ FODVVL¿FDWLRQ PHWKRG )RU WKDW UHDVRQ DOO RWKHU FRPSDULVRQLVGRQHIRU'/'$FODVVL¿HU7KHFODVVLILFDWLRQDFFXUDF\LVREWDLQHGE\LQYHVWLJDWLRQRIWKHDFFXUDF\UDWH HVWLPDWHGE\WKHERRWVWUDSWHFKQLTXH>@7KHUHVXOWVDUHVKRZQRQ)LJ 

)LJ%RRWVWUDSEDVHGFODVVL¿FDWLRQDFFXUDF\ZLWKWKHFRQ¿GHQFHLQWHUYDOIRUGL൵HUHQWIHDWXUHVHOHFWLRQPHWKRGV

7KHEHVWDFFXUDF\UDWHREWDLQHGIRU PHUJHGGDWD0LFURDUUD\DQG3URWHLQGDWD LVDOVRVLJQL¿FDQWO\KLJKHUWKDQ DFFXUDF\ UDWH IRU VLQJOH GDWD DQDO\VLV 0LFURDUUD\3URWHLQ DFFXUDF\ UDWH YHUVXV EHVW UHVXOW IRU VLQJOH GDWDVHW 0LFURDUUD\GDWDVHW FRPSDULVRQWWHVWSYDOXH   )LJXUHSUHVHQWVFODVVL¿FDWLRQDFFXUDF\RI'/'$FODVVL¿HU7KLV¿JXUHVKRZVKRZWKHDFFXUDF\GHSHQGVRQWKH LQFUHDVHGQXPEHURIXVHGIHDWXUHV)RU'/'$FODVVL¿HUWKHEHVWUHVXOWVZHUHREVHUYHGIRUWKH*6IHDWXUHVHOHFWLRQ PHWKRG

943

Sebastian Student et al. / Procedia Technology 22 (2016) 938 – 945

 )LJ%RRWVWUDSEDVHGFODVVL¿FDWLRQDFFXUDF\E\VXFFHVVLYHJHQHVHWUHGXFWLRQVHOHFWHGIRUGL൵HUHQWIHDWXUHVHOHFWLRQPHWKRGVRIWKHEHVW FODVVL¿HUIRU0LFURDUUD\3URWHLQGDWDVHW

:HDOVRSUHVHQWUHVXOWVRIIHDWXUHVHWVVWDELOLW\DQDO\VLV)LJ,QJHQHUDOWKHUHDUHWZRGLơHUHQWDSSURDFKHVWR PHDVXUHWKHVWDELOLW\RIJHQHOLVWV7KH¿UVWDSSURDFKWDNHVLQWRDFFRXQWRQO\WKHFRQWHQWRIJHQHOLVWVDQGLJQRUHV WKHJHQHRUGHU7KHVHFRQGRQHGRHVQRWLJQRUHWKHJHQHRUGHURQFRPSDUHGOLVWV2QHRIWKHPRVWIUHTXHQWO\XVHG KHUHFULWHULDLVWKH3HUFHQWDJHRI2YHUODSSLQJ*HQHV32* >@EHORQJLQJWRWKH¿UVWFODVVRIVWDELOLW\PHDVXUHV 2Q)LJZHKDYHFRPSDUHGWKHVWDELOLW\IRUDOOWHVWHGIHDWXUHVHOHFWLRQPHWKRGV 7R YLVXDOL]H WKH VWDELOLW\ RI WKH RUGHUHG JHQH OLVWV ZH SORW WKH ER[SORWV RI HDFK JHQH UDQN LQ WKH OLVW / DJDLQVW UDQNVLQDOO E ERRWVWUDSLWHUDWLRQOLVWV / B E E   % :HVHWWKHOLPLWWRGHWHUPLQHZKLFKSRLQWVDUHH[WUHPHWR WKHUDQNRXWRIWKHJJHQHOLVW)LJ  5HVXOWV ,Q WKLV ZRUN ZH SURSRVHG DQ LQWHJUDWHG DQDO\VLV RI ELRORJLFDO GDWD IURP GLIIHUHQW PROHFXODU ELRPHGLFDO WHFKQLTXHV$VWKHWHVWGDWDZHKDYHXVHG%UHDVW,QYDVLYH&DUFLQRPDGDWD)RUWKHVDPHWLVVXHVZHKDYHFRPSDUHG FODVVLILFDWLRQUHVXOWVIRUJHQHH[SUHVVLRQPLFURDUUD\&KLS6HTDQGSURWHLQDVVD\H[SHULPHQWVREWDLQHGIRUWKHVDPH SDWLHQWV JURXS +RZHYHU ZKHQ ZH PHUJH WKHVH GDWD WKH ELJJHVW SUREOHP LV WKH KXJH QXPEHU RI IHDWXUHV ZLWK D OLPLWHGQXPEHURIVDPSOHV)RUWKDWUHDVRQWKHIHDWXUHVHOHFWLRQVWHSLVFUXFLDOLQKLJKGLPHQVLRQGDWDFODVVLILFDWLRQ SUREOHP ,Q VRPH FDVHV ZKHQ ZH KDYH XVHG ZURQJ IHDWXUH VHOHFWLRQ PHWKRG ZH KDYH REVHUYHG GHFUHDVLQJ RI WKH FODVVLILFDWLRQ DFFXUDF\ IRU WKH PHUJHG GDWD ,Q FRPSDULVRQ WR VLQJOH PROHFXODU GDWD DQDO\VLV WKH PHUJHG 0LFURDUUD\3URWHLQ GDWD DQDO\VLV VLJQLILFDQWO\ LPSURYHG WKH FODVVLILFDWLRQ DFFXUDF\ DQG IHDWXUH VHOHFWLRQ VWDELOLW\ LQGH[ 

944

Sebastian Student et al. / Procedia Technology 22 (2016) 938 – 945

^ƚĂďŝůŝƚLJŝŶĚĞdž W>^ '^ ^^ͬt^^



ŐĞŶĞƐ  )LJ6WDELOLW\SORWRIFODVVL¿FDWLRQREWDLQHGE\VXFFHVVLYHJHQHVHWUHGXFWLRQVHOHFWHGZLWKDOOIHDWXUHVHOHFWLRQPHWKRGVRIWKHEHVWFODVVL¿HUIRU 0LFURDUUD\3URWHLQGDWDVHW

)LJ6WDELOLW\LQGH[RIWKHFODVVL¿HURQWKHWHVWHGIHDWXUHVHOHFWLRQPHWKRGVIRUXVHGGDWDVHWV



Sebastian Student et al. / Procedia Technology 22 (2016) 938 – 945

945

ZĂŶŬŝŶƚŚĞŽƌŐŝŶĂůĚĂƚĂƐĞƚ

)LJ5DQNER[SORWVLQWKHERRWVWUDSVDPSOHVDJDLQVWUDQNLQWKHRULJLQDOGDWDVHWIRU*6%66:66DQG3/6IURPWKHOHIW IHDWXUHVHOHFWLRQ PHWKRGIRU0LFURDUUD\3URWHLQGDWDVHW

2XUUHVXOWVKDYHVKRZQWKDWLQWHJUDWHGDQDO\VLV ZLWKSURSHUIHDWXUHVHOHFWLRQDQGFODVVLILFDWLRQWHFKQLTXHVXVHG IRUODUJHVFDOHPHWDGDWDLVDSURPLVLQJWHFKQLTXHWKDWQHHGVIXUWKHULQYHVWLJDWLRQDQGGHYHORSPHQW $FNQRZOHGJHPHQWV 7KLVZRUNZDVVXSSRUWHGE\WKH*UDQW1R%.05$8W66 32,*-3 DQG 3%6%.)  5HIHUHQFHV >@ 1JX\HQ'95RFNH'07XPRUFODVVL¿FDWLRQE\SDUWLDOOHDVWVTXDUHVXVLQJPLFURDUUD\JHQHH[SUHVVLRQGDWD%LRLQIRUPDWLFV   >@ +DPLG-6+X35RVOLQ10/LQJ9*UHHQZRRG&0%H\HQH-'DWDLQWHJUDWLRQLQJHQHWLFVDQGJHQRPLFVPHWKRGVDQGFKDOOHQJHV+XP *HQRPLFV3URWHRPLFV  >@ 5HYHUWHU)9HJDV(2OOHU-0.HUQHO3&$GDWDLQWHJUDWLRQZLWKHQKDQFHGLQWHUSUHWDELOLW\%0&6\VW%LRO6XSSO 6  >@ +DLGLFK$%0HWDDQDO\VLVLQPHGLFDOUHVHDUFK+LSSRNUDWLD  >@ 6WXGHQW6)XMDUHZLF].6WDEOHIHDWXUHVHOHFWLRQDQGFODVVL¿FDWLRQDOJRULWKPVIRUPXOWLFODVVPLFURDUUD\GDWD%LRORJ\'LUHFW   >@ 6WXGHQW6'DQFK:LHU]FKRZVND0*RUF]HZVNL.%RU\V'$XWRPDWLF6HJPHQWDWLRQ6\VWHPRI(PLVVLRQ7RPRJUDSK\'DWD%DVHGRQ &ODVVL¿FDWLRQ6\VWHP%LRLQIRUPDWLFVDQG%LRPHGLFDO(QJLQHHULQJ  >@ +XDQJ'4XDQ@ .U]DQRZVNL:-3ULQFLSOHVRI0XOWLYDULDWH$QDO\VLV$8VHU¶V3HUVSHFWLYH--RXUQDORI([SHULPHQWDODQG&OLQLFDO&DQFHU5HVHDUFK   >@ %RVHU%(*X\RQ,09DSQLN9$WUDLQLQJDOJRULWKPIRURSWLPDOPDUJLQFODVVL¿HUV,Q3URFHHGLQJVRIWKHILIWKDQQXDOZRUNVKRSRQ&RP± SXWDWLRQDOOHDUQLQJWKHRU\$&0SS  >@.XPDU06.XPDUDVZDP\@@+RVNXOGVVRQ33/65HJUHVVLRQ0HWKRGV-RXUQDORI&KHPRPHWULFV >@.RRQLQ(9$OWVFKXO6)%RUN3.QRZOHGJHEDVHGDQDO\VLVRIPLFURDUUD\JHQHH[SUHVVLRQGDWDE\XVLQJVXSSRUWYHFWRUPDFKLQHV,Q 3URFHHGLQJVRIWKH1DWLRQDO$FDGHP\RI6FLHQFHVYRO SS  >@*X\RQ,:HVWRQ-%DUQKLOO69DSQLN9*HQHVHOHFWLRQIRUFDQFHUFODVVL¿FDWLRQXVLQJVXSSRUWYHFWRUPDFKLQHV-0DFKLQH/HDUQLQJ  >@(IURQ%7LEVKLUDQL5&URVV9DOLGDWLRQDQGWKH%RRWVWUDS(VWLPDWLQJWKH(UURU5DWHRID3UHGLFWLRQ5XOH7HFKQLFDO5HSRUW1RSS   >@'HQJ1$OOLVRQ--)DQJ+-$VK$6:DUH-(-U8VLQJWKHERRWVWUDSWRHVWDEOLVKVWDWLVWLFDOVLJQL¿FDQFHIRUUHODWLYHYDOLGLW\FRPSDULVRQV DPRQJSDWLHQWUHSRUWHGRXWFRPHPHDVXUHV+HDOWK4XDO/LIH2XWFRPHV >@=KDQJ7/L&2JLKDUD0(YDOXDWLQJUHSURGXFLELOLW\RIGL൵HUHQWLDOH[SUHVVLRQGLVFRYHULHVLQPLFURDUUD\VWXGLHVE\FRQVLGHULQJFRUUHODWHG PROHFXODUFKDQJHV%LRLQIRUPDWLFV