Efficiently Processing 4uery Intensive Databases over a Non ...

 (IILFLHQWO\3URFHVVLQJ4XHU\,QWHQVLYH'DWDEDVHVRYHUD1RQGHGLFDWHG/RFDO 1HWZRUN 3HGUR)XUWDGR '(,&,68& 8QLYHUVLGDGHGH&RLPEUD3RUWXJDO SQI#GHLXFSW  

$EVWUDFW



&KHDS FRPSXWLQJ SRZHU FDQ EH DSSOLHG IRU IOH[LELOLW\ DQG VFDODELOLW\ LQ PDQDJLQJ ODUJH GDWD YROXPHV LI WKH ULJKW FKRLFHV DUH PDGH FRQFHUQLQJ GDWD SODFHPHQW DQG SURFHVVLQJ 2XU ZRUN FRQFHQWUDWHVRQWKHXVHRIFKHDSFRPSXWLQJSRZHULQSRVVLEO\VORZQRQGHGLFDWHGORFDOQHWZRUNVWR DFKLHYH D FRPSXWLQJ SRZHU RYHU GHPDQGLQJ TXHU\LQWHQVLYH GDWDEDVHV WKDW ZRXOG EH XQDFKLHYDEOH ZLWKRXWH[SHQVLYHVSHFLDOL]HGKDUGZDUHDQGPDVVLYHO\SDUDOOHOV\VWHPV7KH1RGH3DUWLWLRQHG'DWD 0DQDJHPHQW 6\VWHP 13'0  ZRUNV RQ FRPSXWLQJ QRGHV RQ QRQGHGLFDWHG ORFDO QHWZRUNV ,Q WKLV SDSHUZHFRQFHQWUDWHRQTXHU\WUDQVIRUPDWLRQVUHTXLUHGIRUHIILFLHQWSURFHVVLQJRYHUDVSHFLDOL]HG TXHU\LQWHQVLYH VFKHPD 7KH GHFLVLRQ VXSSRUW EHQFKPDUN 73&+ LV XVHG DV D VWXG\ FDVH IRU WKH WUDQVIRUPDWLRQVDQGIRUH[SHULPHQWDODQDO\VLV





,QWURGXFWLRQ  7KHDSSOLFDWLRQVWKDWDOORZXVHUVWRDQDO\]HKLVWRULFDOGDWDLQRUJDQL]DWLRQVKDYHWRGHDOZLWK*LJD RU7HUDE\WHVRIGDWDZKLOHXVHUVUHTXLUHIDVWDQVZHUV([SHQVLYHVSHFLDOKDUGZDUHDUFKLWHFWXUHVDQG VSHFLDOL]HG VHUYLFHV DUH UHTXLUHG WR KDYH VFDODEOH V\VWHPV IRU VXFK PDVVLYH GDWD VHWV +RZHYHU DV UHWXUQRQLQYHVWPHQW FRQVLGHUDWLRQV EHFRPH FUXFLDO LQ WHFKQRORJ\ LQYHVWPHQWV RUJDQL]DWLRQV ZRUOGZLGHEHFRPHOHVVZLOOLQJWRLQYHVWODUJHVXPVDQGWLPHLQVSHFLDODUFKLWHFWXUHVDQGFRQVXOWLQJ ZKHQ PXFK FKHDSHU DQG VLPSOHU VROXWLRQV FDQ EH IRXQG ,W LV WKHUHIRUH GHVLUDEOH WKDW VFDODELOLW\ EH REWDLQHGG\QDPLFDOO\DQGDWUHDVRQDEOHFRVWZLWKRXWWKURZLQJDZD\ORZFRVWSODWIRUPV$IIRUGDEOH VFDODELOLW\VKRXOGEHDOHJROLNHG\QDPLFDOO\DGMXVWDEOHFDSDFLW\RIWKHV\VWHPLQZKLFK\RXSOXJRQH PRUH ³VWDQGDUG´ DQG UHDVRQDEO\ LQH[SHQVLYH FRPSRQHQW WR DGDSW WKH V\VWHP WR ODUJHU ZRUNORDGV ZLWKRXWXQZDQWHGFRVWV $ QHWZRUNHG FRPSXWLQJ HQYLURQPHQW FDQ EH DGDSWHG WR KDQGOH FRPSOH[ GDWDEDVH VFKHPDV DQG TXHULHV HIILFLHQWO\ :H HQYLVLRQ D V\VWHP ZKHUH ³SHHU´ FRPSXWLQJ SRZHU FDQ EH DGGHG WR LPSURYH SURFHVVLQJ VSHHG RYHU ODUJH GDWD VHWV VLPSO\ EH UHJLVWHULQJ QHZ QRGHV 2XU EURDG JRDO LV WR LQYHVWLJDWH WKH LPSOHPHQWDWLRQ RI D 1RGH 3DUWLWLRQHG 'DWD 0DQDJHPHQW 6\VWHP 13'0  ZRUNLQJ RYHUDVLPSOHQHWZRUN,QWKLVSDSHUZHLQYHVWLJDWHTXHU\WUDQVIRUPDWLRQVHQDEOLQJVXFKV\VWHPIRU KDQGOLQJ FRPSOH[ TXHU\LQWHQVLYH GDWDEDVHV DQG XVH D FDVH VWXG\ WKH 73& FRXQFLO SHUIRUPDQFH EHQFKPDUN 73&+ >@  WR YHULI\ WKH HIILFLHQF\ RI WKH VWUDWHJ\ 7KLV FRQWH[W KDV GLIIHUHQW FKDUDFWHULVWLFVIURPDW\SLFDOSDUDOOHOV\VWHPZKHUHSURFHVVLQJXQLWVDUHWLJKWO\OLQNHGE\KLJKVSHHG LQWHUFRQQHFWLRQVDQGLVDOVRGLIIHUHQWIURPDW\SLFDOGLVWULEXWHGV\VWHPZKHUHWKHFRQWHQWVRIHDFK VLWHLVW\SLFDOO\GHFLGHGRQDGHSDUWPHQWDORURUJDQL]DWLRQDOEDVHLQVWHDGRIHIILFLHQF\DQGSURFHVVLQJ VFDODELOLW\UHDVRQV

2XU PDLQ FRQWULEXWLRQV LQ WKLV SDSHU DUH ZH GLVFXVV WKH 13'0 V\VWHP ZLWK LWV TXHU\ WUDQVIRUPDWLRQVWUDWHJLHVZHGLVFXVVVFKHPDGDWDSDUWLWLRQLQJLQWKLVHQYLURQPHQWWREHDEOHWRFRSH HIILFLHQWO\ ZLWK KHDY\ GDWD SURFHVVLQJ LQ WKH SUHVHQFH RI QHWZRUN OLQNV ZLWKRXW SHUIRUPDQFH JXDUDQWHHV ZH HQJDJH LQ H[SHULPHQWDO DQDO\VLV WR VKRZ WKH DFKLHYHPHQWV DQG OLPLWDWLRQV RI WKH DSSURDFKZHSURSRVHVRPHLPSURYHPHQWVWRKDQGOHOLPLWDWLRQV 7KHSDSHULVRUJDQL]HGDVIROORZVVHFWLRQGLVFXVVHVUHODWHGZRUN6HFWLRQRYHUYLHZVWKHQRGH SDUWLWLRQHG PDQDJHPHQW V\VWHP 6HFWLRQ  GLVFXVVHV SDUWLWLRQLQJ RI TXHU\LQWHQVLYH VFKHPDV LQ 13'0 DQG VHFWLRQ  GLVFXVVHV PDMRU TXHU\ WUDQVIRUPDWLRQV LQ 13'0 6HFWLRQ  DQDO\]HV H[SHULPHQWDO SHUIRUPDQFH UHVXOWV XVLQJ WKH 73&+ GHFLVLRQ VXSSRUW EHQFKPDUN >@ 6HFWLRQ  FRQWDLQVFRQFOXGLQJUHPDUNV  5HODWHG:RUN  $ ODUJH ERG\ RI ZRUN H[LVWV LQ DSSO\LQJ SDUDOOHO SURFHVVLQJ WHFKQLTXHV WR UHODWLRQDO GDWDEDVH V\VWHPV 7KH REMHFWLYH LV WR DSSO\ HLWKHU LQWHUTXHU\ RU LQWUDTXHU\ SDUDOOHOLVP WR LPSURYH SHUIRUPDQFH,Q>@WKHDXWKRUVUHYLHZSDUDOOHOSURFHVVLQJLQGDWDEDVHVVSHHGXSDQGVFDODELOLW\LVVXHV LQ SDUDOOHO GDWDEDVH V\VWHPV LQFOXGLQJ VKDUHGQRWKLQJ HQYLURQPHQWV DV WKH EDVLV IRU WKH IXWXUH RI KLJKSHUIRUPDQFHGDWDEDVHFRPSXWLQJ 4XHU\SURFHVVLQJLQSDUDOOHODQGGLVWULEXWHGGDWDEDVHVKDVEHHQWKHIRFXVRIPXFKUHVHDUFKRQWKH GDWDEDVHILHOG>@$OWKRXJKVWDQGDUGL]HGUHODWLRQDODOJHEUDRSHUDWRUVXVHGLQGDWDEDVHV OHQG WKHPVHOYHV ZHOO WR SDUDOOHOL]DWLRQ LVVXHV DUH UDLVHG FRQFHUQLQJ PDLQO\ MRLQ SURFHVVLQJ DQG FRPPXQLFDWLRQ RYHUKHDGV DVVRFLDWHG ZLWK ERWK MRLQV DQG RWKHU RSHUDWLRQ QHHGV WR H[FKDQJH GDWD 3DUDOOHO+DVK-RLQDOJRULWKPV>@DQG3ODFHPHQW'HSHQGHQF\>@DUHW\SLFDOSURSRVDOVWR UHGXFHVRPHRIWKHRYHUKHDGDVVRFLDWHGZLWKMRLQRSHUDWLRQV7KHVWUDWHJLHVIURPSDUDOOHOSURFHVVLQJ DUH XVHIXO LQ RXU VHWWLQJ EXW WKH QRQGHGLFDWHG QHWZRUN LQ RXU FDVH GRHV QRW SURYLGH SHUIRUPDQFH JXDUDQWHHVDVDSDUDOOHOV\VWHPZRXOG 'DWDSODFHPHQWKDVEHHQDQRWKHULPSRUWDQWLVVXHLQSDUDOOHODQGGLVWULEXWHGGDWDEDVHDUFKLWHFWXUHV >    @ (DUO\ ZRUNV DUH UHYLHZHG LQ >@ 7KH LVVXH RI SDUWLWLRQLQJ DQG SODFHPHQW LV UHOHYDQW GXH PRVWO\ WR WKH ODUJH GDWD H[FKDQJH RYHUKHDG EHWZHHQ QRGHV UHTXLUHG WR UHSDUWLWLRQ UHODWLRQVIRUMRLQSURFHVVLQJDQGWRSURFHVVWXSOHVHOHFWLRQVDQGDJJUHJDWLRQVLQDSDUDOOHOVHWWLQJHJ VHH>@ 5HSDUWLWLRQLQJLQYROYHVIHWFKLQJWKHUHODWLRQIURPGLVNUHKDVKLQJLWDQGVHQGLQJWKHWXSOHV WR RWKHU QRGHV,QRUGHUWRPLQLPL]HUHSDUWLWLRQLQJUHTXLUHPHQWVSDUWLWLRQLQJDQGSODFHPHQWVKRXOG WDNH LQWR DFFRXQW WKH DWWULEXWHV XVHG LQ KDVKMRLQLQJ DQG DJJUHJDWLRQ 7KH VWUDWHJLHV LQ >  @ GHWHUPLQHWKHSDUWLWLRQLQJEDVHGRQWKHTXHU\ZRUNORDGWRUHGXFHWKHDPRXQWRIUHSDUWLWLRQLQJWKDWLV QHFHVVDU\:HGLVFXVVSDUWLWLRQLQJLQRXUVHWWLQJZLWKWKHREMHFWLYHWRPLQLPL]HFRVWO\GDWDH[FKDQJH RYHUKHDGVZKLFKLVLPSRUWDQWLQWKHQRQGHGLFDWHGQHWZRUNDVVXPSWLRQ  7KH1RGH3DUWLWLRQHG'DWD0DQDJHU  )LJXUH  SURYLGHV DQ RYHUYLHZ RI WKH 13'0 PRGXOHV :H DVVXPH D QRQGHGLFDWHG ORFDO DUHD QHWZRUN/$1 ZLWKWKHQRGHVOLQNHGE\DVZLWFKRUKXE7KHQRGHPDQDJHU10 LVUHVSRQVLEOHIRU DGGLQJ DQG UHPRYLQJ QRGHV IURP WKH V\VWHP LQFOXGLQJ GDWD UHRUJDQL]DWLRQ LQ VXFK DGYHQWV  PDQDJLQJDGGUHVVLQJLQIRUPDWLRQDWHDFKQRGHDQGRYHUVHHLQJUHSOLFDVDQGQRGHUHSODFHPHQWLQFDVH RI QRGH IDLOXUHV 7KH 6WRUDJH 0DQDJHU 60  LV UHVSRQVLEOH IRU WKH DGPLQLVWUDWLRQ RI REMHFW SHUVLVWHQFH 7KLV LQFOXGHV VWRUDJH UHWULHYDO UHSOLFDWLRQ DQG PLJUDWLRQ RI REMHFWV LQ UHVSRQVH WR UHTXHVWV (DFK QRGH KDV D ORFDO GDWDEDVH HQJLQH '%(  ZKLFK LV SDUW RI WKH VWRUDJH PDQDJHU  :H

FRQFHQWUDWHRQWKH'DWD0DQDJHU'0 ZKLFKLVUHVSRQVLEOHIRUVFKHPDSDUWLWLRQLQJDQGTXHU\LQJ )RUWKHXVHUWKH'DWD0DQDJHURIIHUVWKHGDWDPDQDJHPHQWIXQFWLRQDOLW\W\SLFDORID'%06EXWRQD GLIIHUHQWFRQWH[W   1RGH0DQDJHU10  'DWD0DQDJHU'0    6WRUDJH0DQDJHU60     8QGHUO\LQJ1HWZRUN

)LJXUH±13060RGXOHV

7XSOH 0DQDJHPHQW WKH 1306 LPSOHPHQWV SDUDOOHO SURFHVVLQJ RI WXSOHV RYHU WKH QHWZRUN 1RGHV SDUWLFLSDWLQJ LQ WKH VWRUDJH DQG SURFHVVLQJ RI FRPSOH[ TXHULHV DUH H[SHFWHG WR KDYH D VLPSOH GDWDEDVHOLNHHQJLQH'%( DVSDUWRIWKHLUVWRUDJHPDQDJHU60 WRKDQGOHORFDOGDWDPDQDJHPHQW $V LQ UHODWLRQDO GDWDEDVHV VFKHPDV FRPSULVH UHODWLRQV WDEOHV  PDGH RI VHWV RI WXSOHV DQG WKH 60 PDLQWDLQV WKH UHODWLRQ PHWDGDWD DW HDFK QRGH  *LYHQ D UHODWLRQ 5 ZLWK WXSOHV RI WKH IRUP 7DWWU«DWWUQ  WKH UHODWLRQ FDQ HLWKHU EH IXOO\ UHSOLFDWHG RU IXOO\ SDUWLWLRQHG XVLQJ D KDVK WDEOH IXQFWLRQDOLW\ ,Q WKH ODVW FDVH RQH DWWULEXWH PXVW EH FKRVHQ DV WKH WXSOH LGHQWLILHU WLG  E\ WKH SDUWLWLRQLQJDQGSODFHPHQWDOJRULWKP>@/LNHLQDGLVWULEXWHGKDVKWDEOH'+7 HQYLURQPHQW QRGHVUHFHLYHWXSOHVREMHFWV GHWHUPLQHGE\WKHKDVKYDOXHDQGWKHREMHFWWREHLQVHUWHGLVWKHWXSOH ZLWK RLG WLG 7KLV VWUDWHJ\ HQDEOHV D WXSOH WR EH ORRNHG XS GLUHFWO\ LQ WKHQRGHKROGLQJLWXVLQJ LWV WXSOH LGHQWLILHU &RQVLGHU IRU LQVWDQFH WKH UHODWLRQ3$57SDUWNH\QDPHPIJUEUDQGW\SH VL]H  ,I RLG SDUWNH\WKHUHODWLRQZLOOHLWKHUEHIXOO\UHSOLFDWHGLQWRDOOQRGHVRUIXOO\SDUWLWLRQHGE\SDUWNH\  3ODFHPHQWDQG-RLQ3URFHVVLQJLQ13'0 

,Q WKLV SDSHU ZH DUH DVVXPLQJ D GDWDEDVH WKDW LV PRVW IUHTXHQWO\ TXHULHG DQG RQO\ UHIUHVKHG ± ORDGHG ZLWK QHZ GDWD  RU XSGDWHG SHULRGLFDOO\ 7KLV LV WKH W\SLFDO VWUXFWXUH RI ODUJH GDWDEDVHV FRQWDLQLQJKLVWRULFDOGDWDLQDQRUJDQL]DWLRQ6XFKGDWDEDVHVDUHW\SLFDOO\IRUPHGE\DFROOHFWLRQRI VRPHODUJHUHODWLRQVFRQWDLQLQJGHWDLOGDWDDQGRWKHUVPDOOUHODWLRQVFRQWDLQLQJGHVFULSWLYHGDWD 7KH 64/ TXHU\ LQ )LJXUH  LV DQ H[DPSOH RI D TXHU\ VXEPLWWHG WR VXFK GDWDEDVH ZKHUH /, LV D UHODWLRQVWRULQJWKHLQIRUPDWLRQDERXWHDFKVDOHVRIHDFKSDUWIURPHDFKVXSSOLHUDWHDFKWLPHPRPHQW  6(/(&7 SBEUDQG\HDUBPRQWKVXPOBTXDQWLW\ FRXQW  )520 OLQHLWHP/,SDUW3WLPH7VXSSOLHU6 :+(5( -2,1/,376 $1' \HDUBPRQWK!    $1' VXSSOLHU µ;¶ *5283%@ ZKHUH WKHUH LV W\SLFDOO\ D ODUJH FHQWUDO UHODWLRQ ±IDFW ZLWK WXSOHV PHDVXULQJ TXDQWLWLHV DQG SRLQWLQJDWIRUHLJQNH\V WXSOHVIURPVHYHUDOVPDOOHUGHVFULSWLYHUHODWLRQV±GLPHQVLRQV±WRLGHQWLI\ WKHHQWLWLHVWKHPHDVXUHPHQWVUHIHUWR  3DUWLWLRQLQJWKH6FKHPD

 2QHRIWKHPRVWGHPDQGLQJRSHUDWLRQLQWKHTXHU\RI)LJXUHLVMRLQLQJWKHUHODWLRQV-2,1/,3 7 6 DQG LWV FRVW EHFRPHV PXFK ZRUVH ZKHQ ODUJH UHODWLRQV QHHG WR EH MRLQHG 7KH HTXL -2,1 RSHUDWLRQ HVVHQWLDOO\ PDWFKHV WXSOHV IURP UHODWLRQV KDYLQJ HTXDO YDOXHV IRU DQ DWWULEXWH 7KH QRGH SDUWLWLRQHG RUJDQL]DWLRQ LV D YHU\ LPSRUWDQW RSSRUWXQLW\ WR SDUWLWLRQ WKH ODUJH UHODWLRQV WKHUHIRUH SDUDOOHOL]LQJDQGGHFUHDVLQJWKHFRVWRIMRLQDQGRWKHURSHUDWLRQVVLJQLILFDQWO\ )RU HIILFLHQW SURFHVVLQJ RYHU WKH 13'0 V\VWHP WKH ODUJHVW KHDYLHVW UHODWLRQV IDFWV  DUH IXOO\ SDUWLWLRQHGLQWRWKHQRGHVDQGVPDOOHUUHODWLRQVGLPHQVLRQV DUHUHSOLFDWHG7KLVLVVKRZQLQ)LJXUH ZKHUHWKHGLPHQVLRQV'DUHUHSOLFDWHGDQGWKHIDFW)LVSDUWLWLRQHG+HDY\UHSOLFDWLRQLVIHDVLEOHLQ WKHVH VFKHPDV EHFDXVH WKH GDWD LV QRW FRQVWDQWO\ FKDQJHG DQG WKH RQO\ UHIUHVKPHQW LV EDVHG RQ SHULRGLFDOO\ORDGLQJQHZGDWD          

5HSOLFDWH'WXSOHV

'

'

 )DFW )

1RGHL '

'

)L

'LYLGHWXSOHV '

'

'

'

)LJXUH±3DUWLWLRQLQJWKH6WDU6FKHPD

7KH UHDVRQ IRU WKLV SODFHPHQW SDUWLWLRQLQJ WKH ODUJHVW UHODWLRQV  DQG UHSOLFDWLQJ WKH UHPDLQLQJ RQHVIXOO\ LVWREHDEOHWRSURFHVVWKHPRVWWLPHFRQVXPLQJRSHUDWLRQVMRLQV ORFDOO\DWHDFKQRGH WKHUHIRUHPLQLPL]LQJLQWHUQRGHFRPPXQLFDWLRQ7KLVZD\TXHU\SURFHVVLQJGRHVQRWEHFRPHDODUJH EXUGHQ WR WKH QHWZRUN DQG LW LV OHVV GHSHQGHQW RQ QHWZRUN EDQGZLGWK :H VKRZ ZK\ WKLV VWUDWHJ\ PLQLPL]HVLQWHUQRGHFRPPXQLFDWLRQQH[W 4XHULHVRYHUWKHVFKHPDRI)LJXUHW\SLFDOO\MRLQPRUHWKDQRQHVPDOOUHODWLRQGLPHQVLRQV' WR WKHODUJHUHODWLRQIDFW DQGWKHVHDUHYHU\WLPHFRQVXPLQJRSHUDWLRQV$MRLQEHWZHHQWZRUHODWLRQV 5 DQG 5 UHTXLUHV PDWFKLQJ WXSOHV IURP ERWK UHODWLRQV RQ DQ HTXLMRLQ DWWULEXWH 5$ 5$  7\SLFDOO\ WKH IDFW KDV D IRUHLJQ NH\ ). WKDW PXVW EH PDWFKHG WR WKH SULPDU\ NH\ 3. RI HDFK GLPHQVLRQLQHDFKMRLQ'L3.L )).L ,QRUGHUWRPLQLPL]HWKHWLPHWRSURFHVVWKLVRSHUDWLRQLWLV HVVHQWLDOWKDWUHODWLRQVEHSDUWLWLRQHG,IERWKDGLPHQVLRQDQGWKHIDFWDUHSDUWLWLRQHGE\WKHHTXLMRLQ NH\WKHMRLQFDQSURFHHGLQSDUDOOHODWHDFKQRGHLQGHSHQGHQWO\+RZHYHUDVWKHIDFWMRLQVZLWKRWKHU UHODWLRQVDVZHOOGLPHQVLRQVDQGIDFWV WKHMRLQSURFHVVZRXOGUHTXLUHKHDY\UHSDUWLWLRQLQJWKDWLV WKH WXSOHV ZRXOG KDYH WR EH UHKDVKHG E\ 3.L DQG H[FKDQJHG EHWZHHQ QRGHV IRU WKH MRLQ ZLWK HDFK GLPHQVLRQ  %\ UHSOLFDWLQJ WKH GLPHQVLRQV LQVWHDG ZH DUH DEOH WR SURFHVV WKH MRLQV DW HDFK QRGH LQGHSHQGHQWO\XVLQJWKHIROORZLQJSURSHUW\ 

)

 $ '  $ 'L  $ } $ 'Q     L Q

DOO B QRGHV  )  $ '  $ 'L  $ } $ 'Q     L Q M  M

 

$GGLWLRQDOO\ HYHQ WKRXJK H[SUHVVLRQ   LQFOXGHV D XQLRQ RSHUDWRU ZH XVXDOO\ GR QRW KDYH WR FROOHFWWKHSDUWLDOUHVXOWVRIWKHMRLQDQGXQLWHWKHPDWVRPHQRGHEHFDXVHZHFDQSURFHHGZLWKRWKHU SDUDOOHO TXHU\ RSHUDWRUV LQGHSHQGHQWO\ DW HDFK QRGH DV ZH ZLOO VKRZ LQ WKH QH[W VHFWLRQ RQ TXHU\ SURFHVVLQJ 7\SLFDO VFKHPDV LQFOXGH PRUH WKDQ RQH ODUJH UHODWLRQ )LJXUH  VXPPDUL]HV WKH 73&+ VFKHPD >@ ZKLFK UHSUHVHQWV RUGHULQJ DQG VHOOLQJ DFWLYLW\ /,OLQHLWHP 2RUGHUV 36SDUWVXSS 3SDUW 6 VXSSOLHU&FXVWRPHU %HVLGHVWKHVLWXDWLRQRI)LJXUHWKH73&+TXHU\VHWDOVRLQFOXGHVIUHTXHQW MRLQVEHWZHHQWZRRUPRUHODUJHUHODWLRQV 3



6

3DUWNH\

36

/,



2

FXVWNH\

RUGHUNH\

6XSNH\

)LJXUH73&+VFKHPD



&



,QWKLVVFKHPDUHODWLRQV36DQG&DUHUHODWLYHO\VPDOOGLPHQVLRQV DQGWKHUHIRUHDUHUHSOLFDWHG 7KH UHPDLQLQJ WKUHH UHODWLRQV /, 2 36  DUH IXOO\ SDUWLWLRQHG XVLQJ WKH KDVKLQJ VFKHPH 7KH SDUWLWLRQLQJ NH\V IRU WKRVH UHODWLRQV FDQ EH VSHFLILHG PDQXDOO\ RU GHWHUPLQHG XVLQJ DQ DXWRPDWLF ZRUNORDGEDVHGSDUWLWLRQLQJVWUDWHJ\>@7KHRXWSXWIURPWKHDOJRULWKPLVWKHDWWULEXWHWKDWVKRXOGEH XVHGIRUHDFKUHODWLRQDVWXSOHLGHQWLILHUIRUDSSO\LQJWKHKDVKLQJIXQFWLRQ:KHQDSSOLHGWR/,2DQG 36WKHDOJRULWKPGHWHUPLQHVWKDW/,DQG2EHSDUWLWLRQHGE\RUGHUNH\PRVWIUHTXHQWDQGFRVWO\MRLQ RYHU WKH 73&+ TXHU\ ZRUNORDG  -RLQV EHWZHHQ WKHVH HTXLSDUWLWLRQHG UHODWLRQV DUH DOVR KDQGOHG ORFDOO\ ZLWKRXW DQ\ UHSDUWLWLRQLQJ RYHUKHDGV EHFDXVH RI WKH SURSHUW\ LQ   *LYHQ HTXLSDUWLWLRQHG UHODWLRQV)DQG)RYHUDVHWRIQRGHV1DQGWKHLUHTXLMRLQDWWULEXWH( )



 (  ) 

M

1 



) M  (  )  M

EHFDXVH L M 1  L z M)L  ( ) M I  



:LWK WKLV VFKHPH UHSDUWLWLRQLQJ ZKLFK LQYROYHV UHKDVKLQJ WXSOHV DQG VHQGLQJ WKHP WR WKH FRUUHVSRQGLQJ QRGH LV RQO\ UHTXLUHG IRU QRQHTXLSDUWLWLRQHG UHODWLRQV ZKLFK LQ WKH FDVH RI RXU SODFHPHQWIRU73&+LQYROYHVRQO\WKH36DQGIHZTXHULHV  3URFHVVLQJ-RLQ2SHUDWLRQV

 )LJXUHVKRZVWZRH[DPSOHVRITXHU\JUDSKVZKHUHYHUWLFHVFRUUHVSRQGWRUHODWLRQVDQGHGJHVDUH HTXLMRLQV)RUVLPSOLFLW\ZHDVVXPHDOOHTXLMRLQVDUHRQGLIIHUHQWDWWULEXWHV   ' ' ' ' '   )L  )  )L L )L  ' '   ' '  ' )LJXUH±4XHU\([DPSOH*UDSKV



7KH ILUVW VWHS RFFXUV LQ SDUDOOHO DW DOO QRGHV :H WDNH DGYDQWDJH RI WKH UHSOLFDWLRQ RI VPDOO  GLPHQVLRQV WR SURPRWH SDUWLDO MRLQV RI IDFWV ZLWK HTXLMRLQHG GLPHQVLRQV ILUVW 7KLV LV EDVHG RQ

DSSO\LQJWKHVDPHSURSHUW\RIVHFWLRQWRHDFKSDUWLWLRQHGIDFWLQHDFKQRGH)[$'[%«[]'Q œ 8)L[$'[%«[]'Q ,WSURPRWHVMRLQLQJZLWKVPDOOHUUHODWLRQVGLPHQVLRQV ILUVWZKLFKPRVW IUHTXHQWO\ LQWURGXFHV UHVWULFWLRQV WKDW UHVXOW LQ YHU\ JRRG LQWHUPHGLDWH UHVXOW VHOHFWLYLW\ (DFK SDUWLWLRQHGIDFWLVHTXLMRLQHGWRWKHGLPHQVLRQVLWLVOLQNHGZLWK$WWKHHQGRIWKLVSKDVHZHDUHOHIW ZLWKLQWHUPHGLDWHUHVXOWVIRUHDFKSDUWLWLRQHGIDFW )LJXUH  LOOXVWUDWHV WKH ILUVW VWHS DSSOLHG WR WKH TXHU\ JUDSK RI )LJXUH  )LJXUH D  VKRZV WZR DOWHUQDWLYHVDVGLPHQVLRQV'DQG'FDQEHHTXLMRLQHGZLWKRQHRUWKHRWKHUIDFW7KHDFWXDOMRLQ RUGHU EHWZHHQ HDFK IDFW DQG HTXLMRLQHG GLPHQVLRQV DQG ZKLFK RI WKH DOWHUQDWLYHV LQ )LJXUH D LV FKRVHQVKRXOGEHGHWHUPLQHGFRVWEDVHG           

' 

'  '

)L

)L '

'

'

'

' RU

'

)L

'

)L '

' VW

D $OWHUQDWLYHVLQJOHQRGHSURFHVVLQJRI TXHU\

' '

)L

' )L ' 

E 6LQJHQRGHSURFHVVLQJRIQGTXHU\ )LJXUH±,QGHSHQGHQW1RGH4XHU\3URFHVVLQJ

7KH ILUVW VWHS SURGXFHV DV PDQ\ VHWV RI LQWHUPHGLDWH UHVXOWV ,5  LQ HDFK QRGH DV WKHUH DUH SDUWLWLRQHGUHODWLRQVHDFKRIWKHIRUP)L[$'[[%«[]'\ 7KHVH,5VIRUPDFKDLQZLWKRQO\WZR IDFWVLQWKHH[DPSOHVRI)LJXUH ZKLFKFDQQRWEHSURFHVVHGORFDOO\EHFDXVHWKH\DUHQRWSDUWLWLRQHG RQ WKH VDPH  HTXLMRLQ NH\V 7KHVH DUH QRZ SURFHVVHG XVLQJ WKH SDUDOOHO KDVKMRLQ DOJRULWKP 3+- >@7RDSSO\WKH3+-FRQVLGHULQJDMRLQEHWZHHQWZR,5VERWKDUHUHSDUWLWLRQHGRQWKHHTXL MRLQ DWWULEXWH KDVKSDUWLWLRQLQJ  DQG DVVLJQHG D MRLQ SURFHVVLQJ QRGH DFFRUGLQJ WR WKH KDVKYDOXH UHSDUWLWLRQLQJFRVW 7KHMRLQLVWKHQSURFHVVHGLQSDUDOOHOE\WKHQRGHV7KHH[DFWRUGHURIH[HFXWLRQ RIORQJFKDLQVVKRXOGEHGHWHUPLQHGEDVHGRQFRVWV7KHFRPSOHWHDOJRULWKPLV 

3URGXFH5HGXFHG,QWHUPHGLDWH5HVXOWV,5   3URFHVV LQ SDUDOOHO DOO ORFDOO\ FRPSXWDEOH MRLQV DURXQG WKH SDUWLWLRQHG UHODWLRQV ZLWK UHSOLFDWHG GLPHQVLRQV  ,Q WKLV VWHS DSSO\ WKH VHOHFWLRQ RSHUDWRUV UHVWULFWLRQV RQ WKH GLPHQVLRQV DQG RQ WKH SDUWLWLRQHGIDFWLWVHOI SURGXFLQJVHOHFWLYHLQWHUPHGLDWHUHVXOWV,5V   D3DUDOOHO+DVKMRLQ,5V3+-,5   'HWHUPLQHEHVWMRLQRUGHUIRUWKH,5V )RUHDFK,5MRLQVWHS 5HSDUWLWLRQLQWHUYHQLQJ,5VE\MRLQDWWULEXWH 3URFHVVWKHMRLQLQSDUDOOHODWDOOQRGHV  $IWHUDWKHTXHU\SURFHVVLQJSURFHHGVDQGDWVRPHSRLQWSDUWLDOUHVXOWVDUHPHUJHG

7KH SUHYLRXV DOJRULWKP ZDV GHVFULEHG VHTXHQWLDOO\ EXW WKH WZR VWHSV GHVFULEHG DERYH VKRXOG EH SLSHOLQHGIRUPD[LPXPHIILFLHQF\)RULQVWDQFHVWHSUHSDUWLWLRQLQJRUUHSOLFDWLRQFDQEHRYHUODSSHG ZLWKVWHSDV,5VDUHEHLQJFRQVWUXFWHG  

4XHU\3URFHVVLQJLQWKH13'0  :HKDYHDOUHDG\GLVFXVVHGWKHHIILFLHQWSURFHVVLQJRIMRLQVLQVHFWLRQ,QWKHUHPDLQLQJRIWKLV VHFWLRQZHGLVFXVVRWKHUTXHU\FRQVWUXFWV &RQVLGHU D VLPSOH VXP TXHU\ H[DPSOH RYHU D TXHU\LQWHQVLYH VFKHPD (DFK QRGH QHHGV WR DSSO\ H[DFWO\ WKH VDPH LQLWLDO TXHU\ RQ LWV SDUWLDO GDWD DQG WKH UHVXOWV DUH PHUJHG E\ DSSO\LQJ WKH VDPH TXHU\DJDLQDWWKHPHUJLQJQRGHZLWKWKHSDUWLDOUHVXOWVFRPLQJIURPWKHSURFHVVLQJQRGHV)LJXUH LOOXVWUDWHVWKLVSURFHVV 680;  RYHUQ)$&7 6HQGWR*5283%@ 3$ %HUQVWHLQ HW DO ³4XHU\ 3URFHVVLQJ LQ D 6\VWHP IRU 'LVWULEXWHG 'DWDEDVHV 6''O ´ $&0 7UDQV'DWDEDVH6\VWHPVYROQRSS'HF 

>@ &KHQ +DR & /LX ³$Q (IILFLHQW $OJRULWKP IRU3URFHVVLQJ'LVWULEXWHG4XHULHV8VLQJ3DUWLWLRQ 'HSHQGHQF\´,&3$'6  >@ 'DYLG - 'H:LWW -LP *UD\ ³3DUDOOHO 'DWDEDVH 6\VWHPV 7KH )XWXUH RI +LJK 3HUIRUPDQFH 'DWDEDVH3URFHVVLQJ´&RPPXQLFDWLRQVRIWKH$&0  >@'DYLG-'H:LWW5REHUW*HUEHU0XOWLSURFHVVRU+DVK%DVHG-RLQ$OJRULWKPV3URFVWK9/'% &RQIHUHQFH  >@3)XUWDGR³+DVKEDVHG3ODFHPHQWDQG3URFHVVLQJIRU(IILFLHQW1RGH3DUWLWLRQHG4XHU\,QWHQVLYH 'DWDEDVHV´,&3$'6  >@+XD.$DQG/HH&$Q$GDSWLYH'DWD3ODFHPHQW6FKHPHIRU3DUDOOHO'DWDEDVH&RPSXWHU 6\VWHPV´3URFV9/'%&RQI%ULVEDQH$XVWUDOLD  >@.LPEDOO5 7KH'DWD:DUHKRXVH7RRONLW1HZ@ 0 .LWVXUHJDZD + 7DQDND DQG 7 0RWRRND $SSOLFDWLRQ RI KDVK WR GDWDEDVH PDFKLQH DQG LWV DUFKLWHFWXUH1HZ*HQHUDWLRQ&RPSXWLQJ   >@ /LX &KHQJZHQ DQG +DR &KHQ ³$ +DVK 3DUWLWLRQ 6WUDWHJ\ IRU 'LVWULEXWHG 4XHU\ 3URFHVVLQJ´ ('%7  >@ /LX &KHQJZHQ +DR &KHQ :DUUHQ .UXHJHU ³$ 'LVWULEXWHG 4XHU\ 3URFHVVLQJ 6WUDWHJ\ 8VLQJ 3ODFHPHQW'HSHQGHQF\´3URFWK,&'(SS)HE  >@-5DRHWDO³$XWRPDWLQJSK\VLFDOGDWDEDVHGHVLJQLQDSDUDOOHOGDWDEDVH´6,*02'&RQIHUHQFH   >@6KDVKD'HQQLVHWDO³2SWLPL]LQJ(TXLMRLQ4XHULHV,Q'LVWULEXWHG'DWDEDVHV:KHUH5HODWLRQV $UH+DVK3DUWLWLRQHG´$&07UDQVRQ'DWDEDVH6\VWHPV91-XQH  >@ 7HUDGDWD &RUSRUDWLRQ 'DWDEDVH &RPSXWHU 6\VWHP &RQFHSWV DQG )DFLOLWLHV 'RFXPHQW & 7HUDGDWD&RUSRUDWLRQ/RV$QJHOHV2FW  >@73&%HQFKPDUN+7UDQVDFWLRQ3URFHVVLQJ&RXQFLO-XQH$YDLODEOHDWKWWSZZZWSFRUJ  >@