This can significantly reduce the running time of data-intensive jobs. .... Starfish is a self-tuning system for big data analytics with a Cost-based Optimizer.
,QWHUQDWLRQDO-RXUQDORI/DWHVW7UHQGVLQ(QJLQHHULQJDQG7HFKQRORJ\,-/7(7
3KDVHEDVHG5HVRXUFH$ZDUH6FKHGXOHUZLWK -RE3URILOLQJIRU0DS5HGXFH 'KDQ\D6XGKDNDUDQ PG Scholar Department of Computer Science and Engineering Sree Buddha College of Engineering, Alappuzha, Kerala, India
6KLQL5HQMLWK $VVLVWDQW3URIHVVRU Department of Computer Science and Engineering Sree Buddha College of Engineering, Alappuzha, Kerala, India $EVWUDFW0DS5HGXFHKDVEHFRPHDSRSXODUPRGHOIRUGDWDPLQLQJFRPSXWDWLRQLQWKHVHGD\V0DS5HGXFH EUHDN GRZQ HDFK MRE LQWR VPDOO PDS WDVNV DQG UHGXFH WDVNV 7KHVH WDVNV DUH H[HFXWHG LQ SDUDOOHO DFURVV D ODUJHQXPEHURIPDFKLQHV7KLVFDQVLJQLILFDQWO\UHGXFHWKHUXQQLQJWLPHRIGDWDLQWHQVLYHMREV7KHH[LVWLQJ VFKHGXOHUV IRU0DS5HGXFH DUHIRFXVLQJRQ VFKHGXOLQJDW WKHWDVNOHYHODQGWKH\ RIIHU RQO\VXERSWLPDO MRE SHUIRUPDQFH 7KLV LV EHFDXVH UHVRXUFH UHTXLUHPHQWV RI WKH WDVNV DUH YDU\LQJ KLJKO\ GXULQJ WKHLU OLIHWLPH 7KHVHYDU\LQJUHTXLUHPHQWVPDNHLWGLIILFXOWIRUWDVNOHYHOVFKHGXOHUVWRHIIHFWLYHO\XWLOL]HDYDLODEOHUHVRXUFHV WR UHGXFH MRE H[HFXWLRQ WLPH 7R DGGUHVV WKLV OLPLWDWLRQ D ILQHJUDLQHG SKDVH DQG UHVRXUFHDZDUH 0DS5HGXFH 6FKHGXOHU ZDV LQWURGXFHG WKDW GLYLGHV WDVNV LQWR SKDVHV ZKHUH HDFK SKDVH KDV D FRQVWDQW UHVRXUFH XVDJH SURILOH DQG SHUIRUPV VFKHGXOLQJ DW WKH SKDVH OHYHO -RE SURILOLQJ LV QRW FRQVLGHUHG LQ WKLV VFKHGXOHU7RPHHWWKHQHHGIRUMRESURILOLQJ6WDUILVKDMRESURILOHUFDQEHEXLOWLQWRWKHV\VWHP6WDUILVKFDQ SURYLGH DFFXUDWH UHVRXUFH LQIRUPDWLRQ WKDW FDQ EH XVHG E\ WKH VFKHGXOHU VR WKDW LW FDQ WDNH HIIHFWLYH VFKHGXOLQJ GHFLVLRQV DQG UHGXFH WKH MRE H[HFXWLRQ WLPH %\ XVLQJ VXFK D MRE SURILOHU LQWHJUDWHG ZLWKLQ WKH VFKHGXOHUWKHSHUIRUPDQFHRIWKHVFKHGXOHULVHIIHFWLYHWKDQWKHH[LVWLQJWDVNOHYHOVFKHGXOHUV
.H\ZRUGV±'DWDLQWHQVLYHFRPSXWDWLRQ0DS5HGXFH6FKHGXOHU3KDVH/HYHO6FKHGXOLQJ-RE3URILOLQJ6WDUILVK
,,1752'8&7,21 1RZDGD\V WKH QHHG IRU SURFHVVLQJ WKH ODUJHVFDOH GDWD KDV EHFRPH FULWLFDO 0DS5HGXFH LV D SDUDOOHO SURJUDPPLQJ PRGHOIRUODUJHVFDOHGDWDLQWHQVLYHFRPSXWDWLRQV>@,Q0DS5HGXFHDMRELVDFROOHFWLRQRI0DSDQG5HGXFHWDVNV WKDW FDQ EH VFKHGXOHG FRQFXUUHQWO\ RQ PXOWLSOH PDFKLQHV ,W UHVXOWV LQ VLJQLILFDQW UHGXFWLRQ LQ MRE UXQQLQJ WLPH 0DS5HGXFHLVEHLQJXVHGE\PDQ\ODUJHFRPSDQLHVVXFKDV*RRJOH)DFHERRNHWFWRSURFHVVODUJHVFDOHGDWD7KH PDLQFRPSRQHQWRID0DS5HGXFHV\VWHPLVWKHMREVFKHGXOHU7KHUROHRIWKHMREVFKHGXOHULVWRFUHDWHDVFKHGXOHRI 0DS DQG 5HGXFH WDVNV VSDQQLQJ RQH RU PRUH MREV WKDW PLQLPL]HV WKH MRE FRPSOHWLRQ WLPH DQG PD[LPL]HV WKH UHVRXUFHXWLOL]DWLRQ$SDFKH +DGRRS 0DS5HGXFHLV WKHSRSXODULPSOHPHQWDWLRQRI 0DS5HGXFH>@,W FRQVLVWVRI D QXPEHURIPDFKLQHQRGHVZKHUHRQHQRGHDFWLQJDVWKHPDVWHUQRGHUHVRXUFHPDQDJHU DQGRWKHUVDFWLQJDVWKHVODYH QRGHVORFDOQRGHPDQDJHUV 0DVWHUQRGHLVUHVSRQVLEOHIRUVFKHGXOLQJWDVNVRQWKHVODYHQRGHV 0DS5HGXFHWDVNVFDQEHGLYLGHGLQWRWZRPDSWDVNDQGUHGXFHWDVN$PDSWDVNFDQEHGLYLGHGLQWRWZRSKDVHV PDS SKDVH DQG PHUJH SKDVH $ UHGXFH WDVN FDQ EH GLYLGHG LQWR WKUHH SKDVHV VKXIIOH SKDVH VRUW SKDVH DQG UHGXFH SKDVH'XULQJWKHPDSSKDVHWKHGDWDEORFNWKDWLVVWRUHGLQWKHFRUUHVSRQGLQJILOHV\VWHP(J+DGRRS'LVWULEXWHG )LOH6\VWHP LVIHWFKHGDQGVRPHXVHUGHILQHGIXQFWLRQPD\EHDSSOLHG7KLVUHVXOWLVVWRUHGLQDEXIIHU'XULQJPHUJH SKDVHRXWSXWIURPWKHPDSSKDVHDUHJURXSHGEDVHGRQLQWHUPHGLDU\NH\VDQGWKHUHVXOWLVVWRUHGLQPXOWLSOHILOHV 6LPLODUO\DUHGXFHWDVNFDQEHGLYLGHGLQWRWKUHHSKDVHVVKXIIOHSKDVHVRUWSKDVHDQGUHGXFHSKDVH'XULQJVKXIIOH SKDVH RXWSXW IURP WKH PXOWLSOH ILOHV DUH FROOHFWHG DQG SHUIRUP PHUJH RSHUDWLRQ GXULQJ WKH VRUW SKDVH 'XULQJ WKH UHGXFHSKDVHWKHRXWSXWLVWKHQSURFHVVHGDFFRUGLQJWRWKHXVHUGHILQHGIXQFWLRQDQGWKHILQDOUHVXOWLVZULWWHQEDFNWR WKH FRUUHVSRQGLQJ ILOH V\VWHP (DFK RI WKHVH SKDVHV KDYH GLIIHUHQW UHVRXUFH UHTXLUHPHQWV 6R WKHUH LV D QHHG IRU FKDUDFWHUL]LQJWKHUHVRXUFHXVDJH
9RO,VVXH1RYHPEHU
,661;
,QWHUQDWLRQDO-RXUQDORI/DWHVW7UHQGVLQ(QJLQHHULQJDQG7HFKQRORJ\,-/7(7
7KHUHVWRIWKHSDSHULVRUJDQL]HGDVIROORZV6RPHRIWKHUHODWHGZRUNVLQWKHDUHDRI0DS5HGXFHVFKHGXOLQJLV SUHVHQWHGLQVHFWLRQ,,3URSRVHGSKDVHDQGUHVRXUFHDZDUHVFKHGXOHUZLWKMRESURILOLQJV\VWHPLVH[SODLQHGLQVHFWLRQ ,,,&RQFOXGLQJUHPDUNVDUHJLYHQLQVHFWLRQ,9 ,,
5(/$7(':25.6
7KHUH DUH VHYHUDO VFKHGXOLQJ VFKHPHV XVHG LQ +DGRRS 0DS5HGXFH VXFK DV ),)2 VFKHGXOHU )DLU VFKHGXOHU >@ &DSDFLW\ VFKHGXOHU >@ 5HVRXUFH $GDSWLYH VFKHGXOHU 'RPLQDQW 5HVRXUFH )DLU VFKHGXOHU 7KH GHIDXOW 6FKHGXOLQJ DOJRULWKPVXSSRUWHGLV),)2'HIDXOW+DGRRSVFKHGXOHURSHUDWHVXVLQJD),)2TXHXH>@7KHVFKHGXOLQJDOJRULWKP ZDV LQWHJUDWHG ZLWKLQ WKH -RE 7UDFNHU -RE 7UDFNHU SXOOV MREV IURP D ZRUN TXHXH ROGHVW MRE ILUVW $IWHU D MRE LV GLYLGHGLQWRLQGHSHQGHQWWDVNVWKH\DUHHQGHGLQWRWKHTXHXHDQGDOORWWHGWRIUHHVORWVDVWKH\JHWDFTXLUDEOHRQ7DVN 7UDFNHUQRGHV7KLVVFKHGXOLQJVFKHPHZDVVLPSOHWRLPSOHPHQWDQGHIILFLHQW7KLVVFKHGXOHUKDGQRFRQFHSWRIWKH SULRULW\RUVL]HRIWKHMRE$QRWKHUW\SHRIVFKHGXOHULV)DLUVFKHGXOHU7KLVVFKHGXOHUZDVGHYHORSHGDW)DFHERRNWR PDQDJHDFFHVVWRWKHLU+DGRRSFOXVWHUDQGVXEVHTXHQWO\UHOHDVHGWRWKH+DGRRSFRPPXQLW\7KLVVFKHGXOHUDLPVWR JLYH HYHU\ XVHU D IDLU VKDUH RI WKH FOXVWHU FDSDFLW\ RYHU WLPH 8VHUV PD\ DVVLJQ MREV WR SRROV ZLWK HDFK SRRO DOORFDWHGDJXDUDQWHHGPLQLPXPQXPEHURI0DSDQG5HGXFHVORWV7DVNVDUHWKHUHIRUHVFKHGXOHGLQDQLQWHUOHDYHG IDVKLRQ VXSSRUWHG RQ WKHLU SULRULW\ ZLWKLQ WKHLU SRRO 7KH VFKHGXOLQJ VFKHPH GHYHORSHG DW @7KHFRQFHSWRIµMREVORWV¶ ZDVLQWURGXFHG-REVORWLVDQH[HFXWLRQVORWWKDWLVERXQGWRDSDUWLFXODUMREDQGDSDUWLFXODUWDVNW\SHUHGXFHRU PDS ZLWKLQ WKDW MRE *RDO RI WKH VFKHGXOHU LV WR GHWHUPLQH WKH EHVW SRVVLEOH SODFHPHQW RI WDVNV DFURVV WKH 7DVN 7UDFNHUV WR PD[LPL]H UHVRXUFH XWLOL]DWLRQ LQ WKH FOXVWHU 7KH VFKHGXOLQJ DOJRULWKP SURYLGHV VFKHGXOHU ZLWK WKH DGDSWDELOLW\QHHGHGWRUHVSRQGWRFKDQJLQJFRQGLWLRQVLQUHVRXUFHGHPDQGDQGDYDLODELOLW\7RDGGUHVVWKHSUREOHP RI IDLU DOORFDWLRQ RI PXOWLSOH W\SHV RI UHVRXUFHV WR XVHUV 'RPLQDQW 5HVRXUFH )DLU VFKHGXOLQJ '5) VFKHPH ZDV GHYHORSHG >@ ,Q KHWHURJHQHRXV HQYLURQPHQW UHVRXUFH DOORFDWLRQ WR D XVHU VKRXOG EH GHWHUPLQHG E\ WKH XVHU¶V GRPLQDQWVKDUH'RPLQDQWVKDUHLVWKH PD[LPXP VKDUHWKDWWKHXVHUKDVEHHQDOORFDWHGRIDQ\UHVRXUFH,QVLQJOH UHVRXUFH HQYLURQPHQW '5) UHGXFHV WR PD[PLQ IDLUQHVV IRU WKDW UHVRXUFH '5) VHHNV WR PD[LPL]H WKH PLQLPXP GRPLQDQWVKDUHDFURVVDOOXVHUV ,,, 352326('6@FDQEHXVHGIRUWKHVFKHGXOHUGHVLJQ7KHPDLQFRPSRQHQWVLQWKLV VFKHGXOHUDUHPDVWHUQRGHORFDOQRGHPDQDJHUVDQGDMRESURJUHVVPRQLWRU7KHVFKHGXOLQJGHFLVLRQVDUHPDGHDW WKHPDVWHUQRGHWKHORFDOQRGHPDQDJHUVDUHUHVSRQVLEOHIRUFRRUGLQDWLQJSKDVHWUDQVLWLRQVZLWKWKHVFKHGXOHUDQG WKHMRESURJUHVVPRQLWRULVUHVSRQVLEOHIRUFDSWXULQJSKDVHOHYHOSURJUHVVLQIRUPDWLRQ7KLVVHFWLRQLVRUJDQL]HGLQWR IRXUVXEVHFWLRQVZKLFKGHVFULEHWKHVFKHGXOLQJ PHFKDQLVPVFKHGXOLQJ DOJRULWKPWKHV\VWHPDUFKLWHFWXUHDQGMRE SURILOLQJ A. Phase- Level Scheduling Scheme – 6WHSVLQSKDVHOHYHOVFKHGXOLQJPHFKDQLVPLVH[SODLQHGDVEHORZ 6WHS (DFK QRGH PDQDJHU SHULRGLFDOO\ VHQGV D KHDUWEHDW PHVVDJH WR WKH VFKHGXOHU :KHQ D WDVN QHHGV WR EH VFKHGXOHGWKHVFKHGXOHUUHSOLHVWRWKHKHDUWEHDWPHVVDJHZLWKDWDVNVFKHGXOLQJUHTXHVW 6WHS7KHQRGHPDQDJHUWKHQODXQFKHVWKHWDVN 6WHS(DFKWLPHDWDVNILQLVKHVH[HFXWLQJDSDUWLFXODUSKDVHHJVKXIIOHSKDVHRIWKHUHGXFHWDVN WKHWDVNDVNVWKH QRGHPDQDJHUIRUDSHUPLVVLRQWRVWDUWWKHQH[WSKDVH(JUHGXFHSKDVHRIWKHWDVN 6WHS7KHORFDOQRGHPDQDJHUWKHQIRUZDUGVWKHSHUPLVVLRQUHTXHVWWRWKHVFKHGXOHUWKURXJKWKHUHJXODUKHDUWEHDW PHVVDJH 6WHS*LYHQDMRE¶VSKDVHOHYHOUHVRXUFHUHTXLUHPHQWVDQGLWVFXUUHQWSURJUHVVLQIRUPDWLRQWKHVFKHGXOHUGHFLGHV ZKHWKHUWRVWDUWDQHZWDVNRUDOORZDSDXVHGWDVNWREHJLQLWVQH[WSKDVHHJWKHUHGXFHSKDVH DQGWKHQLQIRUPV WKHQRGHPDQDJHUDERXWWKHVFKHGXOLQJGHFLVLRQ
9RO,VVXH1RYHPEHU
,661;
,QWHUQDWLRQDO-RXUQDORI/DWHVW7UHQGVLQ(QJLQHHULQJDQG7HFKQRORJ\,-/7(7
6WHS)LQDOO\RQFHWKHWDVNLVDOORZHGWRH[HFXWHWKHQH[WSKDVHWKHQRGHPDQDJHUJUDQWVWKHSHUPLVVLRQWRWKHWDVN SURFHVV 6WHS2QFHWKHWDVNLVILQLVKHGWKHWDVNVWDWXVLVUHFHLYHGE\WKHQRGHPDQDJHU 6WHS7KHQLWLVIRUZDUGHGWRWKHVFKHGXOHU B. Phase-Level Scheduling Algorithm – 7KHPDLQVWHSVLQSKDVHOHYHOVFKHGXOLQJDOJRULWKPLVDVIROORZV 8SRQUHFHLYLQJDVWDWXVPHVVDJHIURPDPDFKLQHµQ¶REWDLQLWVUHVRXUFHXWLOL]DWLRQ &RPSXWHWKHVHWRIFDQGLGDWHSKDVHVDQGVHOHFWSKDVHVLQDQLWHUDWLYHPDQQHU 'XULQJHDFKLWHUDWLRQIRUHDFKSKDVHFDOFXODWHWKHXWLOLW\IXQFWLRQ8LQ 8LQ 8IDLUQHVVLQ Į8SHUILQ 6HOHFWWKHSKDVHZLWKKLJKHVWXWLOLW\IRUVFKHGXOLQJ 5HSHDW E\ UHFRPSXWLQJ WKH XWLOLW\ RI DOO WKH SKDVHV LQ WKH FDQGLGDWH SKDVH VHW DQG VHOHFW WKH QH[W SKDVH WR EH VFKHGXOHG (QGVZKHQWKHFDQGLGDWHVHWLVHPSW\ 8SRQUHFHLYLQJDKHDUWEHDWPHVVDJHIURPDQRGHPDQDJHUUHSRUWLQJUHVRXUFHDYDLODELOLW\RQWKHQRGHWKHVFKHGXOHU PXVWVHOHFWZKLFKSKDVHVKRXOGEHVFKHGXOHGRQWKHQRGH6XSSRVHWKHUHDUH-MREVLQWKHV\VWHP6SHFLILFDOO\HDFK MREMɽ-FRQVLVWVRIWZRW\SHVRIWDVNVPDSWDVNV0DQGUHGXFHWDVN5/HWʉW ɽ^05`GHQRWHWKHW\SHRIDWDVNW *LYHQDSKDVHLRIDWDVNWWKDWFDQEHVFKHGXOHGRQDPDFKLQHQZHGHILQHWKHXWLOLW\IXQFWLRQRIDVVLJQLQJDSKDVH WRPDFKLQHQDV 8LQ 8IDLUQHVVLQ Į8SHUILQ :KHUH8IDLUQHVVDQG8SHUIUHSUHVHQWWKHXWLOLWLHVIRULPSURYLQJIDLUQHVVDQGMRESHUIRUPDQFHUHVSHFWLYHO\DQGĮLVDQ DGMXVWDEOHZHLJKWIDFWRU C. System Architecture 7RSHUIRUPSKDVHOHYHOVFKHGXOLQJWKHVFKHGXOHUUHTXLUHVSKDVHOHYHOUHVRXUFHLQIRUPDWLRQIRUHDFKMRE7KH SUREOHPRIMRESURILOLQJLVPHWE\WKHMRESURILOHUNQRZQDV6WDUILVK,WFDQSURYLGHDFFXUDWHUHVRXUFHLQIRUPDWLRQ WKDWFDQEHXVHGE\WKHVFKHGXOHU6WDUILVKLVDVHOIWXQLQJV\VWHPIRUELJGDWDDQDO\WLFVZLWKD&RVWEDVHG2SWLPL]HU IRUWKH0DS5HGXFHSURJUDPV,WLQFOXGHVD3URILOHUWRFROOHFWGHWDLOHGVWDWLVWLFDOLQIRUPDWLRQIURPWKH0DS5HGXFH SURJUDPVDQGD:KDWLI(QJLQHIRUILQHJUDLQHGFRVWHVWLPDWLRQ>@7KH0DS5HGXFH6WDUILVKV\VWHPIUDPHZRUNLV VKRZQLQ)LJXUH
)LJXUH+DGRRS6WDUILVK6\VWHP)UDPHZRUN
9RO,VVXH1RYHPEHU
,661;
,QWHUQDWLRQDO-RXUQDORI/DWHVW7UHQGVLQ(QJLQHHULQJDQG7HFKQRORJ\,-/7(7
D. Job Profiling 7KLV VHFWLRQ GHVFULEHV KRZ VWDUILVK SHUIRUPV WKH WDVN RI MRE SURILOLQJ DV JLYHQ LQ >@ $ 0DS5HGXFH MRE FDQ EH UHSUHVHQWHG DV M WKDW UXQV SURJUDP S RQ LQSXW GDWD G DQG FOXVWHU UHVRXUFHV U XVLQJ FRQILJXUDWLRQ SDUDPHWHUVHWWLQJVF3HUIRUPDQFHRIWKHMREMFDQEHUHSUHVHQWHGDV perf )SGUF ) UHSUHVHQWV WKH FRVW PRGHO 3HUIRUPDQFH RI WKH SURJUDP FDQ EH RSWLPL]HG E\ ILQGLQJ WKH FRQILJXUDWLRQ SDUDPHWHUVHWWLQJV&RSW$VHWRIDQDO\WLFDODQGVLPXODWLRQPRGHOVLVXVHGE\WKH:KDWLI(QJLQHWRLPSOHPHQWWKH FRVW IXQFWLRQ ) 7KH 3URILOHU JDWKHUV WKH QHFHVVDU\ LQIRUPDWLRQ IRU SHUIRUPLQJ FRVWEDVHG RSWLPL]DWLRQ ,W LV UHVSRQVLEOHIRUFROOHFWLQJMRESURILOHVVWDWLVWLFDOVXPPDULHVRI0DS5HGXFHMREH[HFXWLRQ$MRESURILOHLVDYHFWRURI ILHOGV ZKHUH HDFK ILHOG FDSWXUHV VRPH XQLTXH DVSHFW RI GDWDIORZ RU FRVW HVWLPDWHV GXULQJ MRE H[HFXWLRQ 7KH LQIRUPDWLRQ LQFOXGHG LQ D MRE SURILOH LV DW WKH ILQH JUDQXODULW\ RI SKDVHV ZLWKLQ WKH PDS DQG UHGXFH WDVNV RI D 0DS5HGXFHMREH[HFXWLRQ7KLVIHDWXUHLVHVVHQWLDOIRUGHWHUPLQLQJWKHDFFXUDF\RIGHFLVLRQVPDGHE\WKH:KDWLI (QJLQH DQG WKH &RVWEDVHG 2SWLPL]HU 7KH MRE SURILOHV DOVR KHOS LQ XQGHUVWDQGLQJ WKH MRE EHKDYLRU DV ZHOO DV LQ GLDJQRVLQJ ERWWOHQHFNV GXULQJ MRE H[HFXWLRQ 7KH 3URILOHU XVHV G\QDPLF LQVWUXPHQWDWLRQ WR FROOHFW UXQWLPH PRQLWRULQJ LQIRUPDWLRQ IURP XQPRGLILHG 0DS5HGXFH SURJUDPV UXQQLQJ RQ +DGRRS :KHQ +DGRRS UXQV D 0DS5HGXFH MRE WKH 3URILOHU FROOHFWV UDZ PRQLWRULQJ GDWD 7KH UDZ GDWD ZLOO XQGHUJR D VHULHV RI SRVWSURFHVVLQJ VWHSVLQRUGHUWRFRQVWUXFWDFRQFLVHMRESURILOH)LUVWWKHUDZGDWDFROOHFWHGIRUHDFKSURILOHGPDSRUUHGXFHWDVNLV SURFHVVHGWRJLYHILHOGVLQDWDVNSURILOH7KHPDSUHGXFH WDVNSURILOHVDUHIXUWKHUSURFHVVHGWRJLYHUHSUHVHQWDWLYH PDSUHGXFH WDVNSURILOHV7KHWDVNSURILOHVDOVRFRQWDLQDOOLQGLYLGXDONH\YDOXHIORZVZKLFKDUHXVHGWRFRPSXWH WKHNH\YDOXHGLVWULEXWLRQVDFURVVDOOPDSDQGUHGXFHWDVNV 7KHLQSXWVJLYHQWRWKH:KDWLI(QJLQHWRSUHGLFWWKHSHUIRUPDQFHRID0DS5HGXFHMREMDUH -RESURILOHJHQHUDWHGIRUMREME\WKH3URILOHU 1HZFRQILJXUDWLRQVHWWLQJVFWRUXQMREM 6L]HOD\RXWDQGFRPSUHVVLRQLQIRUPDWLRQRIWKHLQSXWGDWDVHWGRQZKLFKMREMZLOOEHUXQ &OXVWHUVHWXSDQGUHVRXUFHDOORFDWLRQUWKDWZLOOEHXVHGWRUXQMREM7KLVLQIRUPDWLRQLQFOXGHVWKHQXPEHU RIQRGHVLQWKHFOXVWHUDQGQHWZRUNWRSRORJ\RIWKHFOXVWHUWKHQXPEHURI PDSDQGUHGXFHWDVNVORWVSHU QRGHDQGWKHPHPRU\DYDLODEOHIRUHDFKWDVNH[HFXWLRQ 7KHVH LQSXWV DORQJ ZLWK D GHWDLOHG VHW RI DQDO\WLFDO DQG VLPXODWLRQ PRGHOV GHYHORSHG DUH XVHG WR JHQHUDWH D YLUWXDO MRE SURILOH 7KLV YLUWXDO MRE SURILOH LV IRU WKH K\SRWKHWLFDO MRE M¶ WKDW UHSUHVHQWV M UXQQLQJ ZLWK WKH QHZ FRQILJXUDWLRQ VHWWLQJV 7KH YLUWXDO MRE SURILOH LV XVHG E\ D 7DVN 6FKHGXOHU 6LPXODWRU DORQJ ZLWK WKH PRGHOV DQG LQIRUPDWLRQRQWKHFOXVWHUUHVRXUFHVUWRVLPXODWHWKHVFKHGXOLQJDQGH[HFXWLRQRIWKH PDSDQGUHGXFHWDVNVRIM¶ IURP ZKLFK WKH GHVLUHG SHUIRUPDQFH PHWULFV DUH HVWLPDWHG 7KH &RVWEDVHG RSWLPL]HU PXVW ILQG WKH VHWWLQJ RI FRQILJXUDWLRQSDUDPHWHUV&RSW DUJPLQ)SGUF IRUWKHFRVWPRGHO)UHSUHVHQWHGE\WKH:KDWLI(QJLQHRYHUWKH VSDFH 6 RI FRQILJXUDWLRQ SDUDPHWHU VHWWLQJV IRU D JLYHQ 0DS5HGXFH SURJUDP S 7KH 2SWLPL]HU DGGUHVVHV WKLV SUREOHP E\ PDNLQJ ZKDWLI FDOOV ZLWK VHWWLQJV F RI WKH FRQILJXUDWLRQ SDUDPHWHUV VHOHFWHG WKURXJK DQ HQXPHUDWLRQ DQGVHDUFKRYHU6,QRUGHUWRSURYLGHERWKHIILFLHQF\DQGHIIHFWLYHQHVVWKH2SWLPL]HUPXVWPLQLPL]HWKHQXPEHURI ZKDWLI FDOOV ZKLOH ILQGLQJ QHDURSWLPDO FRQILJXUDWLRQV 6HYHUDO RWKHU PHWKRGV IRU LPSURYLQJ UHVRXUFH XWLOL]DWLRQ ZHUHLQWURGXFHGDVUHVRXUFHVWHDOLQJ>@7KHFRQFHSWRIMRESURILOLQJWKXVFDQEHXVHGIRUFRVWEDVHGRSWLPL]DWLRQ DQGEHWWHUUHVRXUFHXWLOL]DWLRQ ,9&21&/86,21 0DS5HGXFHLVDSRSXODUSURJUDPPLQJPRGHOIRUGDWDLQWHQVLYHFRPSXWLQJ,QVWHDGRIGHVLJQLQJUHVRXUFHHIILFLHQW 0DS5HGXFHVFKHGXOHUVH[LVWLQJZRUNPDLQO\IRFXVHVRQGHVLJQLQJWDVNOHYHOVFKHGXOHUV7RDGGUHVVWKLVOLPLWDWLRQ DSKDVHDQGUHVRXUFHDZDUHVFKHGXOHUZDVLQWURGXFHG,WLVDILQHJUDLQHGUHVRXUFHDZDUHVFKHGXOHUWKDWFRRUGLQDWHV WDVN H[HFXWLRQ DW WKH OHYHO RI SKDVHV 7KH SKDVHOHYHO VFKHGXOLQJ DOJRULWKP LPSURYHV MRE H[HFXWLRQ ZLWKRXW LQWURGXFLQJDQ\VWUDJJOHUV7KHSKDVHEDVHGUHVRXUFHDZDUHVFKHGXOHURIIHUVKLJKUHVRXUFHXWLOL]DWLRQDQGSURYLGHV LPSURYHPHQWLQMREUXQQLQJWLPH7KHSUREOHPRIMRESURILOLQJLVDOVRPHW8VLQJSURILOHVWRLPSURYH0DS5HGXFH MRE SHUIRUPDQFH LV DQ LQWHUHVWLQJ WKRXJKW 6WDUILVK D MRE SURILOHU WKDW FROOHFWV ILQHJUDLQHG MRE FRQILJXUDWLRQ SDUDPHWHUVFDQEHEXLOWLQWRWKHV\VWHPWRVWXG\WKHMRESURILOLQJ6WDUILVKFDQSURYLGHDFFXUDWHUHVRXUFHLQIRUPDWLRQ WKDW FDQ EH XVHG E\ WKH VFKHGXOHU $V D IXWXUH ZRUN LW ZLOO EH LQWHUHVWLQJ WR VWXG\ WKH SURILOLQJ DQG VFKHGXOLQJ SUREOHPIRUPDFKLQHVZLWKKHWHURJHQHRXVSHUIRUPDQFHFKDUDFWHULVWLFV
9RO,VVXH1RYHPEHU
,661;
,QWHUQDWLRQDO-RXUQDORI/DWHVW7UHQGVLQ(QJLQHHULQJDQG7HFKQRORJ\,-/7(7
5()(5(1&(6 >@ 6KLQL 5HQMLWK $QMDOL & ³)LWQHVV )XQFWLRQ LQ *HQHWLF $OJRULWKP %DVHG ,QIRUPDWLRQ )LOWHULQJ $ 6XUYH\´ ,QWHUQDWLRQDO -RXUDQDO RI &RPSXWHU6FLHQFHDQG0RELOH&RPSXWLQJ,&0,&'HFHPEHUSS >@ $83DWLO7,%DJEDQ$33DQGH´5HFHQW-RE6FKHGXOLQJ$OJRULWKPVLQ+DGRRS&OXVWHU(QYLURQPHQWV$6XUYH\´,QWHUQDWLRQDO-RXUQDO RI$GYDQFHG5HVHDUFKLQ&RPSXWHUDQG&RPPXQLFDWLRQ(QJLQHHULQJ9RO,VVXH)HEUXDU\ >@ +DGRRS)DLU6FKHGXOHU>2QOLQH@$YDLODEOHKWWSKDGRRSDSDFKHRUJGRFVUIDLUBVFKHGXOHUKWPO >@ +DGRRS&DSDFLW\6FKHGXOHU>2QOLQH@$YDLODEOHKWWSKDGRRSDSDFKHRUJGRFVVWDEOHFDSDFLW\BVFKHGXOHUKWPO >@ %7KLUXPDOD 5DR 'U /66 5HGG\ ³6XUYH\ RQ ,PSURYHG 6FKHGXOLQJ LQ +DGRRS 0DS5HGXFH LQ &ORXG (QYLURQPHQWV´ ,QWHUQDWLRQDO -RXUQDORI&RPSXWHU$SSOLFDWLRQV 9ROXPH1R1RYHPEHU >@ -RUGD3ROR&ODULV&DVWLOOR'DYLG&DUUHUD@ $ *KRGVL 0 =DKDULD % +LQGPDQ $ .RQZLQVNL 6 6KHQNHU , 6WRLFD ³'RPLQDQW 5HVRXUFH )DLUQHVV )DLU $OORFDWLRQ RI 0XOWLSOH 5HVRXUFH7\SHV´LQIEEE Proc. USENIX Symp. Netw Syst. Des Implementation, SS. >@ 4L =KDQJ0RKDPHG )DWHQ =KDQL @ =KHQKXD*XR)R[*0R=KRX