A New Pruning Algorithm for Neural Network Dimension ... - IEEE Xplore

 

$1HZ3UXQLQJ$OJRULWKPIRU1HXUDO1HWZRUN'LPHQVLRQ$QDO\VLV 'HYLQ6DER;LDR+XD@>@>@>@>@>@  ,QWKLVSDSHUDQHZSUXQLQJDOJRULWKPLVLQYHVWLJDWHGDQG FRPSDUHG ZLWK WKUHH H[LVWLQJ RQHV LQFOXGLQJ WKH ORFDO VHQVLWLYLW\ DQDO\VLV PHWKRG >@ WKH ORFDO YDULDQFH VHQVLWLYLW\ DQDO\VLV >@ DQG WKH FURVV YDOLGDWLRQ SUXQLQJ DOJRULWKP>@7KLVQHZDOJRULWKPFRPELQHVWKHDGYDQWDJHV RI WKH DERYH WKUHH PHWKRGV UHHYDOXDWHV WKH QHWZRUN SHUIRUPDQFH GXULQJ SUXQLQJ SURFHVV DQG LWHUDWLYHO\ SUXQHV WKH QHXUDO QHWZRUN RQ D UHGXFHG VHW RI FRQQHFWLRQV LI D SUXQLQJ HUURU RFFXUV $ YDULHW\ RI GDWD VHWV DQG YDULRXV QHWZRUNFRQILJXUDWLRQVDUHVWXGLHGLQFRPSXWHUVLPXODWLRQV 7KH UHVXOWV VKRZ WKDW WKLV QHZ DOJRULWKP FDQ VLJQLILFDQWO\ UHGXFHV WKH QHXUDO QHWZRUN VL]H ZKLOH VWLOO PDLQWDLQLQJ VDWLVIDFWRU\JHQHUDOL]DWLRQDFFXUDF\RIWKHQHWZRUN ,, 7+(1(85$/1(7:25.3581,1*$/*25,7+0678',(6 $ W\SLFDO QHXUDO QHWZRUN FRQWDLQV DQ LQSXW OD\HU DQ RXWSXWOD\HUDQGRQHRUPRUHKLGGHQOD\HUV7KHQXPEHURI RXWSXWV DQG LQSXWV DUH XVXDOO\ IL[HG ZKLOH WKH QXPEHU RI KLGGHQOD\HUVDQGQXPEHURIKLGGHQQHXURQVLQHDFKKLGGHQ OD\HUFDQEHYDULHG,QWKLVUHVHDUFKZHIRFXVRQWKHVWXGLHV RI SUXQLQJ DOJRULWKPV IRU PXOWLOD\HU IHHGIRUZDUG QHXUDO QHWZRUNV 7KHVLPSOHVWZD\WR ILQGWKHRSWLPXPQHWZRUNVL]HLVWR XVH D EUXWH IRUFH DSSURDFK WKDW SURGXFHV DOO WKH FRPELQDWLRQV RI QHWZRUNV ZLWKLQ D GHVLUDEOH UDQJH WUDLQV WKHPDQGWKHQFKRRVHVWKHEHVWRQH7KLVSURFHVVLVXVXDOO\ QRWDQHIILFLHQWZD\WRVROYHWKHSUREOHP$QRWKHUDSSURDFK WKH ZHLJKW GHFD\ PHWKRG RU WKH SHQDOW\ PHWKRG  LV EDVHG RQ WKH DVVXPSWLRQ WKDW VPDOOHU ZHLJKWV LQ D QHWZRUN KDYH UHODWLYHO\ VPDOO HIIHFW RQ WKH RXWSXW RI D QRGH HVSHFLDOO\ ZKHQ VXUURXQGHG E\ VLJQLILFDQWO\ ODUJHU ZHLJKWV 7KLV PHWKRG DGGV D SHQDOW\ WHUP WR WKH REMHFWLYH IXQFWLRQ WR EH PLQLPL]HG VR WKDW WKHVH VPDOOHU ZHLJKWV FDQ HYHQWXDOO\ EH IRUFHG WR ]HUR +RZHYHU WKLV DSSURDFK PD\ HOLPLQDWH ZHLJKWVWKDWDUHDFWXDOO\FUXFLDOWRWKHRYHUDOODUFKLWHFWXUHRI WKH QHWZRUN DQG WKXV SURGXFH D QHWZRUN ZLWK SRRU SHUIRUPDQFH $OVR WKH DGGHG SHQDOW\ WHUP PD\ FUHDWH DGGLWLRQDOORFDOPLQLPDRQWKHHUURUVXUIDFHGXULQJWUDLQLQJ +X\QK DQG 6HWLRQR >@ LQWURGXFHG WKH FRQFHSW RI FURVV YDOLGDWLRQ7KHZKROHGDWDVHWLVGLYLGHGLQWRWZRSDUWVLH WKHWUDLQLQJVHW7DQGWKHFURVVYDOLGDWLRQVHW&7KHSUXQLQJ

3313 c 978-1-4244-1821-3/08/$25.002008 IEEE

  FULWHULRQ LV VWLOO EDVHG RQ WKH PDJQLWXGH RI HDFK ZHLJKW KRZHYHU D YDOLGDWLRQ VWHS LV DGGHG WR WHVW WKH SUXQHG QHWZRUN $W HYHU\ SUXQLQJ VWHS WKH SHUIRUPDQFH RI WKH QHWZRUNZLWKUHGXFHGVL]HLVFRPSDUHGZLWKWKHSHUIRUPDQFH RI WKH QHWZRUN EHIRUH FXUUHQW SUXQLQJ SKDVH /HW WKH SHUIRUPDQFHFULWHULRQREMHFWLYHIXQFWLRQ RQVHW7DQGVHW& EH - 75 DQG - &9 UHVSHFWLYHO\ZKHUH -‡ FDQEHWKHURRW PHDQVTXDUH HUURU 506  IRU IXQFWLRQ DSSUR[LPDWLRQ RU LGHQWLILFDWLRQ SUREOHP RU WKH SHUFHQWDJH RI PLVFODVVLILHG SDWWHUQVIRUFODVVLILFDWLRQSUREOHP $IWHUSUXQLQJDVPDOOHU QHXUDOQHWZRUNLVREWDLQHGDQGWKHHUURURQVHW 7DQGVHW& ′ DQG - &9 ′ UHVSHFWLYHO\,I EH - 75

′ + - &9 ′ <  - 75 + - &9              - 75 LHWKHSUXQHGQHWZRUNRXWSHUIRUPVWKHXQSUXQHGRQHWKHQ WKHSUXQHGQHWZRUNLVDFFHSWHGDQGWKHSUXQLQJSURFHVVFDQ EH FRQWLQXHG2WKHUZLVHWKHQHWZRUNLVUHVWRUHGWRWKHVL]H EHIRUH WKH FXUUHQW SUXQLQJ VWHS 2EYLRXVO\ WKH XVH RI DQ DGGLWLRQDO FURVV YDOLGDWLRQ VHW DW HDFK SKDVH RI WKH SUXQLQJ WDNHV LQWR DFFRXQW WKDW SUXQLQJLV PHDQWWR QRW RQO\ UHGXFH WKH VL]H RI D QHWZRUN EXW DOVR LPSURYH WKH QHWZRUN JHQHUDOL]DWLRQFDSDFLW\ 5DWKHU WKDQ IRFXVLQJ RQ WKHPDJQLWXGH RI WKH ZHLJKWV LQ WKHQHWZRUNWKHVHQVLWLYLW\ EDVHGDSSURDFKDWWHPSWVWRILQG WKH FRQWULEXWLRQ RI HDFK ZHLJKW LQ WKH QHWZRUN DQG WKHQ SUXQHVWKHZHLJKWVWKDWKDYHWKHOHDVWHIIHFWRQWKHREMHFWLYH IXQFWLRQ 0R]HU DQG 6PROHQVN\ >@ VXJJHVWHG WKDW WKH VHQVLWLYLW\ RI HDFK ZHLJKW FDQ EH IRXQG E\ PHDVXULQJ WKH GLIIHUHQFH RQ WKH SHUIRUPDQFH RI WKH QHWZRUN ZLWKZLWKRXW WKDWZHLJKWLH

V M L  - Z ML   - Z ML  Z MI L  -ZLWKRXW Z ML -ZLWK Z ML           ZKHUH V M L LVWKHVHQVLWLYLW\ZLWKUHVSHFWWRWKHUHPRYDORI FRQQHFWLRQ Z ML  Z ML  LV WKH ZHLJKW RI WKH QHXUDO QHWZRUN IURPQRGHLWRQRGHM

Z MI L

LVWKHILQDOYDOXHRIZHLJKW Z ML 

ZKHQWUDLQLQJLVILQLVKHGDQG-‡ LVWKHREMHFWLYHIXQFWLRQ  &DOFXODWLQJ(T GLUHFWO\PD\EHYHU\WLPHFRQVXPLQJ .DUQLQ>@IRXQGDQHIIHFWLYHZD\WRDSSUR[LPDWHLWIRUWKH EDFNSURSDJDWLRQDOJRULWKP  V ML =

7 −

¦ W =

> ΔZ ML W @



Z MI L

η Z MI L − Z LML

        

ZKHUH 7 LV WKH WRWDO QXPEHU RI LWHUDWLRQV WUDLQLQJ HSRFKV  QHHGHG WR PLQLPL]H WKH REMHFWLYH IXQFWLRQ - η  LV WKH OHDUQLQJUDWHDQG ΔZ ML LVWKHFKDQJHRQZHLJKW Z ML LQRQH

IXQFWLRQ DV GHVLUHG EXW LI WKH WKUHVKROG LV WRR VPDOO QR ZHLJKWV ZLOO EH SUXQHG DW DOO $OVR LQ WKLV PHWKRG DOO WKH VHQVLWLYLWLHV LQ WKH QHWZRUN DUH FRPSDUHG ZLWK WKH VDPH WKUHVKROGLHWKH\DUHWUHDWHGHTXDOO\IRUSUXQLQJ 3RQQDSHOOL HW DO >@ VXJJHVWHG WKDW WKH VHQVLWLYLWLHV RI ZHLJKWVVKRXOGRQO\EHFRPSDUHGZLWKWKRVHUHODWHGZLWKWKH VDPH QRGH LQ WKH VDPH OD\HU 7KXV WKH FRQFHSW RI ORFDO UHODWLYHVHQVLWLYLW\LQGH[/56, LVGHILQHGDVWKHUDWLRRIWKH VHQVLWLYLW\ RI D SDUWLFXODU ZHLJKW DQG WKH VXP RI DOO WKH VHQVLWLYLWLHV RI WKH ZHLJKWV WKDW DUH FRQQHFWHG WR WKH VDPH QRGHIURPWKHSUHYLRXVOD\HU

/56, M L =

SUHGHWHUPLQHG WKUHVKROG WR GHWHUPLQH ZKHWKHU WKH ZHLJKW VKRXOG EH SUXQHG RU QRW 1RWH WKDW WKLV DOJRULWKP UHOLHV KHDYLO\ RQ WKH VHOHFWLRQ RI D WKUHVKROG ZKLFK PXVW EH GHWHUPLQHG EHIRUHKDQG ZKLFKPD\ GLIIHU EHWZHHQ GDWD VHWV DQG DSSOLFDWLRQV ,I WKH WKUHVKROG LV WRR KLJK WRR PDQ\ ZHLJKWV ZLOO EH SUXQHG DQG WKH SUXQHG QHWZRUN PD\ QRW

3314

               

0

¦

V M P

P =

ZKHUH 0 LVWKH WRWDOQXPEHU RI FRQQHFWLRQV WRQRGH M IURP WKHSUHYLRXVOD\HU)RUHDFKQRGHDQ\ZHLJKWWKDWKDVDORFDO VHQVLWLYLW\OHVVWKDQDWKUHVKROGZLOOEHSUXQHG  /56, M L ≤ β                    (YHQ WKRXJK WKH FKRLFH RI WKH WKUHVKROG LH β  VWLOO GHSHQGVRQWKHUXOHRIWKXPELWLVQRZDSHUFHQWDJHZKLFK LV UHODWLYHO\ HDVLHU WR EH FKRVHQ 1RWH WKDW WKLV DOJRULWKP RQO\ FRQVLGHUV ZHLJKW UHPRYDO QRGH SUXQLQJ LV QRW LQFOXGHG7KHRUHWLFDOO\LIDOOWKHZHLJKWVWKDWDUHFRQQHFWHG WR D VLQJOH QRGH DUH SUXQHG WKHQ WKLV QRGH FDQ DOVR EH HOLPLQDWHG +RZHYHU WKLV PD\ WDNH VHYHUDO URXQGV RI SUXQLQJ DQG WUDLQLQJ VR LWPD\ QRW EH D IHDVLEOH VROXWLRQ LQ SUDFWLFH  (QJHOEUHFKW >@ SURSRVHG D PRGLILHG DSSURDFK WR VHQVLWLYLW\ DQDO\VLV ,QVWHDG RI XVLQJ WKH YDOXHV RI VHQVLWLYLWLHV GLUHFWO\ (QJHOEUHFKW IRXQG WKH DYHUDJH VHQVLWLYLW\ RI D QHWZRUN SDUDPHWHU HJ ZHLJKW RU QRGH  RYHU DOO WKH SDWWHUQV DQG WKHQ GHYHORSHG D QHZ PHDVXUH FDOOHGYDULDQFHQXOOLW\7KDWLVLIWKHYDULDQFHRIVHQVLWLYLW\ RI D QHWZRUN SDUDPHWHU RYHU DOO WKH SDWWHUQV GHQRWHG E\

σ θN  IRU SDUDPHWHU θ N  LV FORVH WR ]HUR DQG WKH DYHUDJH VHQVLWLYLW\ DOVR RYHU DOO WKH SDWWHUQV  LV VPDOO WKHQ ZH FRQFOXGH WKDW WKLV SDUDPHWHU KDV OLWWOH RU QR HIIHFW RQ WKH RXWSXWRIWKHQHXUDOQHWZRUNRYHUDOOSDWWHUQVDQGWKHUHIRUH FDQEHHOLPLQDWHG7KHYDULDQFHRIVHQVLWLYLW\LVGHILQHGDV 3

 σ θN =

¦  Vθ

 S

S =

N

− aVθ N 

3 −

             

ZKHUH3LVWKHWRWDOQXPEHURISDWWHUQVXQGHUFRQVLGHUDWLRQ Vθ L LVWKHDYHUDJHVHQVLWLYLW\RYHUDOOWKHSDWWHUQV DQG a 3

WUDLQLQJ LWHUDWLRQ 7KH DEVROXWH YDOXH RI WKH HVWLPDWHG VHQVLWLYLW\ IRU HDFK ZHLJKW V ML  LV WKHQ FRPSDUHG ZLWK D

V M L

a Vθ N =

¦ Vθ

 S

S =

N

3

                 

7KHSDUDPHWHUYDULDQFHQXOOLW\391 IRUHDFKSDUDPHWHULV WKHQGHILQHGDV  γ θN =

 3 −  σ θN

σ 

               

2008 International Joint Conference on Neural Networks (IJCNN 2008)

  ZKHUH σ  LVDVPDOOFRQVWDQWYDOXHUHODWHGZLWKK\SRWKHVLV 



WHVW +  σ θ N < σ  >@ 7KLV DOJRULWKP DOORZV IRU SUXQLQJ RI ERWK QRGHV DQG ZHLJKWVZLWKHDFKSDUDPHWHUKDYLQJDVHSDUDWHIRUPXODIRU WKH VHQVLWLYLW\ FDOFXODWLRQ 7KH H[WHQVLRQ RI D VHQVLWLYLW\ PHDVXUHPHQW WR QRGHV QRW MXVW ZHLJKWV  DOORZV IRU WKH SRVVLELOLW\ RI ILQGLQJ D VPDOOHUQHWZRUNDQGDOVR GHFUHDVHV WKHQXPEHURIWLPHVWRUHWUDLQWKHQHWZRUNEHIRUHREWDLQLQJ LWV ILQDO VL]H 1RWH WKDW LQ WKLV DOJRULWKP WKH SUXQLQJ GHFLVLRQLVEDVHGRQWKHK\SRWKHVLVWHVW+WKXVFKRRVLQJWKH DSSURSULDWH YDOXH RI σ   LV FUXFLDO WR WKH VXFFHVV RI WKLV DOJRULWKP $V ZH GLVFXVVHG HDUOLHU UHO\LQJ RQ RQH VLQJOH YDOXH RI

σ  IRUWKHHQWLUHQHWZRUNFDQOHDGWRSUREOHPV)QDLHFKHW DO >@ VXJJHVWHG WKDW SDUDPHWHUV ZLWKLQ WKH VDPH OD\HU VKRXOG EH FRQVLGHUHG ³ORFDOO\´ UDWKHU WKDQ ³JOREDOO\´ DQG GHILQHG D QHZ SUXQLQJ LQGH[ FDOOHG WKH ORFDO SDUDPHWHU YDULDQFHQXOOLW\/391 7KH391IRUDOOSDUDPHWHUVLQWKH VDPH OD\HU DUH VXPPHG XS WKHQ WKH /391 IRU HDFK SDUDPHWHUZKLFKUHSUHVHQWVWKHUHODWLYHLPSRUWDQFHRI391 RI D SDUDPHWHU LQ WKH OD\HU  FDQ EH REWDLQHG DQG XVHG IRU SUXQLQJ

/γ θ > O @ = N

γ θ>O @ N

.

¦γ θ > O @ N =

               

N

ZKHUH /γ θ > O @  LV WKH /391 IRU OD\HU O DQG . LV WKH WRWDO N

QXPEHURISDUDPHWHUVLQOD\HUO $V D VXPPDU\ DOO WKH DERYH DOJRULWKPV KDYH WKHLU RZQ DGYDQWDJHV DQG OLPLWDWLRQV ,Q WKH QH[W VHFWLRQ D QHZ SUXQLQJ DOJRULWKP ZKLFK FRPELQHV WKH DGYDQWDJHV RI WKH SUHYLRXVO\ PHQWLRQHG DOJRULWKPV DQG SUXQHV WKH QHWZRUN LWHUDWLYHO\RQDUHGXFHGVHWLIQHFHVVDU\ ZLOOEHSURSRVHG ,,, 7+(352326('1(:3581,1*$/*25,7+0 ,Q WKLV VHFWLRQ D QHZ SUXQLQJ DOJRULWKP FDOOHG K\EULG VHQVLWLYLW\DQDO\VLV ZLWKUHSUXQLQJ+6$5 LVLQYHVWLJDWHG %RWK ZHLJKW SUXQLQJDQGQRGH SUXQLQJ DUH FRQVLGHUHG DQG PRVW RI WKH GLVDGYDQWDJHV RI WKH H[LVWLQJ DOJRULWKPV GLVFXVVHG LQ VHFWLRQ WZR FDQ EH HOLPLQDWHG RU UHGXFHG 3UXQLQJLVEDVHGRQVHQVLWLYLW\FDOFXODWLRQDQGORFDOYDULDQFH QXOOLW\ DQG WKH SHUIRUPDQFH RI WKH QHXUDO QHWZRUN LV UHYDOXHG XVLQJ FURVVYDOLGDWLRQ DW HYHU\ SUXQLQJ VWHS 7KH DOJRULWKPLWHUDWLYHO\SUXQHVWKHQHXUDOQHWZRUNRQDUHGXFHG VHWRIFRQQHFWLRQVLIDSUXQLQJHUURURFFXUV7KHPDLQVWHSV RIWKHDOJRULWKPDUHVXPPDUL]HGDVIROORZV  7UDLQ WKH RYHUVL]HG QHXUDO QHWZRUN XVLQJ EDFN SURSDJDWLRQ DQG HYDOXDWH WUDLQLQJ SHUIRUPDQFH XVLQJ FURVV YDOLGDWLRQ (VWLPDWHWKHVHQVLWLYLW\XVLQJ(T  &DOFXODWH WKH DYHUDJH VHQVLWLYLW\ RI HDFK SDUDPHWHU LQ OD\HUOXVLQJ(T  &DOFXODWH WKH YDULDQFH RI WKH VHQVLWLYLWLHV RI WKH SDUDPHWHUV LQ OD\HU O XVLQJ (T   DQG WKH DYHUDJH VHQVLWLYLW\YDOXHVFDOFXODWHGIURPVWHS

)LQGWKH/391RIHDFKSDUDPHWHUXVLQJ(T DQGWKH DYHUDJHVHQVLWLYLW\YDOXHVFDOFXODWHGLQVWHS 6RUW WKH /931 YDOXHV IURP VWHS  LQ DQ LQFUHDVLQJ RUGHU VXFK WKDW WKH ORZHVW YDOXH LV LQGH[HG DV  DQG WKH KLJKHVWYDOXHLVLQGH[HGDV1ZKHUH1LVWKHWRWDOQXPEHURI SDUDPHWHUV LQ HDFK VWHS  ,Q FDVH RI UHSUXQLQJ WKH SDUDPHWHUVZLWKWKHKLJKHVW/931YDOXHVDUHUHPRYHGIURP WKHUHSUXQLQJOLVWILUVWVHHVWHS  3UXQH DQ\ SDUDPHWHU ZLWK D /931 YDOXH IURP VWHS  ZKLFKLVOHVVWKDQFHUWDLQWKUHVKROG 7UDLQ WKH SUXQHG QHWZRUN DQG UHJDWKHU WKH VHQVLWLYLW\ LQIRUPDWLRQ (YDOXDWH WKH SHUIRUPDQFH RI WKH QHZO\ SUXQHG DQG WUDLQHGQHWZRUNXVLQJWKHIROORZLQJ ′ + - &9 ′ <  ζ - 75 + - &9         ρ  ζ - 75 ZKHUH ρ  LV D FRQVWDQW WKDW JLYHV SULRULW\ WR SUXQHG QHWZRUNV ζ   ζ     LV DQRWKHU FRQVWDQW WKDW HQFRXUDJHV JHQHUDOL]DWLRQFDSDFLW\E\IDYRULQJWKHFURVVYDOLGDWLRQHUURU RYHUWKHWUDLQLQJHUURU,IWKHQHZQHWZRUNIDLOVWRVKRZDQ LPSURYHPHQW RYHU WKH ROGQHWZRUNUHVWRUH WKH ROGQHWZRUN WR WKHODVW ZRUNLQJQHWZRUN FRQILJXUDWLRQ DQG JR WR WKHUH SUXQLQJ SURFHVV LQ D  RWKHUZLVH FRQWLQXH WR WKH QH[W SUXQLQJPRGHHLWKHUSUXQLQJZHLJKWRUQRGH  D  &KRRVHWKHUHGXFHGSDUDPHWHUOLVWWRSUXQH QUS  W +  = QUS  W   − λ             ZKHUH QUS  LV WKHQXPEHU RI ZHLJKWV WR UHSUXQH DQG λ  LV WKHSHUFHQWDJHRIZHLJKWVWRGHGXFWDWHDFKUHSUXQLQJVWHS )RUQRGHZHKDYH  QUS  W +  = QUS  W −                LHHDFKVXFFHVVLYHUHSUXQLQJVWHSRQO\VXEWUDFWVRQHQRGH IURP WKH SUHYLRXV DWWHPSW ,I WKHUH DUH QR SRVVLEOH SDUDPHWHUV WR UHSUXQH JR WR WKH QH[W OD\HU RWKHUZLVH FRQWLQXHWRE  E  3UXQH WKH QHWZRUN XVLQJ WKH UHGXFHG SUXQLQJ OLVW IURPD  F  %DFNWRVWHS ,9 6,08/$7,215(68/76 ,QWKLVVHFWLRQWKHSURSRVHGDOJRULWKPLVDSSOLHGWRVROYH GLIIHUHQW SDWWHUQ FODVVLILFDWLRQSUREOHPV DQG FRPSDUHG ZLWK WKUHH H[LVWLQJ DOJRULWKPV GHVFULEHG DERYH LH WKH ORFDO VHQVLWLYLW\DQDO\VLVPHWKRG./6$ >@WKHORFDOYDULDQFH VHQVLWLYLW\ DQDO\VLV /96$  >@ DQG WKH FURVV YDOLGDWLRQ SUXQLQJ DOJRULWKP &93  >@  7KH GDWDVHWV XVHG LQ VLPXODWLRQ FDQ EH IRXQG LQ >@ 7KH\ DUH UHDOZRUOG FODVVLILFDWLRQ SUREOHPV UHODWHG ZLWK SV\FKRORJ\ HJ WKH EDODQFH VFDOH GDWD  DQG PHGLFDO DSSOLFDWLRQV HJ WKH OLYHU GLVRUGHUDQGGLDEHWHVGDWDVHWV 7KH\DUHDYDLODEOHIURPWKH PDFKLQHOHDUQLQJUHSRVLWRU\8QLYHUVLW\RI&DOLIRUQLD,UYLQH ,Q WKH IROORZLQJ VLPXODWLRQ H[DPSOHV GLIIHUHQW LQLWLDO QHWZRUN FRQILJXUDWLRQV DUH FRQVLGHUHG WR IXOO\ WHVW WKH DELOLWLHV RI HDFK SUXQLQJ DOJRULWKP RQ D YDULHW\ RI KLGGHQ QRGH DQG KLGGHQ OD\HU VHWXSV &RQILJXUDWLRQ  KDV RQH KLGGHQ OD\HU ZLWK ILIWHHQKLGGHQQRGHV FRQILJXUDWLRQ KDV

2008 International Joint Conference on Neural Networks (IJCNN 2008)

3315

  WZR KLGGHQ OD\HUV ZLWKWHQQRGHV LQ HDFKKLGGHQ OD\HUDQG FRQILJXUDWLRQ  KDV WKUHH KLGGHQ OD\HUV ZLWK ILYH QRGHV LQ HDFKKLGGHQOD\HU)XUWKHUPRUHHDFKRIWKHWKUHHGDWDVHWVLV GLYLGHGLQWRWHQHTXDOVXEVHWVZKHUHHLJKWRIWKHPDUHXVHG IRU WUDLQLQJ RQH LV XVHG IRU YDOLGDWLRQ DQG WKH UHPDLQLQJ RQH LV IRU WHVWLQJ (DFK RI WKH VXEGDWDVHW LV XVHG IRU WUDLQLQJYDOLGDWLRQDQGWHVWLQJRQDURWDWLRQEDVLVUHVXOWLQJ LQDWRWDORIWHQGLIIHUHQWGDWDFRQILJXUDWLRQV 7KH ZHLJKWV RI DOO WKH QHXUDO QHWZRUNV DUH LQLWLDOL]HG DW UDQGRP EHIRUH WUDLQLQJ 7KH VDPH LQLWLDO FRQGLWLRQV DUH DSSOLHG WR DOO WKH SUXQLQJ DOJRULWKPV LQ HDFK WHVW 7R PLQLPL]H WKH LQIOXHQFHV RI LQLWLDO FRQGLWLRQV WR WHVW UHVXOWV WHQ GLIIHUHQW VHWV RI LQLWLDO FRQGLWLRQV DUH FKRVHQ IRU HDFK QHXUDO QHWZRUN FRQILJXUDWLRQ DQG HDFK VXEGDWDVHW 7KHUHIRUH IRU HDFK QHXUDO QHWZRUN LQLWLDO FRQILJXUDWLRQ D WRWDO RI     VLPXODWLRQ UXQV DUH SHUIRUPHG 7KLV SURFHVVLVUHSHDWHGIRUHDFKRIWKHIRXUSUXQLQJDOJRULWKPV DQGIRUHDFKRIWKHWKUHHGDWDVHWV 7DEOH  EHORZ VKRZV WKH RYHUDOO SUXQLQJ FDSDELOLW\ RI HDFK RI WKH IRXU WHVWHG DOJRULWKPV E\ GLVSOD\LQJ WKH PHDQ DQG VWDQGDUG GHYLDWLRQ SUHVHQWHG DV PHDQ  “ VWG  RI WKH SUXQLQJ SHUFHQWDJH ZLWK UHVSHFW WR WKH RULJLQDO QHWZRUN  )RU H[DPSOH LQ WKH EDODQFH VFDOH WHVW IRU WKH ILUVW QHXUDO QHWZRUNFRQILJXUDWLRQWKH./6$DOJRULWKPFDQSUXQHDERXW  RI WKH WRWDO QHXUDO QHWZRUN ZHLJKWV DQ DYHUDJH IRU UXQVZLWKGLIIHUHQWLQLWLDOFRQGLWLRQVDQGGDWDURWDWLRQV  ZLWK WKH VWDQGDUG GHYLDWLRQ RI  DOVR UHSUHVHQWHG LQ WHUPV RI SHUFHQWDJH  6LPLODUO\ XQGHU WKH VDPH FRQGLWLRQ WKH SURSRVHG +6$5 DOJRULWKP FDQ SUXQH DERXW  RI WKH WRWDO QHXUDO QHWZRUN ZHLJKWV ZLWK D VWDQGDUG GHYLDWLRQ RI  2EYLRXVO\ WKH QHZ DOJRULWKP RXWSHUIRUPV WKH ./6$DOJRULWKP ,Q WDEOHV   DQG  WKH GHWDLOHG LQIRUPDWLRQ RQ WKH DYHUDJH VWDQGDUG GHYLDWLRQ WKH PLQLPXP DQG WKH PD[LPXP RI WUDLQLQJ HUURU WHVWLQJ HUURU QXPEHU RI FRQQHFWLRQV SUXQHG DV ZHOO DV WKH SUXQLQJ SHUFHQWDJH IRU HDFKDOJRULWKPRQHDFKQHWZRUNFRQILJXUDWLRQLVVKRZQ7KH PHDQ DQG VWDQGDUG GHYLDWLRQ DUH VKRZQ LQ WKH IRUP RI PHDQ “VWG ZKLOHWKHPLQLPXPDQGPD[LPXPDUHVKRZQ DV >PLQ PD[@ )RU H[DPSOH LQ WDEOH  QHWZRUN FRQILJXUDWLRQIRU./6$DOJRULWKPRIWKHSDWWHUQV FDQ EH FRUUHFWO\ FODVVLILHG IRU WUDLQLQJ GDWDVHW  ZLWK D VWDQGDUG GHYLDWLRQ RI  WKH UDWH IRU FRUUHFW FODVVLILFDWLRQKDVDPLQLPXPRIDQGDPD[LPXPRI 7KHSHUIRUPDQFHRIWKHSURSRVHG+6$5DOJRULWKPLV YHU\ FRPSDWLEOH ZLWK DQ DYHUDJH UDWH IRU FRUUHFW FODVVLILFDWLRQRIVWDQGDUGGHYLDWLRQ DQGWKH PLQLPXP DQG PD[LPXP DW  DQG  UHVSHFWLYHO\ +RZHYHU WKH ELJ GLIIHUHQFH RQ WKH SUXQLQJ SHUFHQWDJH DYHUDJH  IRU ./6$ DQG  IRU +6$5 LQGLFDWHVWKHQHXUDOQHWZRUNSUXQHGE\WKHSURSRVHG +6$5DOJRULWKPLVPXFKVPDOOHU ,Q VXPPDU\ WKH VLPXODWLRQ UHVXOWV FRQVLVWHQWO\ LQGLFDWH WKH +6$5 DOJRULWKP FDQ UHGXFH WKH QHXUDO QHWZRUN VL]H VLJQLILFDQWO\ZLWKRXWVDFULILFLQJWKHQHWZRUNSHUIRUPDQFH

HIIHFWLYHO\LVSUHVHQWHGLQWKLVSDSHU6DWLVIDFWRU\VLPXODWLRQ UHVXOWV DUH REWDLQHG ZKHQ FRPSDULQJ ZLWK WKUHH RWKHU H[LVWLQJSUXQLQJDOJRULWKPV0RUHWHVWVZLOOEHFRQGXFWHGWR IXUWKHULQYHVWLJDWHWKHSHUIRUPDQFHRIWKLVQHZDOJRULWKP 5()(5(1&(6 >@ )QDLHFK 1 $ELG 6 )QDLHFK ) DQG &KHULHW 0 ³$ PRGLILHG YHUVLRQRIDIRUPDOSUXQLQJDOJRULWKPEDVHGRQORFDOUHODWLYHYDULDQFH DQDO\VLV´)LUVW,QWHUQDWLRQDO6\PSRVLXPRQ&RQWURO&RPPXQLFDWLRQV DQG6LJQDO3URFHVVLQJ SS >@ $ELG6 )QDLHFK)DQG1DMLP 0³$ IDVW )HHG)RUZDUG7UDLQLQJ $OJRULWKP 8VLQJD 0RGLILHG )RUP RIWKH6WDQGDUG%DFN3URSDJDWLRQ $OJRULWKP´ ,((( 7UDQVDFWLRQV RQ 1HXUDO 1HWZRUN   9ROXPH ,VVXHSS >@ $QGUHZV 5 'LHGHULFK - *ROHD 0 DQG 7LFNOH $% ³7KH WUXWK ZLOOFRPHWROLJKW'LUHFWLRQVDQGFKDOOHQJHVLQH[WUDFWLQJNQRZOHGJH HPEHGGHG ZLWKLQ WUDLQHG DUWLILFLDO QHXUDO QHWZRUNV´ ,((( 7UDQVDFWLRQVRQ1HXUDO1HWZRUNV 9ROXPH,VVXHSS  >@ %ODNH ( . & DQG 0HU] & ³8&, UHSRVLWRU\ RI PDFKLQH OHDUQLQJ GDWDEDVHV´ 8QLYHUVLW\ RI &DOLIRUQLD ,UYLQH 'HSDUWPHQW RI ,QIRUPDWLRQDQG&RPSXWHU6FLHQFH >@ (IH 02 ,SOLNFL 6 .D\DQN 2 DQG :LODPRZVNL % ³$Q $OJRULWKP IRU )DVW &RQYHUJHQFH LQ 7UDLQLQJ 1HXUDO 1HWZRUNV´ ,QWHUQDWLRQDO -RLQW &RQIHUHQFH RQ 1HXUDO 1HWZRUNV SS  :DVKLQJWRQ'&-XO\ >@ (QJHOEUHFKW $3 ³$ QHZ SUXQLQJ KHXULVWLF EDVHG RQ YDULDQFH DQDO\VLV RI VHQVLWLYLW\ LQIRUPDWLRQ´ ,((( 7UDQVDFWLRQV RQ 1HXUDO 1HWZRUNV 9ROXPH,VVXHSS >@ *LOHV &/ DQG /DZUHQFH 6 ³2YHUILWWLQJ DQG 1HXUDO 1HWZRUNV &RQMXJDWH *UDGLHQW DQG %DFNSURSDJDWLRQ´ 3URFHHGLQJV RI WKH ,((( ,QWHUQDWLRQDO&RQIHUHQFHRQ1HXUDO1HWZRUNV SS >@ +D\NLQ 6 ³1HXUDO QHWZRUNV D FRPSUHKHQVLYH IRXQGDWLRQ´ 8SSHU 6DGGOH5LYHU1-3UHQWLFH+DOO >@ +X\QK74DQG6HWLRQR5³(IIHFWLYHQHXUDOQHWZRUNSUXQLQJXVLQJ FURVVYDOLGDWLRQ´ ,((( ,QWHUQDWLRQDO -RLQW &RQIHUHQFH RQ 1HXUDO 1HWZRUNV 9ROXPHSS >@ .DUQLQ (' ³$ VLPSOH SURFHGXUH IRU SUXQLQJ EDFNSURSDJDWLRQ WUDLQHG QHXUDO QHWZRUNV´ ,((( 7UDQVDFWLRQV RQ 1HXUDO 1HWZRUNV  9ROXPH,VVXHSS >@ 0DUVODQG 6 1HKP]RZ 68 DQG 6KDSLUR - ³$ VHOIRUJDQL]LQJ QHWZRUNWKDW JURZV ZKHQ UHTXLUHG´LQ1HXUDO1HWZRUNV 9ROXPH  ,VXXHSS >@ 0R]HU 0 & DQG 6PROHQVN\ 3 ³6NHOHWRQL]DWLRQ $ WHFKQLTXH IRU WULPPLQJ WKH IDW IURP D QHWZRUN YLD UHOHYDQFH DVVHVVPHQW´ LQ $GYDQFHVLQ1HXUDO,QIRUPDWLRQ3URFHVVLQJ '67RXUHW]N\ (GSS >@ 3RQQDSDOOL 396 +R .& DQG 7KRPVRQ 0 ³$ IRUPDO VHOHFWLRQ DQG SUXQLQJ DOJRULWKP IRU IHHGIRUZDUG DUWLILFLDO QHXUDO QHWZRUN RSWLPL]DWLRQ´ ,((( 7UDQVDFWLRQV RQ 1HXUDO 1HWZRUNV   9ROXPH,VVXHSS

9 &21&/86,21 $ QHZ K\EULG LWHUDWLYH SUXQLQJ DOJRULWKP WKDW FDQ SUXQH PXOWLOD\HU IHHGIRUZDUG DUWLILFLDO QHXUDO QHWZRUNV YHU\

3316

2008 International Joint Conference on Neural Networks (IJCNN 2008)

  

 7$%/(, $&RPSDULVRQRI(DFK$OJRULWKP¶V3UXQLQJ3HUFHQWDJHV

'DWD

11

%DODQFH 6FDOH 



 /LYHU 'LVRUGHU

 









'LDEHWHV













./6$

/96$

&93

+6$5

“  “  “  “  “  “  “  “  “ 

“  “  “  “  “  “  “  “  “ 

“  “  “  “  “  “  “  “  “ 

“  “  “  “  “  “  “  “  “ 

7$%/(,,, 7KH'HWDLOHG5HVXOWVIRUWKH/LYHU'LVRUGHU'DWD7HVWV  ./6$ /96$ &93 +6$5 11FRQILJ     7UDLQLQJ “ “ “ “ $FFXUDF\     3HUFHQWDJH  > > > >  @ @ @ @ 7HVWLQJ “ “ “ “ $FFXUDF\     3HUFHQWDJH  > > > >  @ @ @ @ 3HUFHQW “ “ “ “ 3UXQHG     > > > >  @ @ @ @ 11FRQILJ     7UDLQLQJ “ “ “ “ $FFXUDF\     3HUFHQWDJH  > > > >  @ @ @ @ 7HVWLQJ “ “ “ “ $FFXUDF\     3HUFHQWDJH  > > > >  @ @ @ @ 3HUFHQW “ “ “ “ 3UXQHG     > > > >  @ @ @ @ 11FRQILJ     7UDLQLQJ “ “ “ “ $FFXUDF\     3HUFHQWDJH  > > > >  @ @ @ @ 7HVWLQJ “ “ “ “ $FFXUDF\     3HUFHQWDJH  > > > >  @ @ @ @ 3HUFHQW “ “ “ “ 3UXQHG     > > > >  @ @ @ @

 7$%/(,, 7KH'HWDLOHG5HVXOWVIRUWKH%DODQFH6FDOH'DWD7HVWV  ./6$ /96$ &93 +6$5 11FRQILJ     7UDLQLQJ “ “ “ “ $FFXUDF\     3HUFHQWDJH  > > > >  @ @ @ @ 7HVWLQJ “ “ “ “ $FFXUDF\     3HUFHQWDJH  > > > >  @ @ @ @ 3HUFHQW “ “ “ “ 3UXQHG     > > > >  @ @ @ @ 11FRQILJ     7UDLQLQJ “ “ “ “ $FFXUDF\     3HUFHQWDJH  > > > >  @ @ @ @ 7HVWLQJ “ “ “ “ $FFXUDF\     3HUFHQWDJH  > > > >  @ @ @ @ 3HUFHQW “ “ “ “ 3UXQHG     > > > >  @ @ @ @ 11FRQILJ     7UDLQLQJ “ “ “ “ $FFXUDF\     3HUFHQWDJH  > > > >  @ @ @ @ 7HVWLQJ “ “ “ “ $FFXUDF\     3HUFHQWDJH  > > > >  @ @ @ @ 3HUFHQW “ “ “ “ 3UXQHG     > > > >  @ @ @ @



   

2008 International Joint Conference on Neural Networks (IJCNN 2008)

3317

   7$%/(,9 7KH'HWDLOHG5HVXOWVIRUWKH'LDEHWHV'DWD7HVWV  ./6$ /96$ &93 11FRQILJ    7UDLQLQJ “ “ “ $FFXUDF\    3HUFHQWDJH  > > >  @ @ @ 7HVWLQJ “ “ “ $FFXUDF\    3HUFHQWDJH  > > >  @ @ @ 3HUFHQW “ “ “ 3UXQHG    > > >  @ @ @ 11FRQILJ    7UDLQLQJ “ “ “ $FFXUDF\    3HUFHQWDJH  > > >  @ @ @ 7HVWLQJ “ “ “ $FFXUDF\    3HUFHQWDJH  > > >  @ @ @ 3HUFHQW “ “ “ 3UXQHG    > > >  @ @ @ 11FRQILJ    7UDLQLQJ “ “ “ $FFXUDF\    3HUFHQWDJH  > > >  @ @ @ 7HVWLQJ “ “ “ $FFXUDF\    3HUFHQWDJH  > > >  @ @ @ 3HUFHQW “ “ “ 3UXQHG    > > >  @ @ @

+6$5  “  > @ “  > @ “  > @  “  > @ “  > @ “  > @  “  > @ “  > @ “  > @

 

3318

2008 International Joint Conference on Neural Networks (IJCNN 2008)