Sentiment-based Classification of Radical Text on the Web

2016 European Intelligence and Security Informatics Conference

6HQWLPHQWEDVHG&ODVVLILFDWLRQRI5DGLFDO 7H[WRQWKH:HE 5\DQ6FULYHQVDQG5LFKDUG)UDQN ,QWHUQDWLRQDO&\EHU&ULPH5HVHDUFK&HQWUH  6FKRRORI&ULPLQRORJ\6LPRQ)UDVHU8QLYHUVLW\ %XUQDE\&DQDGD ^UVFULYHQUIUDQN`#VIXFD  WKH VXEWOH ODQJXDJH± LQ WKH(QJOLVK WH[W ±IRXQG RQZHEVLWHV Abstract ² 7KH WRWDO QXPEHU RI ZHESDJHV KDV JURZQ IHDWXULQJH[WUHPLVWFRQWHQW$UJXDEO\DIXOO\DXWRPDWHGV\VWHP VXEVWDQWLDOO\ VLQFH WKH ELUWK RI WKH ,QWHUQHW 6R WRR KDYH WKH QXPEHU RI ZHESDJHV GHGLFDWHG WR UDGLFDO \HW VXEWOH FRQWHQW $V VKRXOGEHDYRLGHGZKHQKXPDQLQWHOOLJHQFHLVUHTXLUHG>@LH WKHVHQHZFLUFXPVWDQFHVKDYHQHFHVVLWDWHGDJXLGHGGDWDFROOHFWLRQ LGHQWLI\LQJVXEWOHIRUPVRIUDGLFDOWH[WLQ(QJOLVK  PHWKRGRQHWKDWFDQVLGHVWHSWKHODERULRXVPDQXDOPHWKRGVWKDW KDYH EHHQ FODVVLFDOO\ XWLOL]HG VLPSOH NH\ZRUG DQDO\VLV KDV QRW EHHQVXIILFLHQWWRLGHQWLI\UDGLFDOVLWHVRQ:HE±SURH[WUHPLVW DQWLH[WUHPLVW DQG QHZV VLWHV IRU H[DPSOH PD\ XVH WKH VDPH NH\ZRUGV WR GLVFXVV WKH VDPH HYHQW EXW KDYH D YHU\ GLIIHUHQW PRWLYDWLRQ,QDQHIIRUWWRH[SORUHWKLVSUREOHPZHFRPSOHWHGDQ H[HUFLVH LQYROYLQJ WKH XVH RI D ZHEFUDZOHU WR FROOHFW  ZHESDJHV IURP ILYH VHQWLPHQWEDVHG FODVVHV WR DVVHVV WKHLU GLIIHUHQFHV   UDGLFDO 5LJKW VLWHV   UDGLFDO ,VODPLF VLWHV   DQWLH[WUHPLVW VLWHV   QHZV VRXUFH VLWHV GLVFXVVLQJ H[WUHPLVP DQG VLWHVWKDWGLGQRWGLVFXVVH[WUHPLVP3DUWVRI6SHHFK326  WDJJLQJ ZDV XVHG WR LGHQWLI\  RI WKH PRVW IUHTXHQW NH\ZRUGV ZLWKLQ WKH GDWD DQG WKH VHQWLPHQW YDOXH IRU HDFK RI WKHVH NH\ZRUGV ZDV FDOFXODWHG IRU HDFK ZHESDJH XVLQJ VHQWLPHQW DQDO\VLV :LWK WKHVH YDOXHV D GHFLVLRQ WUHH ZDV DSSOLHG WR WKUHH FODVVLILFDWLRQPRGHOV5HVXOWVVXJJHVWWKDWUDGLFDO,VODPLFWH[WFDQ EHFODVVLILHGDWDPXFKKLJKHUUDWHRIVXFFHVVWKDQUDGLFDO5LJKW WH[W

 ,QWKHZRUNRI0HLDQG)UDQNWKHDXWKRUVGHYHORSHGDPRGHO WKDW FRPELQHG VHQWLPHQW DQDO\VLV DQG D ZHEFUDZOHU ZLWK D GHFLVLRQWUHHWRGLIIHUHQWLDWHSURH[WUHPLVWZHESDJHVIURPDQWL H[WUHPLVW SDJHV QHZV SDJHV DQG SDJHV WKDW GLG QRW UHODWH WR H[WUHPLVP +HUH WKH RYHUDOO JRDO RI WKLV VHPLDXWRPDWHG DSSURDFKZDVWRFUHDWHDWRROWKDW PDGHSUHGLFWLRQVDERXWWKH FRQWHQW IRXQG RQ WKH VLWHV LW GRZQORDGHG >@ :KLOH WKLV WHFKQLTXH PDQDJHG WR FODVVLI\ H[WUHPLVW WH[W DW D KLJK UDWH RI DFFXUDF\ LH   ERWK H[WUHPLVWEDVHG ZHEIRUXPV DQG ZHEVLWHV RI H[WUHPLVW RUJDQL]DWLRQV ZHUH DQDO\]HG ZLWKLQ RQH PRGHO$UJXDEO\WKHVHQWLPHQWIRXQGRQERWKW\SHVRISODWIRUPV DUH GLIIHUHQW LQ WHUPV RI WRQH DQG VXEMHFW PDWWHU ,QGHHG WKLV SUREOHPUHTXLUHVIXUWKHUH[SORUDWLRQ ,,

Keywords - Sentiment Analysis; Decision Trees; Extremism

,

,1752'8&7,21

 6LQFH WKH PLGV ZH KDYH VHHQ D UDSLG JURZWK LQ WKH QXPEHURIUDGLFDOZHEVLWHVDFWLQJDVDKXERIWKHPRYHPHQWWKH\ UHSUHVHQW,QPDQ\UHVSHFWVWKH\VHUYHDVDFRQYHUJHQFHVHWWLQJ IRUQHZFRPHUVWRIDPLOLDUL]HWKHPVHOYHVZLWKUDGLFDOWHDFKLQJV >@FRQQHFWLQJXVHUVWRRWKHUVLPLODUZHEVLWHVZHEIRUXPV DQGSDJHVRQWKH'DUN:HEWRQDPHEXWDIHZ>@&RXQWHU H[WUHPLVPRUJDQL]DWLRQVFRQWLQXHWRORRNIRUZD\VWRLGHQWLI\ WKHVH VLWHV >@ \HW WKH ,QWHUQHW LV FRQVWDQWO\ JURZLQJ DQG LQIRUPDWLRQLVEHLQJJHQHUDWHGDWDUDSLGSDFH>@7KLVKDVOHG WR D JURZLQJ IORRG RI GDWD UHVXOWLQJ LQ PDQXDO GDWD DQDO\VLV EHFRPLQJ OHVV HIILFLHQW RU HQWLUHO\ LQIHDVLEOH >@ ,Q UHVSRQVH UHVHDUFKHUV KDYH GHYHORSHG GDWD FROOHFWLRQ WRROV WKDW DOORZ GHFLVLRQVWREHPDGHDERXWWKHWKRXVDQGVRIZHESDJHVWKDWDUH H[WUDFWHGDQGE\H[WHQVLRQFODVVLI\UDGLFDOWH[WRQWKH:HE

A. Webpage Data 5HVHDUFKHUVKDYHVXJJHVWHGWKDWWKHVHQWLPHQWIRXQGRQERWK UDGLFDO 5LJKW > @ DQG UDGLFDO ,VODPLF > @ ZHEVLWHV DUH HOXVLYH$VDPHDQVRIH[SORULQJWKLVFODLPRQWKHSURH[WUHPLVW FRQWHQWDQGHYDOXDWLQJZKHWKHUWKHGHFLVLRQWUHHFRXOGFODVVLI\ FRQWHQW EDVHG RQ WKRVH VXEWOH LGHRORJLFDO PRWLYDWLRQV ILYH GLVWLQFWDQGQRQRYHUODSSLQJFODVVHVRISDJHVZHUHDWWDLQHG

 5HVHDUFKHUVKDYHH[SORUHGWKLVFULWLFDOSRLQWRIGHSDUWXUHYLD DXWRPDWHG FRPSXWDWLRQDO DQG VHPLDXWRPDWHG WRROV ERWK WR LPSURYH WKH PHWKRGV RI FROOHFWLRQ DQG LGHQWLILFDWLRQ RI H[WUHPLVW FRQWHQW &KHQ¶V Dark Web Project IRU H[DPSOH SURGXFHGDQXPEHURIFRPSXWDWLRQDODQGGDWDFHQWULFVWXGLHVRQ WKH FRQWHQW DQG VWUXFWXUH RIVXUIDFH OHYHO DQG 'DUN:HEVLWHV FRQWDLQLQJH[WUHPLVWPDWHULDO>@,QOLJKWRIWKLVFRPSUHKHQVLYH ZRUNWKHHQWLUHSURFHVVZDVDXWRPDWHGDQGWKH\GLGQRWDVVHVV 978-1-5090-2857-3/16 $31.00 © 2016 IEEE DOI 10.1109/EISIC.2016.41

0(7+2'6

7KHSXUSRVHRIWKLVVWXG\ZDVWRDGGWRWKHOLWHUDWXUHE\XVLQJ DVHQWLPHQWDQDO\VLVWRRODQGDGHFLVLRQWUHHWRGLIIHUHQWLDWHSUR H[WUHPLVWZHESDJHVIURPDQWLH[WUHPLVWSDJHVQHZVSDJHVDQG SDJHVWKDWGLGQRWUHODWHWRH[WUHPLVP7KLVZDVGRQHE\EXLOGLQJ WKUHHWH[WEDVHGFODVVLILFDWLRQPRGHOV&KDSWHU,,$ )LUVWZH XVHGWKH7HUURULVPDQG([WUHPLVP1HWZRUN([WUDFWRU7(1(  WR FROOHFW GDWD &KDSWHU   DQG 2SHQ1/3¶V 3DUWVRI6SHHFK 326 DQDO\VLVWRGHYHORSDOLVWRINH\ZRUGV&KDSWHU,,& 7KH VHQWLPHQW H[SUHVVHG ZLWKLQ WKH ZHESDJHV ZDV WKHQ FDOFXODWHG EDVHG RQ WKH 326 NH\ZRUGV GHWDLOLQJ WKH UHODWLYH VHQWLPHQW VFRUHVIRUHDFKSDJHIRUHDFKRIWKHNH\ZRUGV&KDSWHU,,'  )LQDOO\DGHFLVLRQWUHHZDVEXLOWEDVHGRQWKHVHQWLPHQWVFRUHV &KDSWHU,,( ZKLFKSURGXFHGWKHFODVVLILFDWLRQUHVXOWVIRUWKUHH FODVVLILFDWLRQPRGHOV&KDSWHU,,) VHH)LJXUH 

1) 7KH radical Right FODVV ZDV VHOHFWHG WKURXJK WZR PHWKRGV   D *RRJOH VHDUFK RI WKH NH\ZRUGV µH[WUHPLVWZHEVLWHV¶µZKLWHVXSUHPDF\ZHEVLWHV¶DQG RWKHUVLPLODUZRUGVDQG XVLQJDQLQGH[RIH[WUHPLVW VLWHV JHQHUDWHG E\ UHVHDUFKHUV DQG VRXUFHV IRU 104

 )LJXUH±'DWD&ROOHFWLRQDQG7UHH*HQHUDWLRQ3URFHVV

2)

3)

4)

5)

EHWZHHQ-XO\DQG-XO\IRUDPRUHGHWDLOHG GHVFULSWLRQRIWKHFUDZOHUVHH>@ 

H[DPSOHV VHH > @  7KLV FODVV FRQWDLQHG  ZHESDJHVVSUHDGRYHUZHEVLWHV 7KH radical Islamic FODVV ZDV VHOHFWHG WKURXJK WKH NH\ZRUGV µH[WUHPLVW ZHEVLWHV¶ µMLKDG RUJDQL]DWLRQV¶ DQG SUHYLRXVO\ FRPSLOHG LQGH[HV >@ 7KLV FODVV FRQWDLQHGZHESDJHVIURPZHEVLWHV $Qanti-extremist FODVVLQFOXGHGLQWHOOLJHQFHDJHQFLHV DQG RUJDQL]DWLRQV GHGLFDWHG WR FRXQWHULQJ H[WUHPLVP DQG ZHUH LGHQWLILHG YLD DQ RQOLQH VHDUFK RI VXFK NH\ZRUGVDVµFRXQWHUH[WUHPLVP¶DQGµDQWLH[WUHPLVP¶ JURXSV IRU H[DPSOH 7KH VDPSOH FRQWDLQHG  ZHESDJHVIURPZHEVLWHV $ news FODVV LQFOXGHG ZHEVLWHV LI WKH\ ZHUH VRXUFHV WKDWUHSRUWHGRQH[WUHPLVWHYHQWV+HUHLWZDVDVVXPHG WKDW WKH\ SUHVHQWHG D VWRU\ LQ D PRUH ³LPSDUWLDO´ PDQQHUWKDQSURH[WUHPLVWRUDQWLH[WUHPLVWVLWHV7KLV VDPSOHFRQWDLQHGZHESDJHVIURPZHEVLWHV $ILQDOFODVVotherVHUYHGDVWKHFRQWUROJURXSLQWKH VDPSOH ZKHUH RQH ZRXOG QRW H[SHFW WR XQFRYHU PDWHULDO UHODWLQJ WR H[WUHPLVP DQG LQFOXGHG WRSLFV UHODWLQJ WR DUW DQG FXOWXUH HGXFDWLRQ HQWHUWDLQPHQW IRRG WUDYHO DQG OLIHVW\OH VRFLDO PHGLD RQOLQH EORJV DXWRPRELOHVVSRUWVDQGEXVLQHVV)RUWKLVFODVV ZHESDJHVIURPZHEVLWHVZHUHVDPSOHG

C. Parts-of-Speech (POS) Tagging 7KH LQLWLDO VWHS LQ DQDO\]LQJ WKH FRQWHQW RQ ZHESDJHV DXWRPDWLFDOO\ ZDVWRGHWHUPLQHWKHWRSLFVRIGLVFXVVLRQ7RGR WKLV ZH ZDQWHG WR WDNH D GDWDGULYHQ DSSURDFK DQG LVRODWH WKH SDUWLFXODUQRXQVZLWKWKHKLJKHVWUDWHRIRFFXUUHQFHZLWKLQWKH GDWDFROOHFWHG:HDVVXPHGWKDWWKHPRVWIUHTXHQWO\GLVFXVVHG WRSLFVZRXOGPRVWOLNHO\EHWKHRQHVLQZKLFKH[WUHPLVWFRQWHQW ZDVOLNHO\WREHGHWHFWHGEDVHGRQWKHZRUNRI>@)XUWKHU QRXQVZHUHFKRVHQEHFDXVHWKH\DUHWKHZRUGVPRVWOLNHO\WREH VXUURXQGHGE\UHOHYDQWVHQWLPHQWWHUPV>@QDPHVDQGSODFHV DUHRIWHQGHVFULEHGRUGHQRWHGE\WKHDGMHFWLYHVOLQNHGWRWKHP DQG DGMHFWLYHV DUH RIWHQ WKH ZRUGV WKDW KDYH VHQWLPHQW YDOXHV DWWDFKHGWRWKHP>@%\VSHFLI\LQJDGMHFWLYHVDVNH\ZRUGVWKH VHQWLPHQWRIWKHZRUGLWVHOIZRXOGEHORVW 3DUWVRI6SHHFK 326  WDJJLQJ ZDV XVHG WR SHUIRUP WKLV DQDO\VLV326LVDGDWDDQDO\VLVPHWKRGWKDWFROOHFWVDQGGLYLGHV VXSSOLHG WH[W LQWR ZRUG JURXSLQJV VXFK DV QRXQV DQG YHUEV EDVHGRQWKHLUXVDJHDQGSRVLWLRQZLWKLQWKHVHQWHQFHIRUPRUH LQIRUPDWLRQVHH>@  )URPHDFKFODVVZHVHOHFWHGRIWKHPRVWIUHTXHQWQRXQV DQG DJJUHJDWHG WKHP LQWR RQH OLVW ZKLFK DIWHU UHPRYDO RI GXSOLFDWHVDQGPLVFDWHJRUL]HGZRUGVV\PEROVDQGQRQZRUGV IRU H[DPSOH  ZH HQGHG ZLWK  NH\ZRUGV 7KLV ILQDO OLVW IRUPHG WKH NH\ZRUG OLVW IRU WKH VHQWLPHQW DQDO\VLV 'RPDLQ H[SHUWVPD\UHSODFHRUH[WHQGWKHOLVWRINH\ZRUGVZLWKUHOHYDQW GRPDLQVSHFLILF ZRUGV +RZHYHU WKH H[SORUDWRU\ DQG GDWD GULYHQQDWXUHRIWKHVWXG\VXJJHVWHGWKDWZHUHO\RQWKHGDWDWR LGHQWLI\WKHUHOHYDQWNH\ZRUGV



7KUHH PRGHOV ZHUH WKHQ FRQVWUXFWHG IRU WKH SXUSRVHV RI FRPSDULQJDQGFRQWUDVWLQJWKHLUFODVVLILFDWLRQUHVXOWV 

Two-class model: radical RightDQGIslamicYHUVXV  anti-extremistnewsDQGotherZHESDJHV



Four-class model: radical RightDQGradicalIslamic YHUVXV   anti-extremist YHUVXV   news YHUVXV   otherZHESDJHV



Five-class model: radical RightYHUVXV radical IslamicYHUVXV anti-extremistYHUVXV newsYHUVXV  other ZHESDJHV

D. Sentiment Analysis $IWHU WKH NH\ZRUG OLVW ZDV GHYHORSHG LW ZDV QHFHVVDU\ WR LGHQWLI\DQGHYDOXDWHWKHFRQWH[WVXUURXQGLQJWKHNH\ZRUGV7R DOORZDSURSHUDQDO\VLVRIWKHZHESDJHVVHQWLPHQWDQDO\VLVZDV XVHGWRKLJKOLJKWUHOHYDQWWH[W 6HQWLPHQWDQDO\VLVLVDGDWDFROOHFWLRQDQGDQDO\VLVPHWKRG WKDW DOORZV IRU WKH DSSOLFDWLRQ RI VXEMHFWLYH ODEHOV DQG FODVVLILFDWLRQV>@,WFDQHYDOXDWHWKHRSLQLRQVRILQGLYLGXDOVE\ RUJDQL]LQJGDWDLQWRGLVWLQFWFODVVHVDQGVHFWLRQVDQGDVVLJQLQJ DQ LQGLYLGXDO¶V VHQWLPHQW ZLWK D QHJDWLYH RU SRVLWLYH SRODULW\ YDOXH>@,WDOVRSURYLGHVDPRUHWDUJHWHGYLHZRIDGDWDVHWE\

B. Terrorism and Extremism Network Extractor (TENE) 7KH7HUURULVPDQG([WUHPLVP1HWZRUN([WUDFWRU7(1(  DFXVWRPZULWWHQFRPSXWHUSURJUDPWKDWZDVGHVLJQHGWRFROOHFW YDVWDPRXQWVRIGDWDRQOLQHDXWRPDWLFDOO\EURZVHGDQGFDSWXUHG DOORIWKHFRQWHQWIURPWKHZHEVLWHVXVHGIRUFODVVLILFDWLRQ

105

,,, 5(68/76

DOORZLQJIRUWKHGHPDUFDWLRQEHWZHHQFDVHVWKDWDUHVRXJKWDIWHU DQGWKRVHZLWKRXWDQ\QRWDEOHUHOHYDQFH6LQFHWKHSXUSRVHRI WKHFXUUHQWVWXG\ZDVQRWWRSXVKWKHERXQGDULHVRIVHQWLPHQW DQDO\VLVDOJRULWKP6HQWL6WUHQJWK>@ZDVXWLOL]HG

7KHUHVXOWVZHUHLQWHUSUHWHGLQWZRZD\V)LUVWWKHRYHUDOO FODVVLILFDWLRQ UHVXOWV ZHUH HYDOXDWHG IRU HDFK FODVVLILFDWLRQ PRGHO IROORZHG E\ WKHTXDOLW\ RI WKH HQWLUH UXOH VHW IRU HDFK +HUHWKHDQDO\VLVZDVODUJHO\IRFXVHGRQWKHSURH[WUHPLVWFODVV

6HQWL6WUHQJWK LV -DYDEDVHG VRIWZDUH WKDW XVHV D VSHFLILF DOJRULWKP WR UXQ WKURXJK ODUJH YROXPHV RI WH[W DQG FUHDWH VHQWLPHQW VFRUHV IRU WKH VXSSOLHG GRFXPHQWV :KLOH WKHUH DUH VHYHUDOFRQILJXUDWLRQVRIWKHVRIWZDUHZHXWLOL]HGDNH\ZRUG IRFXVHGPHWKRGDVDFHQWUDOIHDWXUHRI6HQWL6WUHQJWKLVLWVDELOLW\ WR HYDOXDWH VHQWLPHQW DURXQG DQ\ JLYHQ NH\ZRUG >@ 7KLV SURFHVV LQYROYHV WKH XWLOL]DWLRQ RI D GLFWLRQDU\ RI FDWDORJXHG WHUPV DQG +DUYDUG¶V JHQHUDO LQTXLUHU GDWDEDVH WR GHWHUPLQH VHQWLPHQWYDOXHV>@,WORFDWHVZRUGVWKDWFRUUHVSRQGZLWKLWV GLFWLRQDU\DQGGDWDEDVHDQGWKHQLWHPSOR\VDVWHPPLQJPHWKRG WRHYDOXDWHDWH[WE\DVVLJQLQJSRODULW\YDOXHVRIHLWKHUSRVLWLYH RUQHJDWLYHWRWKHZRUGV9DOXHVDUHDXJPHQWHGE\FKDUDFWHUV WKDWFDQLQIOXHQFHWKHYDOXHVDVVLJQHGWRWKHWH[WVXFKDVERRVWHU ZRUGVQHJDWLYHZRUGVUHSHDWHGOHWWHUVUHSHDWHGQHJDWLYHWHUPV DQWDJRQLVWLFZRUGVSXQFWXDWLRQDQGRWKHUGLVWLQFWLYHFKDUDFWHUV VXLWHGIRUVWXG\LQJDQRQOLQHFRQWH[W>@IRUPRUHLQIRUPDWLRQ RQ6HQWL6WUHQJWKVHH>@ 

A. Two-Class Model: Results and Quality of Rule-Set 7KH WZRFODVV WUHH DQDO\VLV LQGLFDWHG WKDW  RI ZHESDJHV ZHUH VXFFHVVIXOO\ FODVVLILHG DFURVV FODVVHV 7KH FRPELQHG radical Right DQG radical Islamic FODVV ZDV VXFFHVVIXOO\ FODVVLILHG DW D FRQVLGHUDEO\ ORZHU UDWH WKDQ WKH FRPELQHGanti-extremistnewsDQGotherFODVVLHRI SDJHVZHUHDFFXUDWHO\LGHQWLILHG )XUWKHUPRUHWKHUHFDOORIWKH HQWLUH UXOHVHW LQGLFDWHG WKDW DFWXDO WZR UDGLFDO FODVVHV ZHUH FODVVLILHG DW D UDWH RI  LQ FRPSDULVRQ WR WKH QRQUDGLFDO FODVVHV¶UHFDOOPHDVXUHRIVHHSUHFLVLRQDQGUHFDOOLQ7DEOH   B. Four-Class Model: Results and Quality of Rule-Set 7KH IRXUFODVV WUHH DQDO\VLV UHYHDOHG WKDW  RI ZHESDJHV ZHUH DFFXUDWHO\ FODVVLILHG DFURVV FODVVHV ,QWHUHVWLQJO\newsSDJHVKDGWKHKLJKHVWQXPEHURIFRUUHFWO\ FODVVLILHGSDJHVLH IROORZHGE\WKHFRPELQHGWZR UDGLFDOFODVVHVLH ZKLFKDJDLQVKRZHGPRGHUDWHO\KLJK SUHFLVLRQ UDWHV LH   DQG PRGHUDWHO\ ORZ UHFDOO UDWHV LH NewsFODVVVLWHVZHUHDOVRPLVFODVVLILHGDFURVVWKH UHPDLQLQJWKUHHFODVVHVDWWKHKLJKHVWIUHTXHQF\VHH7DEOH 

E. Decision Tree $WRROLQPDFKLQHOHDUQLQJDQGGDWDPLQLQJWKDWLVZLGHO\ DFFHSWHG DPRQJVW DFDGHPLFV :DLNDWR (QYLURQPHQW IRU .QRZOHGJH$QDO\VLV:(.$ ZDVUXQRQWKHVHQWLPHQWGDWD IRUHDFKRIWKHWKUHHFODVVLILFDWLRQPRGHOV,WVVWDQGDUG-WUHH FODVVLILFDWLRQPHWKRGZDVXVHGEHFDXVHLWFRQWDLQVDQDOJRULWKP IRUWH[WFODVVLILFDWLRQWKDWDOORZVIRUDUXOHEXLOGLQJSURFHVV>@ :(.$ZDVDSSOLHGXVLQJIROGFODVVLILFDWLRQRQWKHRXWSXWWHG VHQWLPHQWWDEOHVVHDUFKLQJIRUGLIIHUHQFHVLQVHQWLPHQWYDOXHV EHWZHHQWKHFODVVHVRIVLWHV$VDUHVXOWWKLVDOJRULWKPSURGXFHG DGHFLVLRQWUHHZLWKUHFXUVLYHOHDYHVDQGEUDQFKHVZLWKHDFKOHDI UHSUHVHQWLQJDVSHFLILFVHWRIVHQWLPHQWWKUHVKROGV

C. Five-Class Model: Results and Quality of Rule-Set 7KH ILYHFODVV WUHH DQDO\VLV LQGLFDWHG WKDW  RI WKH SDJHVZHUHDFFXUDWHO\FODVVLILHG2ISDUWLFXODULQWHUHVWKHUHZHUH WKHradical Right-DQGradical Islamic SDJHV)LUVWWKHFRQIXVLRQ PDWUL[LQGLFDWHGWKDWDPHUHRIradical RightSDJHVZHUH VXFFHVVIXOO\FODVVLILHG:LWKLQWKLVSDUWLFXODUFODVVnewsSDJHV ZHUHPLVLGHQWLILHGDWWKHKLJKHVWUDWHLQWKHPRGHORI LWV SDJHV ZHUH PLVFODVVLILHG DV UDGLFDO ULJKWZLQJ 6HFRQG  RI SDJHV WKDW IHDWXUHG UDGLFDO ,VODPLF FRQWHQW ZHUH DFFXUDWHO\FODVVLILHGNewsSDJHVZHUHDOVRPLVFODVVLILHGDWWKH KLJKHVWUDWHLH ZLWKLQWKHradical Islamic FODVVNews SDJHVDJDLQKDGWKHKLJKHVWQXPEHURIFRUUHFWO\FODVVLILHGSDJHV DFURVVWKHHQWLUHVDPSOH VLPLODUWRWKHFODVVLILFDWLRQ UHVXOWVLQWKHIRXUFODVVPRGHO0HDVXULQJWKHTXDOLW\RIWKHUXOH IRU WKH ILYHFODVV PRGHO WKH DFWXDO radical Right SDJHV ZHUH FODVVLILHG DW DORZ UHFDOO UDWH RI  $W WKH RWKHU HQG RIWKH VSHFWUXPERWKWKHSUHFLVLRQDQGUHFDOOPHDVXUHVLQGLFDWHGWKDW radical IslamicSDJHVZHUHFODVVLILHGDWDKLJKHUUDWHRIVXFFHVV WKDQWKHradical RightFODVVRIWKHradical IslamicSDJHV ZHUHVXFFHVVIXOO\FODVVLILHGDQGDFWXDOSDJHVZLWKLQWKLVFODVV ZHUHVXFFHVVIXOO\FODVVLILHGDWDUDWHRI6HH7DEOH 

,WLVZLWKWKHVHWKUHVKROGVWKDWWKHGHFLVLRQWUHHHVWDEOLVKHG ZKHWKHUDFHUWDLQZHESDJHZDVRUJDQL]HGLQWRWKHradicalright radical Islamic pro-extremist anti-extremist news RU other FODVV)RUH[DPSOHLIWKHNH\ZRUGµPDUULDJH¶ZDVLQFOXGHGLQ WKHWH[WZLWKDVHQWLPHQWYDOXHRIOHVVWKDQRUHTXDOWRDQGWKH WHUPµSULFH¶ZDVDOVRLQWKHWH[WZLWKDVHQWLPHQWYDOXHRIOHVV WKDQRUHTXDOWRDYDOXHRIWKHQWKH-SUXQHGWUHHLQGLFDWHG WKDWRIWKHSDJHVIHOOZLWKLQWKHnewsFODVVLILFDWLRQ F. Classification Results  $IWHU WKH -DOJRULWKP ZDV UXQ RQ WKH VHQWLPHQW GDWD IRU HDFK RI WKH WKUHH PRGHOV :(.$ FUHDWHG D PHDVXUH WKDW VSHFLILHGWKHQXPEHURIDFFXUDWHSDJHVWKDWILWLQWRHDFKRIWKH FRUUHFWFODVVHV,QWXUQDFRQIXVLRQPDWUL[GLVSOD\HGWKHQXPEHU RISDJHVWKDWZHUHFRUUHFWO\DQGLQFRUUHFWO\LGHQWLILHGLHIDOVH SRVLWLYHV DQG IDOVH QHJDWLYHV  LQ HDFK PRGHO PHDVXUHG ZLWK SUHFLVLRQDQGUHFDOO3UHFLVLRQLVDPHDVXUHZKLFKUHSUHVHQWVWKH H[DFWQHVVRIWKHUXOHV,WFDQEHWKRXJKWRIDVWKHQXPEHURIWUXH SRVLWLYHHQWLWLHVLGHQWLILHGE\WKHFODVVLILHURXWRIWKHVHWRIDOO HQWLWLHV LGHQWLILHG DV SRVLWLYH 7KH ODUJHU WKH YDULDELOLW\ LQ WKH UHVXOWVWKHVPDOOHUWKHSUHFLVLRQZLOOEH5HFDOOLVDPHDVXUHRI FRPSOHWHQHVVDQGLVWKHQXPEHURISRVLWLYHHQWLWLHVidentifiedRXW RIallWUXHSRVLWLYHHQWLWLHV$UHFDOOVFRUHRISHUIHFWPHDQV WKDWDOOWUXHSRVLWLYHVDUHLGHQWLILHGZLWKRXWDQ\IDOVHQHJDWLYHV

,9 &21&/86,216  :HVRXJKWWREXLOGRQDVHQWLPHQWJXLGHGZHEFUDZOHUWKDW FRXOG DFFXUDWHO\ FODVVLI\ WKH VXEWOH \HW UDGLFDOEDVHG WH[W RQ UDGLFDO 5LJKW DQG UDGLFDO ,VODPLF ZHESDJHV $ IHZ QRWDEOH ILQGLQJV ZHUH SURGXFHG )LUVW FODVVLILFDWLRQ UHVXOWV IURP WKH WZRDQGIRXUFODVVPRGHOVVXJJHVWHGWKDWWKHVHQWLPHQWJXLGHG ZHEFUDZOHUODFNHGWKHRYHUDOODELOLW\WRGLIIHUHQWLDWHEHWZHHQ WKHVHQWLPHQWIRXQGRQWKHVHOHFWHGZHEVLWHVWKDWGLGDQGGLGQRW SURPRWHUDGLFDOLGHRORJLHV7KLVILQGLQJFRQWUDVWVWKDWRI>@ ZKHUH SURH[WUHPLVW ZHEIRUXP DQG ZHESDJH GDWD ZHUH

106

  $FWXDO3DJHV 

 

3UHGLFWHG3DJHV Radical RightDQGRadical Anti-extremist, News DQG Islamic Other

 

 3UHFLVLRQ

 Radical RightDQG    Radical Islamic  Anti-extremist,    News DQG Other 7DEOH±&RQIXVLRQ0DWUL[IURPWKH-7UHH$QDO\VLVIRUWKH7ZR&ODVV0RGHO









3UHGLFWHG3DJHV Radical RightDQG Anti-extremist News Radical Islamic

Other

 

 5HFDOO  





3UHFLVLRQ

5HFDOO



Radical RightDQG      Radical Islamic       $FWXDO3DJHV Anti-extremist       News       Other 7DEOH±&RQIXVLRQ0DWUL[IURPWKH-7UHH$QDO\VLVIRUWKH)RXU&ODVV0RGHO

   



  3UHGLFWHG3DJHV  Radical Radical 3UHFLVLRQ Anti-extremist News Other   Right Islamic        Radical Right        Radical Islamic        $FWXDO3DJHV Anti-extremist        News        Other 7DEOH±&RQIXVLRQ0DWUL[IURPWKH-7UHH$QDO\VLVIRUWKH)LYH&ODVV0RGHO





 5HFDOO     



FODVVLILHGDWDKLJKUDWHRIDFFXUDF\6HFRQGWKHILYHFODVVPRGHO VKRZHG WKDW UDGLFDO ,VODPLF SDJHV ZHUH FODVVLILHG DW D PXFK KLJKHU UDWH RI VXFFHVV WKDQ UDGLFDO ULJKWZLQJ SDJHV 3UHYLRXV UHVHDUFKVXJJHVWVWKDWWKHVHQWLPHQWRQUDGLFDO5LJKWZHEVLWHV HJ>@ DQGUDGLFDO,VODPLFZHEVLWHVLH>@ LVSUHVHQWHG LQDVXEWOHPDQQHUERWKWRDSSHDOWRDZLGHUDXGLHQFHDQGUHFUXLW QHZPHPEHUVIRUH[DPSOH+RZHYHURXUUHVXOWVOHQGVXSSRUW IRU >@¶V DVVHUWLRQ WKDW ZKLOH UDGLFDO ,VODPLF VLWHV DWWHPSW WR OHJLWLPL]HWKHLUHIIRUWVE\SUHVHQWLQJWKHPVHOYHVDVQHZVVRXUFH ZHEVLWHVWKHVHQWLPHQWLVDOPRVWDOZD\VUHODWHGWRUDGLFDOWRSLFV LH GLVFXVVLRQV RI YLROHQFH  7KDW VDLG D PRUH LQGHSWK DQDO\VLVLVQHHGHGWRDVVHVVZKHWKHUWKHUHLVDFOHDUGLVWLQFWLRQ EHWZHHQDQH[WUHPLVWZHEVLWHIHDWXULQJHOXVLYHPHVVDJHVDQGD QHZVVLWH7KLVSUREOHPFRXOGEHH[SORUHGXVLQJRWKHUWRROVRI FODVVLILFDWLRQVXFKDVUDQGRPIRUHVWVRU%D\HVLDQPHWKRGVWR VXSSRUW YHFWRU PDFKLQHV DQG QHXUDO QHWZRUNV )XWXUH VWXGLHV VKRXOG DOVR LQWHJUDWH D TXDOLWDWLYH XQGHUVWDQGLQJ RI KRZ PDFKLQHOHDUQLQJWRROVPDNHGHFLVLRQVDERXWWKHZHESDJHVWKDW DUHYLVLWHG'RLQJVRPD\LQFUHDVHWKHUHOLDELOLW\RIWKHUHVXOWV DQGLQFUHDVHWKHOLNHOLKRRGRILGHQWLI\LQJUDGLFDOWH[WRQOLQH 9        



           

     

5()(5(1&(6

 

/ %DFN ³$U\DQV 5HDGLQJ $GRUQR &\EHU&XOWXUH DQG 7ZHQW\)LUVW &HQWXU\ 5DFLVP´ LQ Ethnic and Racial Studies, 25    SS  /%RZPDQ*ULHYH³$QWLDERUWLRQ([WUHPLVP2QOLQH´LQFirst Monday, 14   0&DLDQL'GHOOD3RUWD &:DJHPDQQMobilizing on the Extreme Right: Germany, Italy, and the United States2[IRUG2[IRUG8QLYHUVLW\ 3UHVV  + &KHQ Dark Web: Exploring and Data Mining the Dark Side of the Web1HZ