Detecting Cyberbullying Entries on Informal School Websites Based ...

Detecting Cyberbullying Entries on Informal School Websites Based on Category Relevance Maximization

Taisei Nitta, Fumito Masui, Michal Ptaszynski, Yasutomo Kimura, Rafal Rzepka and Kenji Araki

1

I explain about our study, Detecting Cyberbullying Entries on Informal School Websites Based on Category Relevance Maximization.

Introduction He's ugly!

BBS,e-mails, Facebook, Twitter...

You mean Taisei from Kitami?

Cyberbullying (slandering and humiliating people on the Internet) is a new social problem. 2

“Cyberbullying” is a new form of bullying. It is carried out on the Internet instead of classrooms. It takes form of hate messages sent through e-mails, BBS, and so on. Recently, it has become a serious social problem in many countries, one of which is Japan. Some cases of cyberbullying lead the students who were bullied to kill themselves or wrote the bullying entry on the BBS. ※文科省のデータに報告されているいじめの件数があるので説明できるようにしておく

Introduction Examples : 1. 性格悪ーいぶちゃいくー(笑) (Baaad personality, and an ugly hag, lol) 2. 調子乗りすぎいっぺん殺らなあかんで (Don’t get excited that much or I’ll kill ya!) 3. 新田キモイつかキショイほんま死んで (Nitta, you’re ugly, or rather fugly, just die)


Some of the actual examples of harmful entries are like these. These entries are slandering and humiliating student personality, appearance. 最近のいじめってLINEで起きてるみたいだけどそっちでも検出できるの？ →今回は既に書かれた書込みを発見しているが，今後は書込む際に警告を出すようにしていこうと思っているので，LINEにも対応できるはず

Introduction Examples : PTA members are voluntarily performing 1.性格悪ーいぶちゃいくー(笑) Website monitoring (net-patrol): (Baaad personality, and an ugly hag, lol) 1.2.調子乗りすぎいっぺん殺らなあかんで Manually searching for bullying entries (reading Web contents) (Don’t getwhole excited that much or I’ll kill ya!) 2. Sending deletion requests to Web page owners

3.新田キモイつかキショイほんま死んで (Nitta, you’re ugly, or rather fugly, just die)


To deal with the problem, Parent Teacher Association members perform website monitoring activities called “net-patrol”. When net-patrol members detected harmful entries, they send a request to remove the entry to Internet provider or website administrator.

Introduction Problems with net-patrol • It is performed manually. • There are 38,620 informal school Websites (state for 2008.08).

5

Unfortunately, net-patrol has been performed mostly by hand. Therefor, it takes much time and effort to detect harmful entries in a large amount of websites. Moreover, the task comes with a great psychological burden on mental health to the netpatrol members. ネットパトロール担当者はどのくらいいる？ →担当者は主に学校の先生(+生徒の親)，しかもボランティアだから少ない →大雑把な割合とかでもいいから数字で教えると納得してもらえそう

Introduction Problems with net-patrol • It is performed manually. • There are 38,620 informal school Websites (state for 2008.08).

Tatsuaki Matsuba, Fumito Masui, Atsuo Kawai, Naoki Isu. 2011. Gakkou hikoushiki saito ni okeru yuugai jouhou kenshutsu wo mokuteki to shita kyokusei hantei moderu ni kansuru kenkyu [Study on the polarity classification model for the purpose of detecting harmful information on informal school sites] (in Japanese), In Proceedings of The Seventeenth Annual Meeting of The Association for Natural Language Processing (NLP2011), pp. 388-391.

6

To solve the problem, Matsuba proposed a method to detect harmful entries automatically. Their method can detect harmful entries with an accuracy of 83%. 松葉さんのリファーがわかりやすいように

Introduction Matsuba’s method was first tested on data set consisting in 50% of cyberbullying entries In real harmful entries occur less often The method had to be verified under more realistic conditions

7

However, their method was not yet verified usability. Because, they used test data contained 50% of harmful entries. The ratio of harmful entries is not 50% in actual websites.

Introduction Matsuba’s method was first tested on data set consisting in 50% of cyberbullying entries In real harmful entries occur less often The method had to be verified under more realistic conditions Propose a new method and compare under realistic conditions Prepare dataset containing real amount of cyberbullying and use in evaluation 8

In our study, we proposed a method to detect harmful entries automatically. And evaluate our method to verify the usability in the situation close to reality. To evaluate our method, we prepare dataset of assuming the reality.

PMI-IR Classification PMI-IR(word) = PMI(word,excellent) − PMI(word,poor)

Thumbs Up or Thumbs Down? Semantic Orientation Applied to Unsupervised Classification of Reviews [2002, Turney] 9

Next, I will explain about the proposed method. Proposed method is extended the method of PMI-IR. PMI-IR was developed by Turney.


PMI : Pointwise Mutual Information IR : Information Retrieval Thumbs Up or Thumbs Down? Semantic Orientation Applied to Unsupervised Classification of Reviews [2002, Turney] 10

Here, PMI is Pointwise Mutual Information, and IR is Information Retrieval. In other words, PMI-IR is calculating the relevance of each words by used information retrieval(search web).



For example, I calculate relevance of word1 with word2. I search websites that included both word1 and word2, like this query.

PMI-IR Classification PMI-IR(apple,Mac )

PMI-IR(word) = PMI(word,excellent) − PMI(word,poor)

Query : apple AND Mac


For example, I calculate relevance of word1 with word2. I search websites that included both word1 and word2, like this query.

PMI-IR Classification PMI-IR(apple,Mac )

PMI-IR(word) = PMI(word,excellent) − PMI(word,poor)

Query : apple AND Mac

High: relevance Low Information relevance PMI Pointwise Mutual IR : Information Retrieval Thumbs Up or Thumbs Down? Semantic Orientation Applied to Unsupervised Classification of Reviews [2002, Turney] 12

If search result is many, the relevance is high. But, if search result is few, the relevance is low. This example maybe very high relevance.(HAHAHA) And, Turney judge the meaning of review site comment is positive or negative, to calculate relevance of word in comment with excellent or poor.



If search result is many, the relevance is high. But, if search result is few, the relevance is low. This example maybe very high relevance.(HAHAHA) And, Turney judge the meaning of review site comment is positive or negative, to calculate relevance of word in comment with excellent or poor.


PMI-IR(phrase) = max(PMI(phrase, "die","kill","slap") , … … , PMI(phrase,"annoying","gross","ugly"))

Phrase

Harmful polarity words 13

In our method, we calculate the harmfulness of entries in informal school websites. To calculate, we changed the “word” to phrase, and changed "excellent" and "poor" to harmful polarity words. Then, I will explain about phrase and harmful polarity words.

Phrase Harmful 死ね (Die [imp.])

ブス (Ugly)

きもい (Gross) 性格が悪い (Bad personality)

Non-Harmful 性格 (Personality) 悪い (Bad)

Dependency 14

First up is about phrase. The phrase is morpheme pairs that become harmful when 2 words have dependency relation. Some of harmful entries, word itself is non-harmful, but it become harmful by dependency relation.

Phrase Phrase pattern

Noun−noun

Harmful entries

Noun−verb

ex.:サル顔 (monkey face)

ex.:彼を殺す (kill him) Noun−adjective ex.:性格が悪い (bad personality) 15

Therefore, we investigate the harmful entries, and found a tendency that the entries become harmful by these dependency relation. For example, phrase pattern of Noun-Noun is humiliating person’s appearance like “monkey face”. The word of “monkey” is Noun, and “face” is also Noun. So, we defined morpheme pairs as "phrase" that have these dependency relation. And, extract from entries to calculate harmfulness.


PMI-IR(phrase) = max(PMI(phrase, "die","kill","slap") , … … , PMI(phrase,"annoying","gross","ugly"))

Phrase


Next, I will explain about the harmful polarity words.

Harmful polarity words Harmful words ・きもい (gross) ・死ね (die [imp.]) etc... (255 words)

17

Harmful polarity words are very harmful words. If the relevance of phrase with harmful polarity words is high, then the phrase will be harmful. To register harmful polarity words, we had registered 255 harmful words by investigation of harmful entries.

Harmful polarity words Harmful words ・きもい (gross) ・死ね (die [imp.]) etc... (255 words)

Obscene

Violent

Abusive

Ministry of Education, Culture, Sports, Science and Technology (MEXT). 2008. ‘Netto-jou no ijime' ni kansuru taiou manyuaru jirei shuu (gakkou, kyouin muke) [”Bullying on the Net” Manual for handling and collection of cases (for schools and teachers)] (in Japanese). Published by MEXT.

18

And, we have classified harmful words into 3 categories, obscene words, violence words, and abusive words. The Ministry of Education defined words considered as cyberbullying to include obscene, violent, or abusive words. According to Ministry of Education, our study also classified harmful words into 3 categories.

Harmful polarity words Harmful words

Obscene セックス, ヤリマン, フェラ (sex, slut, fellatio)

・きもい (gross) ・死ね (die [imp.]) etc... (255 words)

3 most frequent words for each category

Violent 死ね, 殺す, 殴る (die [imp.], kill, slap)

Abusive うざい, きもい, 不細工 (annoying, gross, ugly)


Furthermore, we selected 3 most frequent words for each category, and defined these words as harmful polarity words. 人によっては暴力よりも

謗に見える語があるけどどうやって分類した？

→文部科学省が定義を決めているのでそれに従いました

PMI-IR Classification PMI-IR(phrase) = max(PMI(phrase, "die","kill","slap") , … … , PMI(phrase,"annoying","gross","ugly"))

20

By using the phrase and harmful polarity words, we calculate harmfulness of phrase.

PMI-IR Classification PMI-IR(phrase) = max(PMI(phrase, "die","kill","slap") , … … , PMI(phrase,"annoying","gross","ugly"))

score = max (PMI -IR(phrase1 , ..., phrasen )) Harmfulness of an entry

21

Next, we calculates the harmfulness of all phrases that have been extracted from the entry. Finally, we set maximum harmfulness of each phrase to “score”. This “score” is the harmfulness of an entry.

PMI-IR Classification Example of calculation score : 可愛いけど性格が悪い女 (Cute girl, but bad personality)

可愛い-女

性格-悪い

悪い-女

(cute - girl)

(bad-personality)

(bad-girl)

22

Now, we explain this process on the following example. First, we extract phrases from this entry. From this entry, we can extract these 3 phrases.

PMI-IR Classification PMI-IR(phrase) = max(PMI(phrase, "die","kill","slap") , … … , PMI(phrase,"annoying","gross","ugly")) Relevance 可愛い-女 (cute - girl)

5 8 10

Obscene Violent Abusive

PMI-IR(可愛い-女) = 10 (cute-girl) 23

Next, we calculate the relevance of phrase “cute-girl” with each 3 category of harmful polarity words. In this case, relevance with abusive words is highest than others. So, we decide the harmfulness of this phrase as 10.

PMI-IR Classification score = max (PMI -IR(phrase1 , ..., phrasen )) PMI-IR(可愛い-女) = 10 (cute-girl)

PMI-IR(性格-悪い) = 15 (bad-personality)

PMI-IR(悪い-女) = 14 (bad-girl) 可愛いけど性格が悪い女 (Cute girl, but bad personality)

score = 15 24

In a similar way, we calculate the score of all phrase that can extracted from this entry. In this case, the highest score is 15 of all phrases. Finally, we decided score of the entry as 15.

PMI-IR Classification score = max (PMI -IR(phrase1 , ..., phrasen )) Entry 1 Entry 2 Entry 3 Low

High

score

25

By using expression like this, we calculate the “score” of entry that can extract the phrase.

PMI-IR Classification score = max (PMI -IR(phrase1 , ..., phrasen )) Entry 1 Entry 2 Entry 3 Low

High

score

26

PMI-IR Classification

Entry 3

Entry 1

Low

Entry 2

High

score

27

And we sort entries in descending order of “score” is high.

PMI-IR Classification Threshold (n) NonHarmful harmful 3rd

Entry 3

Low

2nd

Entry 1

1st

Entry 2

High

score

28

Finally, we set the threshold. We judge entries as harmful than threshold, or it below non-harmful. These are the method of Category Relevance Maximization.

Test dataset - study Goal Find the actual amount of harmful entries on informal school websites

Data 3 random informal school websites

Time 2012/01/27~2012/01/30

29

Next, I will explain about preliminary study to verify the usability of the proposed method. To evaluate our method in conditions close to reality, it is necessary to create test datas with harmful and non-harmful entries mixed at an appropriate rate. Therefore, we performed a preliminary study how much of the entries are harmful on actual websites.

Test dataset - study

Harmful entries = 12% 30

We counted the harmful entries mixing ration on 3 informal school websites. From the results, we found that about 12% of harmful entries are contained on informal school websites.

Preparation of test data Data sets ・500 entries (5 set) Including 12% of harmful entries each set Harmful entries 12%

Data sets

Baseline (Matsuba) test data

500en.

500en.

500en.

500en.

500en.

31

Based on the preliminary study, we had created test data 5 set that include 12% of harmful entries. The entries including each data sets are extracted from baseline test data.

Preparation of test data Data sets ・500 entries (5 set) Including 12% of harmful entries each set

Comparison experiment ・2998 entries (Matsuba used) Including 50% of harmful entries

32

And, to compare the performance with baseline, we also used the test data that used in baseline. Using these data sets, we verified and compared the performance of baseline and proposed method.

Results

33

Experiment result is like this.

Results

34

Above red and blue lines is the performance of proposed method and baseline when used dataset including 50% of harmful entries.

Results

35

Below red and blue lines is the performance of proposed method and baseline when used dataset including 12% of harmful entries. From the results, the proposed method showed high performance than baseline in both datasets. なんで機会学習で閾値を設定しないの？ →実験回数が少なくてデータも少ないから機会学習は難しい．けど今後やりたい →前はSVMでやったけどあんまりいい性能が出なかったから別の方法を模索してるから

Discussion Entries with high scores Harmful entry ・うざいきもいブス (I hate that ugly b*tch) ・アトピーのやつ死ねよ (gotta kill that freak with atopy)

うざい - ブス (Hate - b*tch) アトピー - 死ね (Atopy - kill)

36

Next, I will discussion of the results. First, we investigated entries with high score. Then, these entries are calculated high score for phrases like these. Because, these phrases included seed words, this most probably increased the score of the harmful entry.

Discussion Entries with high scores Harmful entry ・うざいきもいブス (I hate that ugly b*tch) ・アトピーのやつ死ねよ (gotta kill that freak with atopy)

うざい - ブス (Hate - b*tch) アトピー - 死ね (Atopy - kill)

Non-harm. ・県外に住んでいる (I live outside of the prefecture)

外 - 住ん (Outside - live) 37

But, this entry also calculated high score for phrases like these, even though Non-harmful entry.

Discussion Entries with high scores Harmful entry うざいブス Some phrases are used both in ・うざいきもいブス (Hate - b*tch) (I hate that ugly b*tch) harmful and non-harmful アトピー - 死ね・アトピーのやつ死ねよ (gotta kill that freak with atopy) entries (ambiguous) (Atopy - kill) Non-harm. ・県外に住んでいる (I live outside of the prefecture)

外 - 住ん (Outside - live) 38

Because, this is a ambiguous phrase, and appears in non-harmful entries as well as in harmful entries. This phrase appears in cases of exposing personal information about where a person lives. Therefore, the score containing such a phrase is high.

Discussion Could use non-harmful polarity words to disambiguate such cases Check which words/phrases co-occur most often in non-harmful entries

39

As a countermeasure, it could be considered to register some words, which appear only in non-harmful entries, like “splendid”. We call these words as non-harmful polarity words. Non-harmful polarity words are appear in non-harmful entries, but not appear in harmful entries.

Discussion Image

Nonharmful polarity words

Phrase1 Phrase2 Phrase3

Harmful polarity words

40

For example, we calculate relevance of phrases with both non-harmful polarity words and harmful polarity words.

Discussion Image Non-harmful Nonharmful polarity words

Phrase1

Ambiguous Phrase2

Harmful

Phrase3

Harmful polarity words

41

In such cases, the phrase 1 could be considered as non-harmful. Because the relevance of the phrase with non-harmful polarity words is higher than harmful polarity words. Similarly, phrase 3 could be considered as harmful. And, the phrase 2 could be considered as ambiguous. Because the relevance of the phrase with non-harmful polarity words and harmful polarity words is much the same. This could reduce the influence of neutral phrases on the overall performance.

Discussion Entries with low score Ex.：北見工大の新田 (Nitta of Kitami Inst. of Tech.) Entries revealing personal information such as names of a person or school.

At the moment these have low relevance 42

And, we also investigated entries with low score. There are harmful entries which contain personal information such as school names or person’s names, like this. Because, the relevance of personal information with harmful polarity words is low.

Discussion Entries with low score Ex.：北見工大の新田 (Nitta of Kitami Inst. of Tech.) Entries revealing personal information such as names of a person or school.

Could gather words used for describing private information and implement a method for spotting it. 43

To solve this problem, we plan to register some words which have high relevance with personal information, and use them in the score calculation. 個人情報はどんなのをとろうとしている？ →個人を特定できそうなもの（電話番号，住所）

Conclusions PMI-IR based method ・checked the amount of cyberbullying in reality ・tested methods on such data

What influenced the results: Ambiguous phrases Entries containing personal information 44

Conclusions. In this study, we proposed a method of category relevance maximization to detect cyberbullying entries on the Internet automatically. In order to verify the usability of the proposed method, we evaluated the performance for the test data containing similar percentage of harmful entries as in reality. The experiment results showed that the proposed method is high performance than baseline. But, harmful entries contain personal information and neutral phrases are low score.

Future work ・perform a study in non-harmful words ・apply in processing private names ・determine threshold by machine learning

45

In the near future, we plan to register non-harmful polarity words. And we will also investigate a method for assessing the harmfulness score to entries including personal information. Furthermore, we plan to increase the data set, and determine the optimal threshold automatically by using machine learning.