A Fully Automated Approach for Arabic Slang Lexicon Extraction from Microblogs Hady ElSahar and Samhaa R. El-Beltagy Center of Informatics Sciences, Nile University, Cairo, Egypt
[email protected],
[email protected]
Abstract. With the rapid increase in the volume of Arabic opinionated posts on different social media forums, comes an increased demand for Arabic sentiment analysis tools and resources. Social media posts, especially those made by the younger generation, are usually written using colloquial Arabic and include a lot of slang, many of which evolves over time. While some work has been carried out to build modern standard Arabic sentiment lexicons, these need to be supplemented with dialectical terms and continuously updated with slang. This paper proposes a fully automated approach for building a dialectical/slang subjectivity lexicon for use in Arabic Sentiment analysis using lexico-syntactic patterns. Since existing Arabic part of speech taggers and other morphological resources have been found to handle colloquial Arabic very poorly, the presented approach does not employ any such tools, allowing the presented approach to generalize across dialects with some minor modifications. Results of experiments, that targeted Egyptian Arabic, show the approach’s ability to detect subjective internet slang represented by single words or by multi-word expressions, as well as classifying the polarity of these with a high degree of precision.
1
Introduction
Recent years, have seen an enormous increase in the use of microblogging services and social media sites across the world and especially in the Arab World. A study prepared and published by Semiocast in 2012 has revealed that Arabic was the fastest growing language on Twitter in 2011, and was the 6th most used language on Twitter in 2012 [1]. A recent breakdown of Facebook(FB) users by country, places Egypt with 16 million users as the Arabic speaking country with the largest number of FB users, and ranks it at 17 among all countries of the world [2]. This represents a growth of 41% in terms of users, from the previous year [2]. Both Twitter and Facebook are characterized by having a high percentage of highly opinionated posts. The presence of such a large volume of opinionated data highlights the need for sentiment analysis tools which can make use of these whether in marketing, politics or other areas. Since slang and dialectical expressions are very commonly used for expressing sentiments and opinions on social media, augmenting sentiment lexicons with slang terms and expressions can directly impact the sentiment analysis process. The aim of this work is to present an approach for capturing dialectical or slang terms or compound expressions that are highly indicative of subjectivity as well as assigning polarity to these. The rest of this paper is organized as follows: section 2, describes the
different aspects of the problem that this works aims to address, section 3 briefly reviews related work, section 4 provides an overview of the proposed system, section 5 presents the experiments carried out to evaluate the work and their results, and finally section 6 concludes this paper.
2
Problem statement
Sentiment analysis of Arabic social media is a challenging task not only because social media language is rich with colloquial Arabic, compound terms and idioms as well as a lack of resources as detailed in [3], but also because the language used in social media, and twitter in particular, has been shown to be of a highly dynamic and evolving nature [4]. Creative expressions that imply subjectivity are often created on the fly by popular tweeps (twitter users) and then quickly propagated and widely employed by other social media users; they then become strong subjective clauses. Subjective terms often emerge from peculiar exchanges observed on TV shows, advertisements, or trending YouTube videos. For example, the word “”ھيييح, which the presented work was able to learn and which has no meaning in colloquial or MSA Arabic, is now commonly used to indicate a positive sentiment. The word is the written form of a sigh made by a famous Egyptian TV presenter in a positive context. Political situations and public figures are yet another source of inspiration for the creation of new expressions the usage of which indicates high subjectivity. For example, the adjective ““( ”نكسجيsomeone who creates setbacks”) which was created after the 30th of June events in Egypt, is used as a negative reference to a set of people with a specific political opinion. Also the term ““( ”بتاع االستبنaffiliated to the spare tire”), which doesn’t make much sense unless understood in context, is a negative term referring to those who support former Egyptian president Muhammad Morsi. The term “ ”االستبنwhich means “the spare tire” was widely used for former president Muhammad Morsi during the 2012th presidential elections, by those who opposed him. Another problem that this work tries to address is the wide use of transliterated English to reflect sentiment. For example, the presented work is capable of picking up words like “ ”كيوتand “ ”قيوطboth of which are Arabic transliterations of the English word “cute” and which reflect a positive sentiment. “ ”اوفرis another example of a commonly used transliterated English term, which is “over”, and which is used in the social media context to indicate “exaggeration” or to something that is over the top. The presented work also tries to capture subjective slang and dialectical expressions that are well established within a certain culture or country. The goal of this work is not only to try to detect slang and informality in a fully automatic way but to assign polarity to extracted terms. It’s important to re-iterate that the terms of interest can be single words or compound phrases.
3
Related work
There hasn’t been much work that directly targets slang detection, let alone assigning polarity to slang expressions. However, if we consider slang as a special case of lexicon learning, then a lot of work becomes relevant. Research has been carried out to address the question of how to build a polarity lexicon for many languages including but not limited to German, Dutch, Spanish, Chinese, and Japanese, but most notably, English. This section will focus primarily on relevant work carried out for Arabic, which is the focus of this work, and on English where most efforts have been condensed, as well as on approaches that claim language independence. Turney [5] proposed an unsupervised algorithm for inferring the polarity of phrases that have adjectives and adverbs within a POS-tagged corpus. Polarity is calculated based on Pointwise Mutual Information between unknown phrases and the words “excellent and poor”. The approach suffers from several drawbacks if considered for application within an Arabic social media context: 1. it relies on the existence of a huge POS-tagged corpus, which in the case of Arabic social media is particularly challenging as the language used within this media, and especially in microblogs, is highly unstructured and a POS tagger that can actually work on these with any acceptable degree of accuracy, is yet to be developed. 2. The approach only targets adjectives and adverbs. Banea el al [6] presented an approach which they claim can work for building subjective lexicons for languages with scarce resources. The approach requires a small seed of subjective words, and with the aid of a dictionary, uses those to generate a set of other candidate subjective terms. The candidate terms are ranked using Latent Semantic Analysis as a similarity measure between a candidate and the original seed. The approach was applied on Romanian. Based on the description provided for the approach it can be deduced that the approach is incapable of handling slang, which is very common in a social media settings as slang can rarely be found in language dictionaries, and completely ignores multiword expressions, which are very commonly used in Arabic to convey sentiment. Abdul-Mageed & Diab [7] proposed an approach for building a large scale Arabic sentiment lexicon. To do so, they expand on a Modern standard Arabic (MSA) polarity lexicon of 3225 adjectives which was built manually, by using a number of existing English lexicons including SentiWordNet [8]. The authors report having problems with both the coverage and the quality of some of the entries. They also state that they have not tested the system for the task of sentiment analysis. But even without the reported limitations, this approach will still be incapable of encompassing slang, dialectical Arabic, and multiword expressions. Velikovich & Blair-Goldensohn [9] proposed yet another approach for building a polarity lexicon using graph propagation over a phrase similarity graph built using 4 billion unlabeled web documents and a set of seed terms. Results of experiments car-
ried out by the authors show that the derived lexicon improves the accuracy for the sentence polarity classification task. The advantage of using this method is that it is capable of learning slang and multi-word expressions. It is not clear however, how well it will perform if the graph is built using a smaller corpus. Volkova et al. [4], proposed an approach for bootstrapping subjectivity clues from Twitter without relying on language-dependent tools. To do so, strong subjective terms were selected from the MPQA lexicon (the number of which was not specified) and used to annotate a set of 1M Tweets in English. To show that the presented approach can generalize across languages, the selected seed terms were also translated to Russian and Spanish using bilingual dictionaries. Translations were used to annotate a set of Tweets for corresponding languages. The polarity of new terms was determined based on the probability of the term appearing in positive or negative tweets. It was not clear how objective terms were handled. For this approach to work for other languages, either the target language must have a large lexicon that can be used as a seed, or a bilingual dictionary that can be used to translate from the original English seed lexicon to the target language must exist. The work does not address idioms and multi-word expressions.
4
The Proposed Approach
As stated before, the goal of this work is to present an approach capable of detecting commonly used dialectical or slang terms that reflect subjectivity. A term in the context of this work can be made up of a single word or multiple words. Furthermore, the work aims to classify the polarity of detected terms. So, the presented approach is a two phase one. In the first phase, candidate subjectivity terms are detected, and in the second phase, they are classified. For detecting highly subjective words/expression, we have identified a set of lexico-syntactic patterns indicative of subjectivity. The use of handcrafted patterns is not a novel idea in the context of information extraction, for example, Hearst [10] has used such patterns for acquiring hyponyms from large text corpora, while Klaussner and Zhekova [11] have used them for ontology learning. The tags used within the proposed patterns do not require the use of a part of speech tagger; in fact each tag has a list of finite possible values that is dialect dependant. So while the presented patterns themselves, are mostly dialect independent, the range of possible values from which tags in a pattern can be derived, depends on the dialect that is being targeted. Our work and experiments have focused on Egyptian Arabic; and more specifically, the Cairene dialect. In the first phase, the extraction patterns are applied on a large corpus of text obtained from twitter to yield a set of subjective terms. In the second phase, the extracted terms, are assigned polarity based on the normalized point wise mutual information
score between them and positive and negative terms derived from an existing polarity lexicon. Details of the process are presented in the following subsections. Figure 1 summarizes the entire process. 4.1
Extraction Patterns
When identifying extraction patterns, we had to make sure that they satisfy the following set of constraints: 1) patterns shouldn’t have to rely on any part of speech taggers or parsers 2) defined patterns should be as general as possible so as to increase recall, 3) patterns should exhibit a very high rate of accuracy so as to be able to learn subjectivity in a totally automated way. The last point requires individual evaluation of identified patterns.
Fig. 1. Overview of the Lexicon Extraction System
Table 1 shows the tags that we have employed in our extraction patterns. Each tag is accompanied by a brief description and a set of Arabic and English examples as well as the count of the tokens that were used as a value pool for that tag. While most tags are fairly standard, the “Person Reference” [PR], “Strong subjective” [SS], and “Subjective Expression” {SE} are not. The [PR] tag refers to any word that can be made to reference a human being. Egyptian dialectical terms that do so include “”راجل (man), “ ”بتand “ ”بنتboth of which correspond to “young lady” the former being a rather vulgar form for the term, “( ”بني ادمhuman being), “( ”كائنcreature) and “”انسان (human).The [SS] tag refers to a set of strong subjective positive and negative terms generalized to their different masculine, feminine and plural forms. As can be seen from the table, only a very small seed of 25 terms was used. Terms that are learned at the end of our extraction process, can be made to augment this list, to relearn other terms iteratively. The {SE} tag, is simply a place holder for the expression that will be extracted from any pattern Since our work is mainly concerned with slang, tag lists were built to match with slang rather than with Modern Standard Arabic (MSA). For example in the list of “Demonstrative Pronouns”, terms such as “ ”دهwhich corresponds to masculine “this” and “ ”ديwhich is the feminine form of "this" as well as “ ”دولwhich means “those” were included. The same was applied to all other lists. The slang intensifiers list for example included “”خالص, “”موت, “ ”السنينand “ ”بغباوهwhich all corresponds to “a lot”, and affirmations like “ ”صحيحand “ ”بجدwhich mean “indeed”. The personal
pronouns list included only personal prounouns that can be used for addressing someone rather than all personal pronouns. Table 1. Tags used in extraction patterns
Tag
Description
[Neg] [DP] [Ints] [PR]
Negator Demonstrative Pronoun Intensifier Person Reference
[PP] [Conj] [SS] {SE}
Arabic Examples ال, مش,اليمكن ده,دي اوي, طحن,جدا راجل, إنسان,ست
Personal Pronoun يا,إنت Conjunction او,و Strong subjective حقير,جميل Subjective Expression to be extracted
English Examples Not This, that Very much Person, man, woman You And, or Pretty, vile
Count 46 15 40 40 5 3 25 N/A
The initial set of candidate patterns is shown in Table 2. As can be seen from the table, specifically from examples and their translations, the structure of Arabic sentences is quite different from English ones. For example, in English, intensifiers often precede the terms they intensify, while in Arabic they often follow them. So, for pattern 1 to work in English for example, it should become [DP] [PR] [Ints] {SE} and {SE} should be limited to a single word to avoid noise. A literal translation of the matched examples in the table would have not made much sense, so instead we include as an accurate a translation as possible (which does not always correspond to the Arabic structure) and highlight English intensifiers with italics. Table 2. Initial set of candidate patterns with examples
ID P1
Pattern [DP] [PR] {SE} [Ints]
P2
“{ ” ايه الSE} [DP]
P3
“[”الPR] [DP] {SE} [Ints]
P4
“[”الPR] [DP] {SE} [Conj] [SS]
P5
“{ ”اما انكSE} [Ints]
Example of a Match ده عيل }مستفز{ بغباوه This boy is incredibly {irritating} ايه ال}شياكه{ دي what {elegance } this is الراجل ده }حظه وحش{ جدا This man has incredibly {bad luck} الست دي } مبدعه دائما { و فنانه This lady is {always creative} and artistic اما انك }فرفوور{ صحيح You are such a {wimp} indeed
ID P6
Pattern [PP] [SS] [PP] {SE} [PP]
Example of a Match ... يا رائع ياللي }بتفتح النفس{ ياللي You wonderful you who {motivates} who…
P7
[PR] {SE} [Conj] [SS]
راجل }صاحب مبادئ{ و محترم A man {with principles}and respectable
P8
[SS] [Conj] {SE} [Ints]
محترم و }مؤدب{ جدا Respectable and very{polite}
P9
[PP] “{ ”المؤاخذهSE}
P10
[DP] “{ ”حتتSE}
{ دي حتت }سكره She’s a piece of {Sugar}
P11
“{ ”راجل“ ”دهSE}
{ده راجل } مش متربي This man {has no manners}
Excluded Patterns {انت المؤاخذه }غبي you, excuse me, are {dumb}
The set of patterns listed in Table 2, are based on observing how people commonly express sentiments about people or things in Arabic. Before carrying out any rigorous evaluation, each of these patterns was used to extract a limited set of subjective terms from a subset of the used twitter corpus (described in details in the next section). A quick examination revealed that patterns P9, P10 and P11, often result in noisy matches. For example, one of the expressions captured by pattern P9 was “ سكره ايه اللي ”حصلwhich translates to “sugar what happened”. While “sugar” is indeed a subjective term, the rest of the expression “what happened” is simply noise. What all three patterns have in common is being open ended, meaning that the captured slang expression can match with any words that follow the initial pattern with no constraints. This has the potential of introducing too much noise and should be avoided which is why these patterns were omitted from the final pattern set. A thorough examination of how well the remaining patterns perform is presented in section 5. All remaining patterns, except for patterns P2 and P5 are dialect independent. To adapt those to other dialects, the text hardwired in these patterns needs to be translated to the target dialect. The intensifier at the end of pattern 1, and the personal pronoun at the end of pattern 3, simply ensure that the extracted term is of high subjectivity and that multi-word sentiments are extracted as {SE} can match with one or more words. Patterns P4, P6, P7 and P8 capitalize on that fact positive or negative terms usually occur together by using of conjunctions or using multiple “Personal Pointers” [PP].
4.2
Polarity Classification
After extracting candidate subjective terms, there is a need for assigning polarity to these. In order to do so, this work proposes the use of co-occurrence statistics between each of candidate term and known positive and negative terms in a large collection of microblogs or tweets. The reason a microblog or tweet is favored as a unit of information instead of a document or a longer posting, is that the short nature of tweets increases the likelihood that only sentiments with similar polarity will cooccur in a single tweet. Tweets are also rich with slang. By tagging individual tweets in a large corpus using an existing sentiment lexicon, co-occurrence can be calculated using normalized point wise mutual information [12] as represented by equation (1), where represents the candidate subjective term and is the polarity class which can be positive or negative . , =
5
ln
,
− ln ,
1
Experiments and Results
To determine whether or not the proposed approach is capable of achieving its goals, the proposed patterns had to be applied on a large corpus in order to extract terms. The extracted terms then had to be annotated with polarity. Both the system’s ability to extract subjectivity and slang and its ability to assign polarity were individually evaluated. Section 5.1 describes the dataset used in the experiments, while sections 5.2 and 5.3 describe the experiments carried out to evaluate the system’s ability to detect subjectivity or slang and its ability to assign polarity respectively. 5.1
The Used Data Set
To build the corpus on which to apply the patterns described in the previous section, a set of approximately 11 million Arabic tweets was collected using the twitter API [13] and a set of trending hashtags as search terms. A preprocessing step was then made on individual tweets to remove unwanted features and noise. The process included: 1) Removal of hyperlinks 2) Removal of the hash letter ‘#’ to capture subjective text in hash tags if available and also the replacement of underscores by spaces in hash tags to convert them into regular words 3) Text normalization according to the rules described in [14] as Arabic speakers usually mix between characters like ""يand " "ىor " "أand " "اor " "هand ""ة 4) Removal of redundant tweets, usually resulting from re-tweets. Instead of checking for exact matches, the cosine similarity function [15] with a threshold value of 0.7 was used. This guarantees uniqueness of instances in the dataset and ex-
cludes quoted re-tweets that may give a false indication for the high occurrence of a specific term. The pre-processing step reduced the size of the corpus to 7.5M unique normalized tweets. 5.2
Evaluation of Slang Extraction
To evaluate the process of detecting and extracting slang terms, the proposed set of eight predefined patterns was applied to the dataset consisting of 7.5M preprocessed tweets. The pattern matching process resulted in the extraction of a set of 633 unique terms. Three graduate students, who are also Arabic native speakers, were asked to manually annotate the set of 633 terms extracted from the pattern matching process using positive, negative, or not-a-sentiment tags. Each term was assigned the tag which had the most consensuses from the three judges. In addition, out of dictionary terms were labeled as Slang, swear terms and profanity were labeled as Profanity, and multi-word terms were labeled as Compound. The tagged dataset was then treated as a ground truth against which obtained results were compared. For each of the eight predefined patterns subjectivity precision was measured by calculating the percentage of terms that were assigned a subjective label (positive or negative) from the total number of extracted terms. Table 3 summarizes the results. Table 3. Summary of patterns and their corresponding statistis # of extracted terms
P1 P2 P3 P4 P5 P6 P7 P8 Total
144 14 135 4 21 222 123 98 633
Profanity 0.23 0.43 0.24 0 0.86 0.53 0.33 0.1 0.32
Slang 0.56 0.93 0.7 0.5 0.91 0.59 0.49 0.44 0.6
Compound 0.23 0.14 0.27 0 0.48 0.24 0.19 0.27 0.29
Precision 0.92 1 0.81 1 1 0.9 0.93 0.84 0.89
By analyzing the output of the different patterns, as reflected by Table 3, we find that P6 which was based on the detection of multiple personal pointers [PP] showed relatively higher performance than the rest of the patterns with 222 extracted terms and 90% precision. On the other hand P4, which is considered to be a mixed pattern, was able to extract only 4 terms due to having relatively more restrictions. Having said that, the count of the extracted terms would have been significantly higher if we could have overcome the limitation of the Twitter API by querying it directly for
tweets that match our patterns instead of querying over a downloaded corpus. The overall pattern matching technique resulted in a total precision of 88.6%. Some examples of extracted terms are shown in Table 4. The examples illustrate the system’s ability to detect slang terms (both established and newly evolved) as well as subjective expressions. For example the term “( ”امنجيsocial security secret agent), extracted by pattern 3, is considered a relatively new term with a negative connotation, as it is often used to refer to squealers. The expression “”دمه خفيف1 (has a good sense of humor), picked up by pattern 7, is a well established expression that refers to funny people and is usually used to reflect a positive sentiment. In isolation, the individual words that make up this expression carry no sentiment. The same applies for the expression “( ”تفتح النفسwhets the appetite), also picked up by the same pattern. The fact that the system was capable of detecting these compound expressions, shows that it can add value to existing sentiment lexicons. Some of the detected words, are terms transliterated from other languages like ““( ”كيوتcute”) and ““( ”برنسprince”). It’s also interesting to see that the system has picked up the English word “cool” (pattern 1) as it is often used in its original English form within Arabic text, to mean cool. Table 4. Examples of terms that were detected for each pattern
P1
Examples ذو قيمه, cool
Respective translations Of value, Cool
P2 P3
السع, مشكوك فيه اتشحور, امنجي, ھييييييح
P4 P5 P6
برنس, ھتموت اعمي القلب والنظر, برجوازي متسلق ابن الخروفه, اشطه, ارھابي, بتاع االستبن الطرطور, شاغل بالي
P7 P8
تغريداته مفيده, دمه خفيف, قميل احساسه عالي, تفتح النفس, كيوت
Insane, Suspicious Was viciously beaten, Security state agent, sigh Prince, You will die Sightless, Social climber Son of a sheep, Fine, Terrorist Affiliated to the spare tire the clown on my mind Posts useful tweets, Funny, Beautiful Is very sensitive, Motivating, Cute
5.3
Evaluation of Polarity Assignment
A seed lexicon Lseed of 2K strongly subjective terms was selected from the Egyptian dialect lexicon Ltotal created by El-Beltagy and Ali [16][3] and was used for automatically labeling the preprocessed tweets dataset. Each tweet in the tweets dataset was considered to have positive polarity if it contained one or more positive terms from our seed lexicon and was considered to have negative polarity if it contained one or more negative terms from the seed lexicon. Tweets containing both positive and negative terms were excluded from our calculations and so were tweets containing 1
The literal translation of this tern is “has light blood”
negations and disjunctive conjunctions. Handling negations could have been done by converting subjective words preceded by a negation word to the opposite polarity but this method always leaves some unhandled noise especially when the negation word does not directly precede the subjective word. Since disjunctive conjunctions such as “( ”بسbut) and “( ”لكنhowever) usually reverse the polarity of subsequent terms, tweets containing those, were also omitted. The annotation process using the selected seed lexicon resulted in 1.3M positively annotated tweets and 1.5M negatively labeled ones. To examine how the size of the lexicon affects the results, the same tweet dataset was annotated with the entire lexicon developed by El-Beltagy and Ali [3]. Table 5 reports the label specific statistics resulting from annotating the tweets dump using both Lseed and Ltotal. Table 5. Labels resulting from the tagging process
Lseed Ltotal
POS=1 1M 1.18M
NEG=1 1.17M 1.43M
POS ≥ 2 0.3M 0.4M
NEG≥2 0.35M 0.57M
Mixed 0.45M 0.76M
Neutral 4M 3M
Total 7.5M 7.5M
Following the tagging process, [12] was used to measure the co-occurrence of a term with positive and negative labeled tweets in the tagged dataset using equation (1). For each term in the selected candidate terms, not in Lseed and that occurs at least Ω times (we used Ω =10) in the labeled dataset we calculate it’s with the positive labeled tweets , and also with negative labeled tweets , . Polarity is then assigned according to the nPmi value which has the highest value and that has an absolute difference α of a value greater than a specific threshold θ, where α is calculated using equation 2. α = | , − , |
2
α is thus used as an indication of the confidence of the process of polarity assignment. Terms that have a value of α < θ, are simply ignored. So selecting a relatively high θ value will affect the overall recall by excluding terms with relatively lower confidence. Fig. 2 shows how precision of the results is affected by the changes in the confidence threshold α where precision is calculated by comparing the terms with assigned polarity against the polarity of their counterparts in the ground truth dataset described in the previous section. The value of θ was set to equal 0.001, which resulted in a total precision of 84.5% with 344 terms out the total 377 labeled correctly. The new terms were then added to the original 2K lexicon Lseed. This addition constitutes an increase of 18.85% of the size of the lexicon. To keep a lexicon updated with new terms, the process of building a twitter corpus and applying patterns to it, should be done periodically.
Fig. 2. Graph showing precision against difference in nPmi (α)
6
Conclusion and Future work
In this paper we presented an approach for detecting subjective slang terms with the aim of building slang Arabic lexicons without depending on any languagedependent part of speech taggers or parsers. The proposed technique showed through experimentation, an ability to deal with informality which was illustrated by the precentages of extracted slang, profanity and compound-terms. The approach also showed that it was able to deal with the evolving nature of the language used in social media by picking up slang subjective terms that only trended very recently. In the future, we plan on measuring the effect of augmenting a slang lexicon to one or more existing lexicons, on the task of sentiment analysis. Future work will also target extending the number of extracted terms by both increasing the number of defined patterns, and continuously feeding our system with recent tweets from which to extract terms. We are also looking into ways for increasing the precision of the obtained results without adversely affecting the recall. Automatically detecting extraction patterns, rather than having handcrafted ones, is yet another area of future work. We would also like to assign positive and negative weights to extracted terms rather than absolute polarity as many subjective terms can be used in positive, negative or neutral contexts.
References 1. Semiocast, “Geolocation analysis of Twitter accounts and tweets by Semiocast,” 2012. [Online]. Available: http://bit.ly/1kwY9OZ.
2. D. Farid, “Egypt has the largest number of Facebook users in the Arab world,” Daily News Egypt, Sep-2013. 3. S. R. El-Beltagy and A. Ali, “Open Issues in the Sentiment Analysis of Arabic Social Media : A Case Study,” in in proceedings of 9th International Conference on Innovations in Information Technology (IIT), 2013, pp. 215 – 220. 4. S. Volkova, T. Wilson, and D. Yarowsky, “Exploring Sentiment in Social Media : Bootstrapping Subjectivity Clues from Multilingual Twitter Streams,” in Proceedings of the 51st Annual Meeting of the Association for Computational Linguistics, 2013, pp. 505–510. 5. P. Turney, “Thumbs up or thumbs down?: semantic orientation applied to unsupervised classification of reviews,” in Proceedings of the 40th Annual Meeting on Computational Linguistics (ACL), 2002, no. July, pp. 417–424. 6. C. Banea, R. Mihalcea, and J. Wiebe, “A Bootstrapping Method for Building Subjectivity Lexicons for Languages with Scarce Resources,” in Proceedings of the Sixth International Conference on Language Resources and Evaluation (LREC’08), 2008, pp. 215 – 220. 7. M. Abdul-Mageed and M. Diab, “Toward Building a Large-Scale Arabic Sentiment Lexicon,” in Proceedings of the 6th International Global WordNet Conference., 2012, pp. 18–22. 8. A. Esuli and F. Sebastiani, “Determining the semantic orientation of terms through gloss classification,” in Proceedings of the 14th ACM international conference on Information and knowledge management, 2005, pp. 617–624. 9. L. Velikovich and S. Blair-Goldensohn, “The viability of web-derived polarity lexicons,” in Proceedings of The 11th Annual Conference of the North American Chapter of the Association for Computational Linguistics (ACL), 2010, pp. 777–785. 10. M. Hearst, “Automatic Acquisition of Hyponyms from Large Text Corpora,” in Proceedings of the 14th conference on Computational linguistics - Volume 2, COLING ’92, 1992, pp. 539–545. 11. C. Klaussner and D. Zhekova, “Lexico-Syntactic Patterns for Automatic Ontology Building,” in Proceedings of the Student Research Workshop associated with RANLP, 2011, pp. 109–114. 12. J. Xu and W. B. Croft, “Corpus-Based Stemming using Co-occurrence of Word Variants 1 Introduction,” ACM Trans. Inf. Syst., vol. 16, no. 1, pp. 61 – 81, 1998. 13. “Twitter REST API version 1.1.” [Online]. Available: https://dev.twitter.com/docs/api/1.1. 14. L. S. Larkey, L. Ballesteros, and M. E. Connell, “Light Stemming for Arabic Information Retrieval” in Arabic computational morphology., 2007, pp. 221–243. 15. A. Singhal, “Modern Information Retrieval : A Brief Overview” in Bulletin of the IEEE Computer Society Technical Committee on Data Engineering, 2001, pp. 35–43. 16. S. R. El-Beltagy and A. Ali, “unWeighted Opinion Mining Lexicon (Egyptian Arabic)” 2013. [Online]. Available: http://bit.ly/MGtMqU.
