Sentiment Classification by Sentence Level Semantic ... - ExcelingTech

539 Int. J Comp Sci. Emerging Tech

Vol-2 No 4 August, 2011

Sentiment Classification by Sentence Level Semantic Orientation using SentiWordNet from Online Reviews and Blogs Aurangzeb khan, Baharum Baharudin Department of Computer and Information Sciences, Universiti Teknologi PETRONAS Perak, Malaysia. [email protected], Abstract: Sentiment analysis is the procedure by which information is extracted from the opinions, appraisals and emotions of people in regards to entities, events and their attributes. In decision making, the opinions of others have a significant effect on customers ease in making choices regards to online shopping, choosing events, products, entities. In this paper, the rule based domain independent sentiment analysis method is proposed. The proposed method classifies subjective and objective sentences from reviews and blog comments. The semantic score of subjective sentences is extracted from SentiWordNet to calculate their polarity as positive, negative or neutral based on the contextual sentence structure. The results show the effectiveness of the proposed method and it outperforms the machine learning methods. The proposed method achieves an accuracy of 87% at the feedback level and 83% at the sentence level for comments and 97% at feedback and 86 % at sentences for customer reviews. Keywords: Sentiment Analysis, Classification. Blog Mining.

1

Opinion

Mining,

Introduction

Essential for online services, that exist today, is the abundance of applications for sentiment analysis. Mining customer reviews or gathering feedback from opinions about a given product (e.g., digital camera, car, mobile phone, etc.) can give companies valuable information as to the satisfaction or dissatisfaction of their customers. This information is also immensely valuable for customers in their decisions to purchase a particular product. Furthermore, sentiment analysis enables trend watchers, marketing research teams and recommendation systems to track emotions or opinions over time; tracking of online trends (mood) provides interesting as well as valuable data for these groups. In the case of moderating, opinion mining has also proven to be very useful whereby the ability to react quickly and efficiently to messages which have been posted on forums or discussion boards wherein dissatisfied consumers discuss product deficiencies. Moreover, being able to respond appropriately to any “heated debates” or “flame wars” that may be going on is also a benefit. This study focuses on sentiment analysis at the sentence level using lexical rule based method for different type data like (blogs, movie reviews, social communication networks e.g. twitter, and product reviews). KDT (knowledge discovery in texts) or text data mining or text mining are terms used for the mining of unstructured or semi-structured data. It is a

slightly new sub-discipline of data mining that considers textual data. The fact is that, "text data mining" is an intermediate evolutionary lexical form, [1].The Majority of the online information about data mining is “misleading”. Such ambiguous/misleading information implies to the mining metaphor that it is like “extracting precious nuggets of ore from otherwise worthless rock" as found in or finding "gold hidden in mountains of textual data" in (D orre et al. 1999).A more narrow definition of what text mining does is needed to distinguish it from the traditional IR, which is basically "information access" as argued by Hearst [1]. Retrieval of documents relevant to the information needs of a user, is the primary concern of the traditional IR (perhaps a more appropriate name would be data retrieval); however, the user is left on his/her own to find the desired information in the documents. In Hearst‟s opinion, data mining has not only directed dealings with the information, but it also attempts to uncover or glean previously unknown, information from the data (text). different linguistic levels (words, sentences and documents)highlighting the key differences between supervised machine learning methods, that rely on annotated corpora or corpusbased, and unsupervised/lexicon-based methods in sentiment classification. Three main steps are always involved in the process of text mining and sentiment classification; they are (a) acquiring texts which are relevant to the area of concern usually called IR; (b) presenting contents collected from these texts in a format that can be processed, such as statistical modelling, natural language processing, etc.; and (c) actually using the information in the presented format,[3] [4] communities [5] [6]. This is true for the management of news groups, maintenance of web directories and even emails. A good tutorial on hypertext mining can be found in [7]. Mining from services is also a growing area in the field of information extraction. This is greatly in part due to the staggering number of online services like Usenet, digital libraries, news groups, customer comments and reviews and mailing lists which are popping up all over the Web. It should be noted that, a very thin line separates Web content mining from IR; this is because there is a deep association of WCM with intelligent IR Mining. On the other hand, it is claimed by some that IR on the Web is a part of WCM. Two strategies are used for WCM; one is the direct mining of document content as in Web page content mining and the other is improving on the search for content


using tools like search engines as in search result mining. This makes the job of the search engine closely associated with WCM [8]. In this work, we proposed a domain independent rule based method for semantically classifying sentiment from online customer reviews and comments. The method is effective as it takes reviews, checks individual sentences and decides its semantic orientation considering the sentence structure and contextual dependency of each word. The rest of the paper is organized as follows: Section-2 presents the related research of the proposed work. Section-3 describes the proposed method with all steps. Section-4 highlights the results and finally Section-5 concludes the proposed method.

2

Background and Related Work

The early work of sentiment analysis began with subjectivity detection, dating back to the late 1990‟s. Later, it shifted its focus towards the interpretation of metaphors, point of views, narrations, affects, evidentiality in text and other related areas. Shown below is the literature describing the early works of subjectivity and detection of affects in the text. With the increase in internet usage, the Web became a source of importance as text repositories. Consequently, a switch was slowly made away from the use of subjectivity analysis and towards the use of sentiment analysis of the Web content. Sentiment analysis has now become the dominant approach used for extracting sentiment and appraisals from online sources. Separating non-opinionated, neutral and objective sentences and texts from subjective sentences carrying heavy sentiments is a very difficult job; however, it has been explored earnestly in a closely related yet separate field, (J. M. Wiebe, 1994). It concentrates on making a distinction between 'subjective' and 'objective' words and texts; on one hand, the subjective ones give evaluations and opinions and on the other, the objective ones are used to present information which is factual (J. Wiebe, Wilson, R. Bruce, Bell, & Martin, 2004) (J. Wiebe & Riloff, 2005). This is different than sentiment analysis in regards to the set of categories into which language units are classified by each of these two analyses. Subjectivity analysis focuses on dividing language units into two categories: objective and subjective, whereas sentiment analysis attempts to divide the language units into three categories; negative, positive and neutral. The area of concentration in some of the early works was with subjectivity detection only (J. M. Wiebe, 2000). With the passage of time and a need for better understanding and extraction, momentum slowly increased towards sentiment classification and semantic orientation. Like other developing fields of research today, sentiment analysis terminology is yet to be matured; moreover, just attempting to define a sentiment can be difficult to accomplish [13]. The words sentiment [14][15], polarity [16] [12] [17], opinion [18], [19],


semantic orientation [12] [20], attitude [21] and valence [22] are used to represent similar if not the same ideas. These words are, more often than not, used either to make reference to various aspects of one particular phenomenon, an example being [23] [15] where sentiment is defined as an affective part of opinion, or simply used as synonyms for each other without any true definition of their own. Furthermore, some of these words can be confusing because of their multiple meanings already in linguistic tradition (e.g. polarity, valence) and therefore are confusing. For our present work, the focus is on capturing expressed sentiment in a text as negative, positive or neutral; therefore, we will refer this domain of research as sentiment analysis. Our preference of using the term “sentiment analysis” is due to the fact that: (1) the possibility of confusing this work with research in other areas is not likely because the term is not associated with any other research tradition, (2) the kind of data which is extracted from the texts is accurately reflected (unlike in the case of opinions which could also possess a topical component), and (3) it is parsimonious and precise [24] [25]. Recently, there has been a change of attitude in the field of sentiment analysis whereby the concentration is now on classification, which has added a third category known as neutrals [26], [15]. Therefore, it is no longer focused on the binary classification of only positive/negative [20]. Through empirical observations, there came a realization that it is much easier to separate positive elements from negative ones than it is to differentiate positives or negatives from neutrals. Majority of disagreements amongst human annotators as well as the errors resulting from utilizing automatic systems are associated with attempting to separate neutral words, sentences or texts from those that are either negative or positive [15] . Moreover, a problem arises from the meaning attributed to the term "neutrals”. This is because "lack of opinion" [27] as well as "a sentiment that lies between positive and negative" [13] are both meanings of "Neutrality" used in related literature. The latter definition is favoured by sentiment analysis while the field of subjectivity analysis mostly uses the former interpretation. However, it is the latter meaning of the word that will be utilized in this dissertation [24] [25]. A rating inference as a metric-labelling problem was developed by [28]. They achieved this by first applying two n-array classifiers, which included onevs.-all (OVA) SVM and SVM regression, in order to classify reviews in regards to multi-point rating scales. After applying the classifiers, a metriclabelling algorithm was utilized so that the results of the n-array classifiers were completely changed in order to guarantee that like items receive like labels. A similarity function was determined from this. It is true that a typically used similarity function in topic classification is the overlapping of terms; however, when attempting to identify reviews having like ratings, it is not particularly effective [28]. The Positive Sentence Percentage (PSP) similarity



function was subsequently introduced which calculates the number of sentences which are considered positive divided by the number of sentences in a review that are considered to be subjective. Results of experiments generally have shown an improvement in n-ary classifier performance when making use of metric-labelling with PSP. Pang and Lee‟s work was later augmented by [29] where they used transductive semi-supervised learning in their study. It was shown that classification accuracy could be improved upon with the help of reviews without user-specified ratings, in other words, unlabelled reviews [24]. A kernel regression algorithm which was introduced by [30] 2007, made use of order preferences of unlabelled data and was successfully applied to sentiment classification. The order preference of a pair of unlabeled data, x and x , i

j

indicates that x is preferred to x to some degree, even i

j

though the exact preferences for x and x are i

j

unknown. For example, in the framework of sentiment analysis, when presented with two reviews of unknown rating values, it is quite possible to determine which review is more positive. They executed their algorithm with the rating inference problem. As a result, it was evidenced that by utilizing order preferences the performance of rating inference was much better than standard regression [24] [25]. Corpus-based machine learning method or methods based on compilations are able to compile lists of negative and positive words with a high accuracy of up to 95%. However, in order to reach their full potential, most of these approaches need immense annotated training datasets. Corpus-based methods can overcome some of these limitations by utilizing dictionary-based approaches since these approaches depend on existing lexicographical resources (such as WordNet) to provide semantic data in regards to individual senses and words [31] [24] [25]. [32] suggested that when analysing sentiment, semantical similarity does not necessarily imply sentimental similarity. This suggestion was made on the basis of statistical observations from a compilation of movie reviews. Subsequently, a method for determining the semantic orientation of an opinion is proposed on the basis of relative frequency. An estimation of the opinion strength of a word and the semantic orientation in regards to a sentiment class and its relative frequency of appearance in that class is carried out using this method. For example, if the word “best” appeared 8 times in Positive reviews and 2 times in Negative reviews, its strength with respect to Positive semantic orientation is then 8/(8+2) = 0.8 [24]. Introduction of new features, that are conceptually related to keyphrase-frequency were done in [33]. On the basis of the candidate phrases in the input document, these new features can be generated, by issuing queries to a Web search engine. An improvement in keyphrase extraction has been

experienced with these new although they are neither domain specific nor training-intensive. The feature values are calculated from the number of hits for the queries (the number of matching Web pages). A large collection of unlabeled data, approximately 350 million Web pages without manually assigned key phrases, has been mined for lexical knowledge to derive these new features. Simple methods for combining individual sentiments [34] and supervised [35] statistical techniques were proposed which can measure sentiment on the phrase or sentence level using opinion oriented words. Another method, proposed by [36], makes use of both lexical and syntactic features for sentiment analysis and is a machine learning approach. This method, however, missed pertinent contextual information which indicates that the individual sentence itself is vital when extracting semantic orientation. An alternative method was suggested by [37] for utilizing WordNet‟s synonymy relations for tagging words with Osgood's three semantic dimensions. The shortest path joining a particular word to the words „good‟ and „bad‟ was calculated through WordNet relations in order to assign values of positive or negative to the word. Dictionary-based methods for sentiment classification at the word-level have no need of large corpora, or search engines having special functionalities. Rather, they depend on readily available lexical resources existing today such as WordNet. They are able to compile comprehensive, accurate and domain-independent word lists containing their sentiment and subjectivity annotated senses. Such lists provide a vital resource for sentence or text sentiment classification and because of early compilation they are able to increase efficiency of sentiment classification at text and sentence level. In contrast to the other works this work presents sentence level lexical/dictionary knowledgebase method to tackle the domain adaptability problem for different types data [31] [6]. Dictionary-based techniques make use of the data found in references and lexicographical resources, such as WordNet and the thesaurus which can be used for assigning sentiments to a large number of words. Majority of these methods utilize various relationships between words (synonymy, antonymy, hyponymy /hyperonymy) in order to find the seed words and other entries as described earlier. The data existing in dictionary definitions is made use in word-level sentiment orientation in some of the recent methods. For semantic orientation lexical based semantic terms are extracted using dictionaries like SentiWordNet, ConceptNet etc. for the sentence level classification. According to [13]. The first try at employing WordNet relations in word sentiment annotation was made by [15][18]. They made the suggestion about an extension to lists of manually tagged positive and negative words by adding to the list the synonyms for those words. They began with just 54 verbs and 34 adjectives. The method was applied in two occurrences and acquired 6079 verbs and 12113 adjectives. Then, on the basis of the


strength of sentiment polarity which had been assigned to each word, the words which had been acquired were ranked. This strength-of-sentiment score or rank for each word was calculated by maximizing the probability of the category of the word‟s sentiment in regards to its synonyms [24] [25]. Semantic characteristics, like word sentiment, of each word are greatly acknowledged as good indicators of semantic characteristics of a phrase or a text that contains them, e.g. in (B. Baharudin, 2010) [20]. A sentence or text level sentiment annotation system uses words as indicators (features) of sentiment and therefore, requires the creation of words lists annotated with sentiment markers. The research on word-level sentiment annotation has produced a number of such lists of words that were manually or automatically tagged as sentiment or classified as related to sentiment. [39] [19] suggested a method that would use different information occurring at the same time in order to acquire words related to opinion (e.g., disapproval, accuse, commitment, belief) from texts as a way to carry out analysis of subjectivity at the word level. Two different techniques were used. The log-likelihood ratio is computed with the first technique; using data obtained by calculating how often words obtained from one sentence occur with seed words taken from [40]. Relative frequencies of words found in documents, either subjective or objective, are computed by using the second technique. When NLP and statistical techniques are utilized, much importance is given to sentiment analysis at the word or feature level because it is an analysis of the text with the most detail. The semantic orientation of a phrase or an opinion word is determined by the techniques proposed by [18] and [41]. Several researchers used a preset seed word to enable extraction of opinion-oriented words and features [42] [43] and form a list used for semantic orientation, extraction and classification of opinion. Determining the polarity and subjectivity of a text is not the only aim of sentiment analysis. On the contrary, what the author of the text specifically likes or dislikes regarding an object is also of importance [44]. Our main focus here is to discuss sentence and document level sentiment analysis. Sentence level analysis decides what the primary or comprehensive semantic orientation of a sentence is while the primary or comprehensive semantic orientation of the entire document is, handled by the document level analysis [13] [43]. Document level sentiment analysis deals with a document as a whole and classifies all the sentiments which have been expressed about a certain object by the authors showing whether the overall document is positive, negative or neutral. However, the text documents or reviews are broken down into sentences for sentiment analysis at the sentence level. These sentences are then evaluated by utilizing lexical or statistical methods in order to determine their semantic


orientation. Three main steps are involved, namely Pre-processing, Text Analysis and Sentiment Classification. A compilation of specific reviews are taken as input by the model and are then processed according to the above three steps to obtain results. Review classification and evaluation of sentences or expressed opinions in the reviews are the results produced by the model. The machine learning method and topic classification are similar in the sense that topics are classes of sentiment such as Negative and Positive [14]. This is how it works: a review is broken down into phrases or words, the review is then presented as a document vector (bagof-words model), and finally, the review is classified on the basis of the document vectors. It is apparent that classifying a sentiment can easily be formulated as a supervised learning problem which has two class labels (negative and positive). In regards to the assumption above, it is not a surprise that the reviews utilized in existing research regarding data for training and testing are mostly product based. Data for training and testing is easily available due to any typical review site having already assigned a reviewer rating (e.g. 1-5 stars) to each review [45]. Commonly, a thumbs-up or positive review will be assigned 4-5 stars while a negative or thumbs-down review is assigned with only 1-2 stars. Studies present to date have taken unlabeled data from the domain of interest with labelled data from another domain as well as general opinion words and made use of them as features for adaptation[46] [12] [18]. In this work a technique for domain independent sentence level classification of sentiment is introduced [47]. Rules for all parts of speech are applied so that they can be scored on the strength of their semantics, contextual valence shifter, and sentence structure or expression on the basis of dynamic pattern matching. Moreover, word sense disambiguation to extract accurate sense of the sentence has also been addressed. Opinion type, confidence level, strength and reasons are all can be identified using this system. SentiWordNet and WordNet are utilized as the primary knowledge base which has the further capability of being strengthened by using modifiers, information in the contextual valence shifter and all parts of speech.

3

Lexical Based Semantic Orientation

In this section, the proposed sentence level sentiment classification method is described in detail. In the first step, sentences are split into subjective and objective ones based on lexical dictionary. Subjective sentences are further processed for extraction to classify as positive, negative or neutral opinions. A rule based lexicon method is used for the classification of subjective and objective sentences. From subjective sentences, the opinion expressions are extracted and their semantic scores are checked using the SentiWordNet directory. The final weight of each individual sentence is calculated after considering the whole sentence structure, contextual


information and word sense disambiguation. The steps below described the overall process of the sentiment analysis of the proposed method.  Split reviews into sentences and make a Bag of Sentences (BoS).  Remove noise form sentences using spelling correction, convert special characters and symbols (photonics) to their text expression. Use POS for tagging each word of the sentence and store the position of each word in the sentence.  Make a comprehensive dictionary (feature vector) of the important feature with its position in the sentence.  Classify the sentences into objective and subjective sentences using lexical approach.  Using a lexical dictionary as a knowledge base, check the polarity of the subjective sentence as positive, negative or neutral.  Check and update polarity using the sentence structure and contextual feature of each term in the sentence. Text Pre-processing involves data preparation and cleaning of the datasets which is essential for the accurate execution of the next step i.e. Text Analysis. Some pre-processing steps typically used in data preparation/cleaning are as follows: removal of content that is non-textual as well as mark-up tags (for HTML pages), and removal of non-essential review data, such as dates of the reviews and names of the reviewers, as this type of data is not needed for the sentiment analysis. Taking samples of the reviews in order to build a classifier may also be a part of the data preparation. A number of works by (Balahur & Montoyo, 2009) [20] [48] disclosed that review datasets are often predominated by positive reviews. In regards to this, during classifier training a number of researchers used review datasets with balanced class distributions in order to help with the demonstration of their algorithms‟ performance [14] [49].

3.1

Creating and using Knowledgebase for Domain Independent Sentiment Classification

Effective sentiment orientation of phrases and text is dependent upon annotated words list with lexical semantic features. Dictionaries (lexicon-based approach) and corpora (corpus based approach) can be used to produce these lists of annotated words. Dictionaries or lexicon-based approach is independent of domain and uses sentiment bearing words to classify text for sentiments. It takes advantage of totality and general nature of dictionaries like SentiWordNet and WordNet. In contrast, corpus-based method is domain dependent to acquire these features. It is more sensitive towards domain changes and needs more time and efforts to obtain the desired training for a particular domain. On the other hand lexicon-based approach is advantageous over corpus-based machine learning approach because it relies on the existing resources and does not depend on specific search facilities and


it is domain independent. Corpus-based methods or methods based on compilations are able to compile lists of negative and positive words with comparatively highly accuracy. However, in order to reach their full potential, most of these approaches need immense annotated training datasets. Corpusbased methods can overcome some of these limitations by utilizing dictionary-based approaches; these approaches depend on existing lexicographical resources (such as WordNet) to provide semantic data in regards to individual senses and words. Yet the performance of lexicon method is limited in terms of its adoption to different domains and genres. Extraction of word sentiment information from dictionaries and lexical resources is important for feature acquisition, subsequent annotation, semantics and lexicography in the development of automatic extraction systems. These systems are able to automatically assign a semantic tag to each term or feature to classify sentiments into positive, negative or neutral category.

3.2

WordNet

The research efforts of the Department of Linguistics and Psychology at Princeton University for better understanding of English language and semantics resulted in WordNet lexicon. It is a complete lexicon where English language terms and semantics can be searched and retrieved as per their conception and semantic affiliations. At its third version, WordNet is available as a database, searchable via web interface or via a variety of software APIs, providing a comprehensive database of over 150,000 unique terms organised into more than 117,000 different meanings (WORDNET, 2006). WordNet also grew with extensions of its structure applied to a number of other languages (WORDNET, 2009). According to [13], the first try at employing WordNet relations in word sentiment annotation was made by [15][18]. They made the suggestion about an extension to lists of manually tagged positive and negative words by adding to the list the synonyms for those words. They began with just 54 verbs and 34 adjectives. The method was applied in two occurrences and acquired 6079 verbs and 12113 adjectives. Then, on the basis of the strength of sentiment polarity which had been assigned to each word, the words which had been acquired were ranked. This strength-of-sentiment score or rank for each word was calculated by maximizing the probability of the category of the word‟s sentiment in regards to its synonyms. An alternative method was suggested by [37] for utilizing WorldNet‟s synonymy relations for tagging words with Osgood's three semantic dimensions. The shortest path joining a particular word to the words „good‟ and „bad‟ was calculated through WordNet relations in order to assign values of positive or negative to the word [25]


3.3


SentiWordNet

SentiWordNet is sentiment analysis lexical resource made up of synset from WordNet, a thesaurus-like resource; they are allocated a sentiment score of positive, negative or objective. These scores are automatically generated using the semi-supervised method which is described in [51]. It is also available freely for research purpose on web. SentiWordNet is one of the sources of sentiment analyses. It is a semi-automatic way of providing word/term level information on sentiment polarity by utilizing WordNet database of English terms and relations. Each term in WordNet database is assigned a score of 0 to 1 in SentiWordNet which indicates its polarity. Strong partiality information terms are assigned with higher scores whereas less bias/subjective terms carry low scores. SentiWordNet is made up of a semi-supervised method which refers to a subset of seed terms to obtain semantic polarity. Each set of synonymous terms is assigned with three numerical scores ranging from 0 to 1 which indicates its objectiveness i.e. positive and negative bias [50] . One of the key features of SentiWordNet is that it assigns both positive and negative scores for a given term according to the following rule [51]: For a synset s, we define [50].   

Pos(s) Neg(s) Obj(s)

Positive score for synsets. Negative score for synsets. Objectiveness scores for synsets.

To illustrate how opinion information appears in SentiWordNet, the table below (Table 2) presents sample rows extracted from the raw database file. Table 2. Sample SentiWordNet Data Offset

PosScore

NegScore

SynsetTerms

POS a

10073761

0.125

0.625

n

10036762

0.375

0.125

v r

311113 139759

0.25 0.125

0.25 0.125

Strained, forced constrained Feat, exploit, effort Slur, dim, blur Unsuitably, inappropriately

As described in the above section by [50] , SentiWordNet terms are sorted according to their meaning, expression or the part of speech the term is used in a given sentence. [51] presented how opinion scores are affected by the parts of speech which are illustrated in Table 3 below. Table 3. Scoring Statistics Per Part of Speech [51]

Part of Speech

% Synsets with Objectivenes= 1

Average Objective Score

Average Pos. Score

0.94

Average Neg. Score

0.

0.

Noun

83.50 %

Verb

81.05 %

Pos(s) + Neg(s) + Obj(s) = 1 ; The positive and negative scores are always given, and objectiveness can be implied by the relation: Obj(s) = 1 – (Pos(s) + Neg(s))

Adverb

32.97%

Adjective

44.71%

Polarity scores according to synset and relevant part of speech are grouped by SentiWordNet database as a text file. The table below describes the columns for one entry in the database reflecting opinion information of a synset. Table-1 shows the details.

From the above (Table 3) data it can be seen that nouns and verbs are mainly objective in nature with little or no polarity. Weaker association of nouns and verbs with other terms in WordNet, carrying positive or negative bias, has been realized in the building process of SentiWordNet. Adverbs and adjectives are such part of speech which possesses the highest percentage of terms with positive subjective score. According to [51] using adjectives or adverbs (modifiers) are more common in expressing subjective opinion than verbs and nouns, which are more frequently used in objective scenarios. One more observation about adverbs is that although they own substantial polarity weight (only 32.97% of terms contain no subjective bias) yet their average score is significantly positive.

Then the following scoring rule applies:

Table 3. 1 SentiWordNet Database Record Structure Table 1. SentiWordNet Database Record Structure Fields POS

Descriptions Part of speech linked with synset. This can take four possible values:  a = adjective=jj  n = noun=nn  v = verb=vb  r = adverb=rb

Offset

Numerical ID which associated with part of speech uniquely Identifies a synset in the database.

Positive score

Positive score for this synset. This is a numerical value ranging from 0 to 1.

Negative Score

Negative score for this synset. This is a numerical value ranging from 0 to 1.

Synset Term

List of all terms included in this synset.

4

022 0.94

0

0. 026

0.69 8

0. 034

0. 235

0.74 3

034

0. 067

0. 106

0. 151

A relevant tag is assigned to each term, such as verb, noun, adjective etc, which specifies its role in the sentence. Although a number of standards exist for tagging formats yet the most popular are Penn Treebank annotated corpus [52] and the various instances of the CLAWS tag sets, derived from the original tag set for the brown corpus [53]. Penn Treebank tag set relevant to SentiWordNet has been demonstrated as an example in Table 3.7 below. The


complete set of tags is available in the appendix section. Table 4. Penn Treebank Tags [52]for Parts of Speech Present in SentiWordNet Part of Speech Adjective Noun

verb adverb

Penn Treebank Tags JJ, JJR (Comparative), JJS (Superlative) VB, VBD (Past tense), VBP (Present tense), VBZ (Present tense 3rd person), VBG (Gerund), VBN (Past participle). RB, RBR (Comparative), RBS (Superlative) NN, NNP (Proper noun), NNPS (Proper noun, plural), NNS (Plural)


sense number is extracted from SentiWordNet, which gives efficient results. If patterns are not exactly matched, then it checks for the nearest pattern and the score of that nearest pattern is extracted from SentiWordNet, There comes an issue while evaluating a particular term in SentiWordNet as to what specific WordNet synset this terms belongs and what would be its score. Consider the example for the term “mad”, with four synsets in WordNet [50]. Table 5. Example of Multiple Scores for The Same Term in SentiWordNet Synset

SentiWordNet

(Pos,

huffy, mad, sore (roused to

Contextual pattern information is done on the information which has been tagged so that it can be used with SentiWordNet database. Development of an application for the process of correctly reading and matching the tagged documents terms and their part of speech tags to a SentiWordNet score is needed.

3.4

Word Sense Disambiguation

WSD is an important step in semantic orientation to extract the correct sense of a term or expression in a sentence. Sentiment analysis, in most cases, relies on lexicons of words that may be used to express prejudice or subjectivity. These works do not address the peculiarity of different senses of a word in a way that its true sense is not categorized. Moreover, subjective lexicons are not accumulated as word meanings; rather they are compiled as lists of keywords. In most cases, these keywords have both opinionated and factual senses. Depending upon the contextual appearance, some degree of positive or negative polarity can be experienced even with the purely subjective sense. The following research work illustrates the benefits of utilizing WSD in determining the positive and negative nature of opinions and the role of existing resources. Quite few methods were presented for word sense extraction. [54], developed a supervised disambiguation system which is evaluated by which determines the subjective or objective sense of words in a given context. Inventory of subjective and objective senses can be viewed as an inventory of the senses of words but with coarse granularity in this approach. Supervised learning approaches are mostly used for sense word sense disambiguation. The contribution of this work is to check the WSD using unsupervised approach using the existing public resources [6]. The proposed method extracts the semantic pattern of the desired sentence using the opinion expression position in the sentence. Then, all possible patterns for that opinion expression for all possible senses are extracted based on the WordNet glossaries; the system locates an exact pattern match of the desired sentence and extracts the sense number from the WordNet synset. The semantic score for that

Gloss

Score

0.0

Neg)

0.1 25

anger)

"she gets mad when you wake her up so early"; "mad at his friend"; "soreover a remark"

brainsick, crazy, demented,

0.0

0.5

"a man who had gone mad"

disturbed, mad, sick, unbalanced, unhinged (affected with madness or insanity) delirious, excited, frantic, mad, 0.375

0.125

"a crowd of delirious baseball

unrestrained (marked by

fans";" something frantic in their

uncontrolled excitement or

gaiety"; "amad whirl of pleasure"

emotion) harebrained, insane, mad (very 0.0 foolish)

0.25

"harebrained ideas"; "took insane risks behind the wheel"; "a completely mad scheme to build a bridge between two mountains"

In the above example, (Table.5), four meanings can possibly be referred to the adjective “mad”. The question here is as to what meaning this word is referring in a particular sentence and what particular score, positive or negative, should be assigned to it in SentiWordNet. Determining which synset needs to be applied on a specific context is analogous to the problem of WSD [50]. In this work, experiments with different data sets have been performed for semantic orientation using WSD technique and its impact on the complexity of the data sets have been addressed. As a first step, part of speech tagging is used to obtain some level of disambiguation for extracting semantic scores from SentiWordNet. However if there occurs a multiple sense within the same part of speech a simpler approach can be used to assign scores based on the following rules. (1)Evaluate scores for each synset for a given term(2)If there are conflicting scores e.g. positive and negative scores exist for the same term – then check the sense of the sentence using their contextual lexical pattern(3)Return the SentiWordNet positive or negative score according to the sense of the sentence . It is apparent that classifying a sentiment can easily be formulated as a supervised learning problem which has two class labels (negative and positive). In regards to the assumption above, it is not a surprise that the reviews utilized in existing research regarding data for


training and testing are mostly product based. Data for training and testing is easily available due to any typical review site having already assigned a reviewer rating (e.g. 1-5 stars) to each review [45]. Commonly, a thumbs-up or positive review will be assigned 4-5 stars while a negative or thumbs-down review is assigned with only 1-2 stars. Studies present to date have taken unlabeled data from the domain of interest with labelled data from another domain as well as general opinion words and made use of them as features for adaptation[46] [12] [18]. In [14] a machine learning technique with a minimum cuts algorithm is utilized for sentiment classification. A document based on topic oriented classification models is usually represented as a set of words in which topic sensitive terms are vital. On the other hand, sentiment-oriented classification considers polar terms like the words “excellent” and “worst” as essential. Features based on individual polar terms are less expressive than the actual sentiment structures in the context of the sentence [55]. „The full story of how lexical items reflect attitudes is more complex than simply counting the valences of terms‟ [22]. So the rule based method is used to check the polarity of sentences and the contextual information at the sentence level. The process is used to extract the contextual information from the sentence and calculate its semantic orientation using SentiWordNet, WordNet and predefined intensifier dictionaries. From the results, it is clear that contextual information and consideration of sentence structure for correct sense extraction is very important for useful sentiment classification. The contributions of this work are sentence level semantic pattern extraction for word sense disambiguation, consideration all the parts of speech (POS) of the sentence for semantic orientation and it is a domain independent based polarity classification. However, the limitations of this work include the dependency on a lexical dictionary and limited word sense disambiguation. We evaluate the system on several datasets and online comments that it‟s outperformed.

4

Experiments and Results

For evaluation of our proposed method, 1000 comments are collected from the twitter datasets publically available for research purposes [56] and extract 500 short comments from cricinfo1, the cricket word cup 2011. Table-1 shows the blog comment dataset information. For customer reviews, we collected three types of online datasets to check the system performance (1) popular publicly available corpus from movie-review polarity dataset v2.0 IMDB movie reviews2 . The data set consists of 1000 positive and 1000 negative reviews in individual text files, and also the sentences polarity dataset (includes 5331 positive and 5331 negative processed sentences / snippets [28]. We take the positive and negative sentences to check the 1

http://www.cricinfo.com


performance of our proposed system. (2)We extracted 1000 reviews from the Skytrax3, where more than 2.5 million independent reviews for over 670 airlines and 700 airports. After spilt the reviews into sentences there 8 average sentences per review is found. We extracted the subjective lexicons and semantic orientation from all the positive and negative sentences. (3) We performed our experiments on the dataset of about 2600 hotel reviews downloaded, which is collected from TripAdvisor4, that is one of the popular review sites about hotels and traveling. We extracted only text of reviews using text file. In this section, the pre-processing steps used in this work are described. After removing the noise, the reviews/comments are split into sentences to extract the feature level sentiment score from SentiWordNet. A BOS is made from the split sentences, and each sentence is stored with a Review-ID and SentenceID. After applying the POS, the position of each word in the sentence is also stored for further processing. The noisy text degrades the performance of the classifier and is a main hurdle in semantic orientation. A machine based learning methodology was proposed by [57], where pre-processing for noise removal is done using a generative model and noisy channel method. Unrestrained vocabulary, spelling mistakes, casual capitalization of words, white spaces etc are assumed to be possibly contained in the text. Sentence boundary detection for speech transcripts is yet another well researched issue. Majority of these systems make use of Hidden Markov Models (HMM) as well as a set of lexical and prosodic features which have been learnt from a manually tagged training set [49]. In this work we follow the idea of [49] for noise removal with some modification and implementation of new technique for text cleaning and their possible semantic orientation especially in case of blogs.

4.1 Sentence Boundary Identification For sorting of reviews/comments into correct sentences, sentence boundary identification is very important. A rule based module has been implemented for this purpose. In this method, “.” is considered as the sentence boundary, if it is not preceded by a predefined word: i.e., Pvt., Ltd., etc. The “.” is also ignored after an abbreviation list (defined in the dictionary) and immediately after digits which do not follow a space character. Sentences commonly begin with a capital letter which is the most identifiable marker for sentence breaks. It is rational to consider two lines as merging together if it starts with a small letter and the line before it does not have any recognizable punctuation symbol. However, if a new line begins with a small letter with a non dictionary term, then the first word the last sentence is checked and merged with it, if it become a dictionary word and then a sentence is made by joining the contents of the two lines.

4.2 Sentence Cleanliness To remove noise from a text, we apply an algorithm to remove symbols, check spellings and



correct those words which are incorrectly written. We extract the semantic score of those symbols, from which the reviewer wants to express something meaningful. In online Web forums, social networks, blogs etc., people mostly write short forms of words and use symbols in comments to express their views. How these symbols and short forms are made is useful in extracting their semantics from such sentences. Up till now, no such tool has been available to extract and calculate the semantic score of such words and symbols because there is no standard rule available for writing comments, reviews on online forums, blogs etc. Such symbols or shortened words are B4, Gr8, bcz, :), ##123, @@@, >> etc. Here, we try to overcome this problem by collecting such symbols and words that are most often used in conveying special messages instead of writing a full sentence. We apply the same algorithm to online customer reviews and comments and receive very encouraging results as well as improvement in the sentiment analysis process. However, there should be proper rules for writing reviews and comments to express our views on online forums and blogs. Table-4.3 shows a collection of such types of symbols with their meanings. When viewing these little things, which are called "emoticons", often the idea is to turn your head sideways so a picture is made on a lot of the smiley faces [;-)]for example, where the [ ; ] semicolon are the eyes, the [ - ] hyphen is the nose, and the [ ) ] parenthesis is the mouth. Also, you see some people use the hyphen [-] to show the nose, while others will show the same expression without the nose, e.g.: [;-) and ;)] represent the same thing. These symbols show emotions and expressions which are very important for sentiment analysis [58]. So the datasets are processed to remove noise, clean up the special characters and symbols, and check for spelling mistakes; furthermore, we apply the POS tagger and classify the sentences into subjective and objective. The twitter comments have already been processed for positive and negative sentiments mainly for testing purpose. Also the blog comments are process as positive and negative and the subjective sentences are consider for further processing to find the semantic orientation at the individual sentence level. Table-6 shows the classification of subjective and objective sentences for blogs comments. Table-6 Sum of Opinion Sentences Datasets twitter Cricket World Cup 2011 Movie Reviews Airlines Reviews Hotel Reviews

Commt 1000

Sent 2045

Subj 1636

Obj 409

Percent 80/20

500

1630

1238

392

76/24

Already Already 10662 process 10662 8530 processed ed

10 2132

2132

1000

7730

5405

2325

70/30

2600

25663

17704

7969

68/32

The subjective sentences are processed for semantic orientation by taking the contextual features and using the SentiWordNet for the semantic score. 250 sentences and 75 feedbacks have taken containing positive and negative opinion respectively, from twitter and cricinfo shown in Tables-6 & 7. These sentences and feedbacks are then evaluated to test the performance the proposed method using Accuracy (Precision), Recall and F1 measures. It is clear from the results that average accuracy is 83% at sentence level and 87% at feedback level is achieved. Hence it can be concluded that our lexical based method‟s performance is better and is adoptive with different domains datasets. Table-7. Accuracy of Opinion Orientation for Positive and Negative Sentiments for Twitter Comments Sentences

Positive Negative

Total

Positive

Negative

Accuracy

Recall

F1 Value

250 250

212 47

38 203

0.848 0.812

0.819 0.842

0.833 0.827

0.853

0.831

0.842

75

64

11

75

13

62

0.827

0.849

0.838

Positive Feedback

Negative

Table-8. Accuracy of Opinion Orientation for Positive and Negative Sentiments for Cricket Blog Comments

Sentences

Feedback

Total

Positive

Negative

Accuracy

Recall

Positive

250

216

34

0.864

0.837

F1 Value 0.850

Negative

250

42

208

0.832

0.860

0.846

Positive

75

67

8

0.893

0.848

0.870

0.840

0.887

0.863

Negative 75

12

63

The results of proposed method were compared with corpus based machine learning methods on same datasets from the recent research work. [59] Presented a machine learning method for classifying sentiment of twitter messages and achieved an accuracy of 80% for positive and negative sentiments. The results are also compared with the online twitter comments classification system “twitrratr”. In twitrratr a list of positive keywords and a list of negative keywords were created. Their accuracy is 76% and 80% on positive, negative and neutral opinion. The method is also compared with [31], presented machine learning based lexical method for different dataset with accuracy of 71% in blogs dataset and 82% for movie reviews and news datasets. The proposed method achieved better results than this approach. Most corpus based techniques use flat feature vector or BoW methods to represent the documents. In [43] the authors have taken the feature list as a seed for the opinion orientation. Table-8 shows the overall performance of our proposed system in comparison with the machine learning corpus based methods. The contribution of this work is the extraction of sentence level semantic orientations taking into account all parts of speech and sentence contextual structure. Figure-1 shows the



performance of the proposed method as compared to machine teaching [59], [31] and Twitrratr methods.

Sentiment Orientation Accurecy at Sentence…

To evaluate the performance of the proposed method on binary classification we use precision, recall and F-measure shown in Table 6. 47 negative and 47 positive feedbacks were taken form movie reviews and 500 positive and 500 negative sentences. 87% precision with 85% recall and 84 % precision with 87% recall is recorded for positive and negative sentiments at sentence level respectively. For the same dataset at feedback level we achieve 95% precision with 91% recall and 93% precision with 93% recall values for positive and negative sentiments respectively. Similarly for the hotel reviews 26 positive and 26 negative feedbacks were taken for feedback level evaluation, and each 300 positive and negative sentence were taken. The system achieved 85% precision and 82% recall for positive reviews with 83% F1 value and 81% precision and 84% recall for negative reviews with 82% F1 test values at feedback level. For sentence level revelation 82% precision and 78% recall for positive reviews, which gives the 80 % F1 test values and 76% precision and 81% recall is recorded with 78% F1 value for negative reviews. Table-9. Evaluation of proposed method for Blog comments Twitrratr

Andreevskaia Bergler,(2008)

Go, Bhayani, Huang,(2009)

Proposed Method

Sentiment Orientation at

Sentence Level

76

71

80

83

Feedback Level

80

82

82

87

Figure.-1. Comparison with Other Methods using Blogs Datasets For airline reviews better results were achieved in terms of person and recall for positive or negative sentiments orientation both at sentence and feedback levels shown in Table 6. For sentence level evaluation we achieved 87.8% precision, 84.7% recall and 86.3% F1 measure value for positive text, similarly for negative reviews 84.2% precision and 87.4% recall value with 85.7 % F1 test value is achieved. For feedback level performance on the same dataset, the proposed method achieved 95.9% precision, 88.6% recall and 92.1% F1 value for positive reviews, similarly for negative reviews at sentence level evaluation 90.4% precision, 95.7% recall and 93% F1 valued is achieved as shown in Table 6. Hence it can be concluded that our lexical based method‟s performance is better and is adoptive with different domains datasets in customer reviews and blogs comments. Table 10- Overall Accuracy of Customer Reviews using Precision Recall and F-Measure

Movie

Total

Positive

Negative

Accuracy

Recall

F1 Value

Positive

500

436

64

0.872

0.850

0.861

Negative

500

77

423

0.846

0.869

0.857

47

45

3

0.957

0.918

0.938

47

4

44

0.936

0.936

0.936

Positive

300

245

55

0.817

0.775

0.795

Negative

300

71

229

0.763

0.806

0.784

26

22

4

0.846

0.815

0.830

26

5

21

0.808

0.840

0.824

Positive

600

527

73

0.878

0.847

0.863

Negative

600

95

505

0.842

0.874

0.857

73

70

3

0.959

0.886

0.921

73

9

66

0.904

0.957

0.930

Sentences

Feedback

Positive Negative

Hotel

Sentences

Feedback

Positive Negative

Airline

Sentences

Feedback

Positive Negative



In this study different factors of the Lexical based sentiment orientation approach is examine. This includes those aspects that can result in the enhancement of a lexical based classifier‟s performance. These factors involves are the acquisition of Knowledge-rich lexicon, Noise removal from the text and word or expression actual sense extraction. The study of Knowledge-rich lexicon-based methods to achieve sentiment orientation has received relatively little attention in the literature as compared to corpus-based methods. Thus it is clear from the above mention results that the contributions of this work is a rule based sentence level lexical based methods for sentiment classification is the exploration into how other types of information like noise removal from text, WSD, weights or scores, and valence shifters can be added to enhance lexical based systems.

5

[3]

P. Falinouss, “Stock trend prediction using news articles: a text mining approach,” Lule{å} tekniska universitet, 2007.

[4]

M. Sharp, “Text mining,” Rutgers University, School of Communication, Information and Library Studies, Allen, J. Natural Language Understanding, Second Edition. Redwood City, CA: Benjamin/Cummings, 2001, pp. 112.

[5]

A. Khan, B. Baharudin, L.H. Lee, and K. Khan, “A Review of Machine Learning Algorithms for Text-Documents Classification,” Journal of Advances in Information Technology, vol. 1, 2010, pp. 420.

[6]

A. Khan, B. Baharudin, and K. Khan, “Sentiment Classification from Online Customer Reviews Using Lexical Contextual Sentence Structure,” Communications in Computer and Information Science, Software Engineering and Computer Systems,, Springer Verlag, 2011, pp. 317-331.

[7]

S. Chakrabarti, “Data mining for hypertext: A tutorial survey,” ACM SIGKDD Explorations Newsletter, vol. 1, 2000, pp. 1-11.

[8]

R. Kosala and H. Blockeel, “Web mining research: A survey,” ACM SIGKDD Explorations Newsletter, vol. 2, 2000, pp. 115.

[9]

J.M. Wiebe, “Tracking point of view in narrative,” Internation Journal of Computational Linguistics, vol. 20, 1994, pp. 233-287.

[10]

J. Wiebe, T. Wilson, R. Bruce, M. Bell, and M. Martin, “Learning subjective language,” Internation Journal of Computational linguistics, vol. 30, 2004, pp. 277-308.

[11]

M.A. Hearst, “Untangling text data mining,” Proceedings of the 37th annual meeting of the Association for Computational Linguistics on Computational Linguistics, 1999, pp. 3-10.

J. Wiebe and E. Riloff, “Creating subjective and objective sentence classifiers from unannotated texts,” Computational Linguistics and Intelligent Text Processing, 2005, pp. 486-497.

[12]

D.J. Orre, P. Gerstl, and R. Seiffert, “Text mining: finding nuggets in mountains of textual data,” Proceedings of the fifth ACM SIGKDD international conference on

J.M. Wiebe, “Learning subjective adjectives from corpora,” Proceedings of the National Conference on Artificial Intelligence, 2000, pp. 735-741.

[13]

B. Pang and L. Lee, “Opinion mining and sentiment analysis,” Foundations and Trends

Conclusion and Future Work

In this paper, a rule based sentiment analysis approach is proposed for opinion classification. The contextual information and the sense of each individual sentence are extracted according to the pattern structure of the sentence. The semantic score for the extracted sense is assigned to the sentence using SentiWordNet. The final semantic weight is calculated after checking semantic orientation of each term in the sentence. The decision is made to check the polarity of positive, negative or neutral opinions. The results show that the sentence structure and contextual information in the review are important for the sentiment orientation and classification. The sentence level sentiment classification performs better than the document level semantic orientation. The limitations include the dependency on lexicons and the lack of term sense disambiguation. From the results, it is clear that the proposed method achieves an average accuracy of 83% at the sentence level and 87% at the feedback level for blogs comments datasets and 97% at feedback and 86% at sentence level for reviews. In future, extraction of the acute sense of sentence and remove noisy text for an efficient semantic orientation. Furthermore, the knowledgebase need to improve for the semantic scores of all parts of speech.

References [1]

[2]

Knowledge discovery and data mining, 1999, pp. 398-401.


in Information Retrieval, vol. 2, 2008, pp. 1135. [14]

[15]

[16]

[17]

[18]

[19]

[20]

[21]

[22]


[23]

S. Hariharan, R. Srimathi, M. Sivasubramanian, and S. Pavithra, “Opinion mining and summarization of reviews in web forums,” Proceedings of the Third Annual ACM Bangalore Conference, 2010, pp. 1-4.

[24]

S.M. Kim and E. Hovy, “Determining the sentiment of opinions,” Proceedings of the 20th international conference on Computational Linguistics, 2004, pp. 13671374.

C.W.K. Leung and S.C.F. Chan, “Sentiment Analysis of Product Reviews,” Encyclopedia of Data Warehousing and Mining Information Science Reference,, 2008, pp. 1794-1799.

[25]

A. Andreevskaia and S. Bergler, “Mining WordNet for fuzzy sentiment: Sentiment tag extraction from WordNet glosses,” Proceedings of EACL, 2006, pp. 209-216.

J.M. Wiebe, “Identifying subjective characters in narrative,” Proceedings of the 13th conference on Computational linguistics-Volume 2, 1990, pp. 401-406.

[26]

M. Koppel and J. Schler, “The importance of neutral examples for learning sentiment,” Computational Intelligence, vol. 22, 2006, pp. 100-109.

[27]

T. Zagibalov and J. Carroll, “Unsupervised classification of sentiment and objectivity in Chinese text,” IJCNLP, 2008, pp. 304-311.

[28]

B. Pang and L. Lee, “Seeing stars: Exploiting class relationships for sentiment categorization with respect to rating scales,” Proceedings of the ACL, 2005, pp. 115-124.

[29]

A.B. Goldberg and X. Zhu, “Seeing stars when there arenʼt many stars: Graph-based semi-supervised learning for sentiment categorization,” Proceedings of the First Workshop on Graph Based Methods for Natural Language Processing, 2006, pp. 4552.

[30]

X. Zhu and A.B. Goldberg, “Kernel regression with order preferences,” PROCEEDINGS OF THE NATIONAL CONFERENCE ON ARTIFICIAL INTELLIGENCE, 2007, pp. 681-686.

[31]

A. Andreevskaia and S. Bergler, “When specialists and generalists work together: Overcoming domain dependence in sentiment tagging,” Proceedings of ACL-08: HLT, 2008, pp. 290-298.

[32]

C.W.K. Leung, S.C.F. Chan, and F. Chung, “Integrating collaborative filtering and sentiment analysis: A rating inference approach,” Proceedings of The ECAI 2006 Workshop on Recommender Systems, 2006, pp. 62-66.

[33]

P.D. Turney, “Mining the web for lexical knowledge to improve keyphrase extraction: Learning from labeled and unlabeled data,”

B. Pang, L. Lee, and S. Vaithyanathan, “Thumbs up?: sentiment classification using machine learning techniques,” Proceedings of the ACL-02 conference on Empirical methods in natural language processingVolume 10, 2002, pp. 79-86.

A. Esuli and F. Sebastiani, “Determining term subjectivity and term orientation for opinion mining,” Proceedings the 11th Meeting of the European Chapter of the Association for Computational Linguistics (EACL-2006), 2006, pp. 193-200. S.M. Kim and E. Hovy, “Automatic detection of opinion bearing words and sentences,” Companion Volume to the Proceedings of the International Joint Conference on Natural Language Processing (IJCNLP), 2005, pp. 61-66. S. Bethard, H. Yu, A. Thornton, V. Hatzivassiloglou, and D. Jurafsky, “Automatic extraction of opinion propositions and their holders,” 2004 AAAI Spring Symposium on Exploring Attitude and Affect in Text, 2004, pp. 2224-2236. P. Turney, “Thumbs up or thumbs down? Semantic orientation applied to unsupervised classification of reviews,” Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics (ACLʼ02), 2002, pp. 417-424. S. Argamon, K. Bloom, A. Esuli, and F. Sebastiani, “Automatically determining attitude type and force for sentiment analysis,” Human Language Technology. Challenges of the Information Society, 2009, pp. 218-231. L. Polanyi and A. Zaenen, “Contextual valence shifters,” Computing attitude and affect in text: Theory and applications, 2006, pp. 1-10.



Journal of Artificial Intelligence Research, Arxiv preprint cs/0212011, 2002, pp. 1-34. [34]

[35]

[36]

[37]

[38]

[39]

[40]

A. Esuli and F. Sebastiani, “Determining the semantic orientation of terms through gloss classification,” Proceedings of the 14th ACM international conference on Information and knowledge management, 2005, pp. 617-624. C.O. Alm, D. Roth, and R. Sproat, “Emotions from text: machine learning for text-based emotion prediction,” Proceedings of the conference on Human Language Technology and Empirical Methods in Natural Language Processing, 2005, pp. 579-586. T. Wilson, J. Wiebe, and P. Hoffmann, “Recognizing contextual polarity in phraselevel sentiment analysis,” Proceedings of the conference on Human Language Technology and Empirical Methods in Natural Language Processing, 2005, pp. 347-354. J. Kamps, M.J. Marx, R.J. Mokken, and M. De Rijke, Using wordnet to measure semantic orientations of adjectives, European Language Resources Association (ELRA), 2004. B. Baharudin, “Sentence Based Sentiment Classification from Online Customer Reviews,” Frontiers of information Technology, FITʼ10, LAHORE: ACM, 2010, pp. 1-6. A. Balahur and A. Montoyo, “A Semantic Relatedness Approach to Classifying Opinion from Web Reviews,” Procesamiento del lenguaje natural, vol. 42, 2009, pp. 47-54. V. Hatzivassiloglou and K.R. McKeown, “Predicting the semantic orientation of adjectives,” Proceedings of the eighth conference on European chapter of the Association for Computational Linguistics, 1997, pp. 174-181.

ACM SIGKDD international conference on Knowledge discovery and data mining, 2004, pp. 168-177. [44]

B. Liu, “Sentiment analysis and subjectivity,” Handbook of Natural Language Processing,, 2010, pp. 978-1420.

[45]

K. Sarvabhotla, P. Pingali, and V. Varma, “Supervised Learning Approaches for Rating Customer Reviews,” Journal of Intelligent Systems, vol. 19, 2009, pp. 79-94.

[46]

J. Zhao, K. Liu, and G. Wang, “Adding redundant features for CRFs-based sentence sentiment classification,” Proceedings of the Conference on Empirical Methods in Natural Language Processing, 2008, pp. 117-126.

[47]

A. Khan, B. Baharudin, and K. Khan, “Sentiment Classification Using Sentencelevel Lexical Based Semantic Orientation of Online Reviews,” vol. 6, 2011, pp. 11411157.

[48]

K. Dave, S. Lawrence, and D.M. Pennock, “Mining the peanut gallery: opinion extraction and semantic classification of product reviews,” Proceedings of the 12th international conference on World Wide Web, 2003, pp. 519-528.

[49]

L. Dey and S.K. Haque, “Opinion mining from noisy text data,” Proceedings of the second workshop on Analytics for noisy unstructured text data, Singapore: ACM, 2008, pp. 83-90.

[50]

B. Ohana, “Opinion mining with the SentWordNet lexical resource,” Dublin Institute of Technology, 2009.

[51]

A. Esuli and F. Sebastiani, “Sentiwordnet: A publicly available lexical resource for opinion mining,” Proceedings of LREC, 2006, pp. 417-422.

[41]

A. Neviarouskaya, H. Prendinger, and M. Ishizuka, “Semantically distinct verb classes involved in sentiment analysis,” Proc. IADIS Int. Conf. Applied Computing, 2009 IADIS AC (1), 2009, pp. 27-35.

[52]

M.P. Marcus, M.A. Marcinkiewicz, and B. Santorini, “Building a large annotated corpus of English: The Penn Treebank,” Computational linguistics, vol. 19, 1993, pp. 313-330.

[42]

A.M. Popescu and O. Etzioni, “Extracting product features and opinions from reviews,” Proceedings of the conference on Human Language Technology and Empirical Methods in Natural Language Processing, 2005, pp. 339-346.

[53]

R. Garside, G. Leech, and G. Sampson, “The computational analysis of English: A corpusbased approach,” Computational Linguistics, vol. 14, 1988, pp. 90-97.

[54]

C. Akkaya, J. Wiebe, and R. Mihalcea, “Subjectivity word sense disambiguation,” Proceedings of the 2009 Conference on Empirical Methods in Natural Language

[43]

M. Hu and B. Liu, “Mining and summarizing customer reviews,” Proceedings of the tenth


Processing: Volume 1-Volume 1, 2009, pp. 190-199. [55]

Y. Hu and W. Li, “Document sentiment classification by exploring description model of topical terms,” Computer Speech & Language, ScienceDirect, vol. 25, 2010, pp. 386-403.

[56]

D.A. Shamma, L. Kennedy, and E.F. Churchill, “Tweet the debates: understanding community annotation of uncollected sources,” Proceedings of the first SIGMM Workshop on Social Media, 2009, pp. 3-10.

[57]

A. Clark, “Pre-processing very noisy text,” Proc. of Workshop on Shallow Processing of Large Corpora, 2003, pp. 12-22.

[58]

W. Wong, W. Liu, and M. Bennamoun, “Integrated scoring for spelling error correction, abbreviation expansion and case restoration in dirty text,” Proceedings of the fifth Australasian conference on Data mining and analystics-Volume 61, 2006, pp. 83-89.

[59]

A. Go, R. Bhayani, and L. Huang, “Twitter sentiment classification using distant supervision,” CS224N Project Report, Stanford, Stanford University, 2009, pp. 112.
