Verb Sentiment Scoring: A Novel Approach for

Download PDF
0 downloads 0 Views 2MB Size Report
Comment
Aug 3, 2017 - scoring. Specially a new sentence level scoring algorithm has been developed considering adjective-verb-adverb combinations. All the newly ...
2017 IEEE/ACM International Conference on Advances in Social Networks Analysis and Mining

Verb Sentiment Scoring: A Novel Approach for Sentiment Analysis Based on Adjective-Verb-Adverb Combinations Y.H.P.P Priyadarshana

L. Ranathunga

IFS R&D International (Pvt) Ltd Sri Lanka [email protected]

University of Moratuwa Sri Lanka [email protected] A country's stability and productivity is based on the citizen's perception towards the parties which they are governed. Although there are various reasons behind this vast improvement but the involvement and the rapid growth of social media takes the highest priority. Parts of speech play a major role in sentiment analysis. When considering sentiment scoring which is a segment of sentiment analysis, the importance of adjectives, adverbs, verbs and nouns can be understood well. A substantial amount of work has been done to show off the importance of adjectives and adverbs in sentiment scoring yet very limited research work has been done for verbs and nouns. The aim of this research is to bring up verbs on the stage and build up novel sentiment scoring algorithms based on verbs to be used in measuring sentiment score. Verbs can be considered as the core of any given sentence. Verbs denote the performing action of the subject hence verbs can be found in any given sentence. VerbNet is the largest online verb lexicon where describes the organization of verb classes based on the semantics of verbs [1]. Based on the semantics some rules have been defined and those rules are used for verb centric sentiment analysis. Even though adjectives can be introduced as the most important part of speech when comes to sentiment scoring but still verbs and their synonyms also can make a huge impact for the overall sentiment score of a given sentence.

Abstract - Opinion mining has become a major aspect of determining business analytics and business intelligence in today's business. Someone's perception on a product can be the sole decision maker which decides the selling pattern of the product. Parts of speech are key elements of measuring the sentiment score of a given sentence. Verbs can be considered as the driving force of a sentence hence verbs play a vital role in sentiment analysis. All the existing mechanisms to measure the computational value for verbs are based on adjective centric approaches. This research introduces a novel approach in measuring verbs score. We have introduced new verb semantics which cover all the categories of verbs. Based on those semantics, new verb scoring axioms are implemented. Then new set of hybrid algorithms have been formulated in terms of word, sentence and paragraph level scoring. Specially a new sentence level scoring algorithm has been developed considering adjective-verb-adverb combinations. All the newly implemented functions have been compared with the existing scoring functions using movie reviews as the data domain. It can be experienced that the novel functions are performed better than existing functions where the novel functions perform an accuracy of 83% in precision. Keywords – Sentiment analysis; verb scoring; verbs of degree; verb scoring axioms; adjective-verb-adverb scoring

I. INTRODUCTION

Sentiment scoring can be considered as a more subjective approach compared to other information mining tasks. There are plenty of work has been done for aiming opinion mining but almost all of those scoring algorithms are implemented based on adjectives and their synonyms of the given context [2] [3]. Adjectives can make a huge impact for the overall sentiment score of a given sentence but still no one hardly argue the logic behind neglecting other parts of speech such as verbs, nouns and adverbs [4]. Pseudo expected value scoring function, which can be considered as the pioneer sentiment scoring algorithm, was introduced considering adjectives and their synonyms as the impact factor for scoring the given document data set [5]. Almost all the other later approaches were either totally based on the pseudo expected value scoring function or minor modifications of the original approach.

Sentiment analysis or opinion mining can be considered as one of the most rapidly improving fields with the involvements of various technological verities and innovations. Analyzing people's attitude or perception in a computational manner is a great need in today's context. Majority of people use customer reviews and recommendations to choose their preferred restaurants. Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

ASONAM '17, July 31-August 03, 2017, Sydney, Australia © 2017 Association for Computing Machinery. ACM ISBN 978-1-4503-4993-2/17/07…$15.00 http://dx.doi.org/10.1145/3110025.3110101

533

2017 IEEE/ACM International Conference on Advances in Social Networks Analysis and Mining Although verbs are important for building up the skeleton of a meaningful sentence, yet there is an absence of considerable work done to show up the importance of verbs in opinion mining. For understanding the importance of verbs in sentiment scoring, there are three things need to be discussed and those are verb lexicon, verb semantics and verb scoring algorithms. There is some work has been done regarding those three categories and those must be considered before implementing a novel approach.

different sentiment contexts, aligning verb polarity and verb scoring etc. [16]. Still there are many verbs to be modelled to be used for sentiment scoring. Another main limitation of those verb lexicons can be identified as verb negation is yet to be implemented. The next important segment is the verb semantic categories. Verb lexicons will be used for categorization purposes and end up as verb semantics. Verb classification plays a major role here. Beth Levin's verb classification can be considered as the pioneer approach in verb classification, which can be used for building up verb semantics [17]. Some classes of verbs have been identified as important in sentiment scoring out of Levin's verb classification [18]. So, there are some gaps to be fulfilled in order to cover all the verb categories which are important in opinion mining. Another way of categorizing verbs can be done as main verbs and auxiliary verbs. Main verbs don’t have to depend on any other verbs to express their meaning but auxiliary verbs must depend on other verbs. Although some researchers have found out that only main verbs are useful in sentiment analysis [19], still other segments cannot be simply neglected since those can also generate a huge impact for the overall sentiment score in each context. Some semantic verb categories [20] have been implemented based on the seed verbs [21] by including the synonyms of them as well [22]. But there are some categories should be considered and by adding all together, a well-organized verb semantic categories can be built so that those can be used to introduce verb scoring axioms as well. Another approach was the experiments done based on the largest online verb lexicon, the VerbNet [1]. VerbNet consists with 73 verb classes which are useful in sentiment analysis. There are rules have been developed based on those classes which are used for sentiment phrase level analysis [23].

There are available resources such as lexicons to fulfill the polarity and subjectivity opinion mining. There are English small and large word lexicons which can be for information opinion mining purposes [6] [7]. For generating a lexicon model, proper word banks have been used. The largest online word database which is the WordNet was introduced for fulfilling such purposes and WordNet has been introduced for implementing new lexicon databases [8]. Opinion generating words and their synonyms of WordNet were used for sentiment scoring in many experiments and has shown positive results [9] as well as failures [10]. Wiktionary, which is the online Wikipedia dictionary and extension of WordNet, was used to measure the sentiment score based on the verbs in the given context. But still Wiktionary is also determined the polarity of adjectives in the given textual context [4]. So, the goal of proving the importance of verbs in sentiment scoring will not be achieved. Another publicly available lexicon model called SentiWordNet was developed aiming the synsets as the main target segment of WordNet [11]. But there was an issue in the reliability of the SentiWordNet sprang up at the first stage of the lexicon exploration, after analyzing the scoring values of adjectives. Thus, cannot be used for generating a verb based sentiment scoring model. SentiFul lexicon database was developed to address the main drawback of existing approaches, which was the absence of scalability [12]. SentiFul can be used to identify new words from the document corpus and computationally build up a scoring mechanism for those identified new words. This was also again used mainly for scoring adjectives. Another drawback was the absence of the implementation of verb negation. Another approach was implemented by combining complex model with SentiWordNet to express the multiple attitudes of verbs in deep sentiment analysis [13] which was only based on 580 subjective verb lexical units. But the above mentioned mechanism was performed better results compared to existing other mechanisms at that period. Another lexicon model was developed as a Dutch lexical database extension where combining two resources such as the Dutch WordNet and the Dutch reference lexicon [14]. The mechanism was totally based on the Dutch verbs so that there was an issue in applying the system with different lexicons. Another combined lexicon database mechanism was developed based on supervised learning in order to perform in many domains [15]. But still there were drawbacks such as the difficulty in applying with

The last and the most important point to be discussed is the existing verb scoring algorithms. Before moving directly to verb scoring functions, it’s better to have a look on the sentiment scoring functions and their origins. The pioneer sentiment scoring function, which is the pseudo expected value function was invented by Cesarano and his co-researches based on adjectives and their synonyms [5]. After that several different scoring functions were introduced but all were based on OASYS, the opinion analysis system which was based on the pioneer function, the pseudo expected value function. Even though there are existing verb scoring functions, all are based on the adjective based pioneer function [19]. So, there is a demand for a novel and a valid function for measuring verb based sentiment analysis. Since there is a high demand for implementing verb negation scoring functions, our mechanism should support that as well. Another important fact is that the novel mechanism which is going to be implemented should support to the existing sentiment scoring mechanisms where the approach can be considered as a hybrid mechanism in sentiment scoring.

534

2017 IEEE/ACM International Conference on Advances in Social Networks Analysis and Mining Another important aspect is the data set which is used for training and testing purposes. Movie reviews and comments of the movies which are listed in the international movie database (IMDB) will be the dataset in our approach. These movie reviews contain number of verbs and their synonyms which can be useful for implementing our approach. The existing adjective centric verb scoring functions will not provide a provable and a solid solution for such domains.

falls to the main verb category which explains something in the past. The second one explains something happened in the past but there is a present impact of the past action. There are two different sentiment impacts can be seen from first and second scenarios. Third scenario shows the importance of other parts of speech for verb sentiment scoring where the adverb badly improves the sentiment impact of the verb was broken. The last example shows the negation impact on verb scoring. It can be clearly seen the different impacts of verb sentiment scoring when we consider all the verb categories. In this paper, we develop verb scoring axioms based on all the verb categories including main, auxiliary and negation which are important in verb sentiment scoring. First of all, verbs of degree have to be classified and for main positive verbs those are already been categorized as follows.

There are key features and contributions of our approach and those will be discussed as follows. 1) Verb scoring axioms can be considered as the key elements when we implement scoring functions. For implementing those axioms, verbs of degree play a major role. These verb scoring axioms set up the scope where the scoring algorithms have to be implemented. These axioms will be described in detailed manner in section 2 of this paper. 2) Using those defined scoring axioms, verb sentiment scoring functions will be implemented. These will be three functions to calculate verb word level sentiment score, sentence level sentiment score based on verbs and paragraph level sentiment score based on verbs in the context. These novel functions and their usages will be implemented and described in sections 3, 4 and 5. Then the experimental results of the whole approach based on the movie reviews and comments taking as the datasets, will be described in section 6.

1. Verbs of cognition: these include verbs such as consider, hope, think, know and so on. 2. Verbs of perception: these include verbs such as see, feel, here and son on. 3. Verbs of attitude: these include verbs such as enjoy, hate, love and so on. 4. Verbs of activity: these include verbs such as read, work, explain and so on. 5. Verbs of event: these include verbs such as become, reply, pay, soon and so on. 6. Verbs of process: these include verbs such as change, increase, grow and so on. Then we need to define verbs of degree for auxiliary verbs and negation verbs. This is the novel thing which is going to be implemented whereas all the prior approaches were used the degrees which are above described. Auxiliary verbs of degree can be classified as follows.

II. VERB AXIOMS Verb scoring axioms are the gateway to build up verb based sentiment scoring functions. The scope can be considered as one of the main aspects in defining axioms. The categories of verbs play a major role here. According to the Levin's verb categorization, there are certain categories of verbs which have to be taken in to account in measuring verb score [17]. As it is discussed earlier in the paper, main verbs have already been categorized in to six semantic categories. But still the auxiliary verbs have to be categorized and the minimizers or the verb negation also should be considered. So, the aim is to build up verb scoring semantic categories including all the verb nature aspects for the first time in verb sentiment scoring. As we have experienced, all the verb sentiment scoring approaches were implemented around main positive verbs. Still plenty of work is needed to be done for auxiliary verbs and negation as well. The following examples show the importance of all verb categories in verb scoring. • • • •

7. Verbs of passive: these include verbs such as is produced, are produced, was produced, were produced and so on. 8. Verbs of progressive: these include verbs such as is writing, was writing and so on. 9. Verbs of perfective: these include verbs such as have broken, has broken and so on. 10. Verbs of model: these include verbs such as can, could, shall, should, may, might, will, would, must and so on. Negation of degree has been defined previously from the previous research of sentiment negation scoring. For verb context, three of them can be used and those are as follows. 11. Verb negation of imperatives: these include negation verbs such as no, cannot, don’t, might not and so on. 12. Verb negation of prefixes and suffixes: these include negation verbs such as misunderstanding, misbehaving, endless and so on. 13. Verb negation of emphasizing: these include negation verb modifiers such as a bit, nothing at all, least and so on.

He broke his leg. He has broken his leg. His leg was broken badly. He could not have broken his leg.

Considering the above sentences, the importance of all verb categories and their impacts can be understood. First sentence

535

2017 IEEE/ACM International Conference on Advances in Social Networks Analysis and Mining So, there are thirteen (13) semantic verb categories including all the aspects of verbs. Although the first six categories were introduced and we just use them, the novelty becomes with the innovation of last seven categories. There was no work has been done targeting those semantics and those new categories will add a considerable value for the new verb sentiment scoring approach. Now based on those sematic categories, verb scoring axioms can be introduced. When we consider the above categories, there are 3 main categories such as main positive verbs, auxiliary verbs and negation can be seen. Out of them if we consider main positive verbs category, this can also be divided into two segments such as physical type category and mental type category. Verbs of cognition, verbs of perception and verbs of attitude can be separated into mental type category and the rest into physical type category. As a summary, there are 4 main categories can be seen. Considering all these semantic categories, two types of verb scoring axioms can be defined as intra verb scoring axioms and inter verb scoring axioms.

attention to the semantics between main verb categories. Hence following inter verb scoring axioms can be implemented. Axiom 7: Each main verb category of mental or each auxiliary verb category has a verb sentiment score which is less than or equal to each main verb category of physical. Axiom 8: Each main verb category of mental or each auxiliary verb category has a sentiment score which is less than to each negation verb category. These are the verb scoring axioms can be implemented using the verb semantics which we described earlier. These axioms have to be proven reasonable enough before implementing verb sentiment scoring algorithms based on those defined axioms. When we take axiom 1, it compares the two sub categories of main positive verb category. Physical actions generate more polarity and weight than the conceptual behavior. When someone says, The movie was good, expresses a higher sentiment value than I think the movie was good. So, axiom 1 can be taken as a reasonable axiom for implementing verb scoring algorithms. When we consider an inter verb scoring axiom such as axiom 7, it can be proven as reasonable since main physical verb category expresses more sentiment weight comparing to main mental verb category or auxiliary verb category.

Intra verb scoring axioms focus on the semantics inside each four main categories. Here we don’t consider anything between the main categories. But inter verb scoring axioms describes the semantics between each four main categories. Based on this approach, a novel set of verb scoring axioms can be defined in order to use them for implementing verb sentiment scoring algorithms. The intra verb scoring axioms can be introduced as follows.

Axiom 5: Each verb of passive and each verb of progressive or each verb of perfective has a verb sentiment score which is less than to each verb of model.

Based on the above defined verb scoring axioms, we then move for the implementation of verb scoring algorithms. Here our aim is to implement verb scoring function to determine verb score given in a sentence, implement a sentence level scoring function based on adjective-verb-adverb combination and implement a paragraph level scoring function. All of these algorithms cover both positive and negation polarities hence these will be novel algorithms for verb sentiment scoring. Also, the sentence level adjective-verb-adverb combination scoring function will be an enhanced approach to the existing adjective centric approach. Adjective centric sentiment scoring is an outdated approach whereas industries who highly depend on customer feedback and customer reviews require something better than the adjective centric sentiment scoring. Since we have already implemented a novel approach of adjectiveadverb, that can be even extended in order to include verbs as well whereas we can end up introducing a novel adjective-verbadverb sentiment scoring approach. Also, our approach can be plugged with any other sentiment scoring algorithm for accomplishing more computational objectives so that our novel approach can be introduced as a hybrid approach.

Axiom 6: Each verb negation of emphasizing or each verb negation of prefixes and suffixes has a verb sentiment score which is less than to each verb negation of imperative.

III. ENHANCED VERB SCORING

Axiom 1: Each verb of cognition or each verb of perception or each verb of attitude has a verb sentiment score which is less than to each verb of activity or each verb of event or each verb of process. Axiom 2: Each verb of event or each verb of process has a verb sentiment score which is less than or equal to each verb of activity. Axiom 3: Each verb of perception or each verb of cognition has a verb sentiment score which is less than or equal to each verb of attitude. Axiom 4: Each verb of passive and each verb of progressive has a verb sentiment score which is less than to each verb of perfective or each verb of model.

As it is explained earlier, almost all the other existing sentiment scoring algorithms were based on an adjective centric approach. Since the Pseudo-expected value function, which was based only on adjectives and their synonyms in the given

Intra verb scoring axioms describe the inner semantics of main four verb scoring categories. Still we should pay our

536

2017 IEEE/ACM International Conference on Advances in Social Networks Analysis and Mining context, was introduced, plenty of sentiment scoring approaches were based on it. Even though some researches came up with various approaches such as adjective-adverb combination sentiment scoring, adjective-verb-adverb approach etc. all of those were based on Pseudo-expected value function. Even those approaches are useful in some contexts, still those are adjective centric approaches. So, our aim is to implement new verb centric approach to determine sentiment score of verbs. The target audience will be the verbs and related synonyms compared to the total number of opinion expressing words which are adjectives and adverbs. Negation also will be taken into consideration whereas no other approach was considered negation separately before. For implementing verb sentiment scoring, main verbs, auxiliary verbs and verb negation, which are the main sentiment categories of verbs, are considered. There are few definitions to be introduced before implementing the algorithm.

comments. Then considering all scores a value is picked up as the rating which is a value between 1 to 10. Suppose S={s1,…,sm} set of movie reviewers who set a rating value ji(r). Let’s assume (sdsk(r)) set of R movie reviews. Let k ≥ 1 any integer value. Then we have to calculate the mean (µ) and standard deviation (α) values for the ratings given by set of reviewers for a given movie review r and then need to calculate the k value as well. If 94% of ratings are inside the standard deviation, then the k value will be taken as 6. Then we can derive our verb sentiment scoring function as follows. We use the (sdsk(r)) notation to denote the remaining average score of the reviews.

𝑒𝑣𝑠 𝑘 (𝑟) =

Definition 1: (n (v, r, neg)) This notation is used to denote the number of occurrences of verb v or it’s synonyms are used in the given movie review or comment r. This verb v can be either a main verb or auxiliary verb. Since the negation is also needed to be included, neg is used to denote the negation factor upon the given verb v.

(2)

Now verb scoring algorithms can be generated using the above implemented function, which can be used for verb sentiment scoring purposes. The advantage is since this is a hybrid approach, this can be plugged with the existing sentiment scoring functions for computational purposes. This can be used to generate an automatic rating for movie reviews which is our domain in this research. Not only that, for many industries which are valuing customer perception and feedback, can use this novel approach for accomplishing various other future purposes.

Definition 2: We use notation oeaa(r) and oeaa(R) in order to represent set of all the opinion expressing words which are adjectives and adverbs including their synonyms in the given movie review or comment r or any given collection of movie reviews or comments R. Earlier the opinion expressing words were restricted to adjectives but here we use all the sentiment expressing parts of speech in the given domain.

IV. ENHANCED SENTENCE SCORING We have enhanced the existing sentence level sentiment scoring algorithms as well. This novel approach can be described as an adjective-verb-adverb combined sentiment scoring approach. There are existing adjective-verb-adverb approaches to calculate sentiment score but all those algorithms are based on adjective centric sentiment scoring mechanisms [18]. But we use almost all the opinion expressing words, not only adjectives, in order to fulfill our need. This approach is also a hybrid sentiment scoring approach, whereas all the other existing sentiment scoring functions can be plugged with this for accomplishing various other computational purposes.

Now using these two definitions we can build up an expression which shows the proportion of the series of opinion expressing verb and the synonyms of it compared to the total number of adjectives, adverbs and negation in the given movie review set R. This relative proportion expresses the relative sentiment proportion of the given verb v, with respect to the total number of opinion expression words in the domain.

𝑛(𝑣, 𝑟, 𝑛𝑒𝑔) ∑𝑣′ ∈𝑜𝑒𝑎𝑎(𝑅) 𝑛(𝑣 ′ , 𝑟, 𝑛𝑒𝑔)

𝑛(𝑣, 𝑟, 𝑛𝑒𝑔) ∑𝑣′∈𝑜𝑒𝑎𝑎(𝑅) 𝑛(𝑣 ′ , 𝑟, 𝑛𝑒𝑔) ∑𝑟∈𝑅 𝑠𝑑𝑠𝑘 (𝑟)

∑𝑟∈𝑅 𝑠𝑑𝑠 𝑘 (𝑟)×

(1)

In order to determine sentence level sentiment score, we have to calculate each score of the opinion expressing words in each sentence. As an example, if someone take, The film has been produced in early 90’s but still worth than many films in the current industry, there are plenty of opinion expressions take into consideration. First, we have to identify the opinion expressing parts of speech and then calculate the sentiment score of each. Also, negation expressing words cannot be neglected since they play a vital role in here as well.

This has a computational value which can be used to determine the verb sentiment scoring function. For that we need to focus on another definition, which helps to drive towards the verb sentiment scoring function. Definition 3: Currently there is a rating called IMDB rating for the movies listed in IMDB. This rating system is done using set of movie reviewers. Those reviewers go through each of the comments for a movie and they assign a score based on the

537

2017 IEEE/ACM International Conference on Advances in Social Networks Analysis and Mining Adjectives and adverbs can be identified using the algorithms which were implemented through the prior research and the sentiment score of those can be calculated using the enhanced scoring algorithms which were implemented earlier [24]. In order to identify the verbs, VerbNet which is the largest online verb lexicon, has been used. JVerbNet 1.2 library has been used for retrieving verb classes to identify the category of the extracted verbs. Then using the verb sentiment scoring function which we have implanted here, verb sentiment score can be determined. Suppose the sentiment scores of adjectives, adverbs and verbs and their synonyms are denoted as SCadjk(r), SCadvk(r) and SCverk(r) respectively. Also, we need to retrieve the negation impact of the given sentence and to achieve that the negation impact algorithm can be used. By considering all these factors, a new sentence level sentiment scoring function can be implemented. First, we implement the function for calculating each opinion expressing word score as follows. This function has been named as POS Score which is to denote the Parts Of Speech Score.

V. ENHANCED PARAGRAPH SCORING A paragraph can be described as a collection of sentences. A movie review or comment consist with more than one reviews or sentences. So, first the individual score of all the sentences in the reviews has to be calculated using the Enhanced Sentence Scoring function. After determining those scores, then we have to calculate the standard deviation and mean values of those scores and remove the outliers which are not in the scope of the standard deviation. Then the remaining scores should be taken (Sr) for the next calculations. Then using the remaining number of review sentences (N), and their accumulated scores (Sr) we can implement the paragraph level sentiment scoring function. Even though this is a very straight forward process and similar to work done in the past, the core of the calculation is based on the Enhanced Sentence Scoring function. This function is developed an innovation by filling all the necessary gaps which have not been filled through past researches. So, the paragraph scoring process is based on the sentence level scoring, hence this approach is also an enhancement of the previous work done on sentiment paragraph level scoring.

𝑃𝑂𝑆 𝑆𝑐𝑜𝑟𝑒 𝑘 (𝑟) = ∑ 𝑟∈𝑅

𝑆𝐶 𝑎𝑑𝑗𝑘 (𝑟) + ∑

𝑟∈𝑅

𝑆𝐶 𝑎𝑑𝑣𝑘 (𝑟) + ∑

𝑟∈𝑅

We have developed three (3) new algorithms to measure the sentiment score of movie reviews and comments. Since these are hybrid modeled algorithms, any given sentiment scoring function can be connected for determining various computational objectives. These enhanced algorithms are tested using training and testing different movie reviews and comments sets but can be used for different other domains as well. For accomplishing the testing, 2000 movie reviews are used as training data set and another 1000 different movie reviews are used as for the testing purposes. These movie reviews are extracted from the IMDB, based on the movies which were produced between 2010 to 2016.

𝑆𝐶 𝑣𝑒𝑟𝑘(𝑟) (3)

Negation impact is important when calculating sentence level scoring. We have already addressed the sentiment negation and sentence level negation impact from the prior research [25] so that it can be used for accomplishing this purpose as well. Negation impact algorithm is as follows.

𝑖𝑚𝑝(𝑤, 𝑟, 𝑛𝑒𝑔) =

𝑛(𝑤, 𝑟, 𝑛𝑒𝑔) ∑𝑤 ′ ∈𝑛𝑒𝑤(𝑅) 𝑛(𝑤 ′ , 𝑟, 𝑛𝑒𝑔)

(4)

VI. IMPLEMENTATION AND EXPERIMENTATION Now it’s just matter of combining above 3 and 4 algorithms to implement the enhanced sentence sentiment scoring function as follows.

𝑒𝑠𝑠 𝑘 (𝑟) =

𝑃𝑂𝑆 𝑆𝑐𝑜𝑟𝑒 𝑘 (𝑟)×𝑖𝑚𝑝(𝑤, 𝑟, 𝑛𝑒𝑔) ∑𝑟∈𝑅 𝑠𝑑𝑠𝑘 (𝑟)

We have implemented the novel scoring algorithms, Java was used as the programming language to implement the scoring functions, user interface and other integration modules. We have used 2000 movie reviews and comments extracted from IMDB as the data set and JVerbNet 1.2 as an external java library to identify the verb semantics which we have developed earlier. MySQL was used as the database to store the respective data. Then we have conducted two main experiments such as experimentation on verb scoring and experimentation on sentence scoring. Those experimentations and results can be described as follows.

(5)

Enhanced sentence scoring function is based on accumulate scores of all the opinion expressing words in the given sentence. This approach is reasonable since earlier approach was adjective centric whereas the other parts of speech such as verbs, adverbs etc. were neglected in measuring sentiment score. The final scoring function is the paragraph level scoring function and that can be implemented as follows.

A. Experimentation on Verb Scoring We have enhanced the existing verb scoring mechanism and we have to prove our function performs better than the existing

538

2017 IEEE/ACM International Conference on Advances in Social Networks Analysis and Mining approach in terms of human subjects. The aim is to compare both approaches on the basis of Pearson correlation coefficient to measure the best performing algorithm. The existing scoring algorithm has been developed with respect to human subjects and in our approach also we have implemented algorithm with respect to human subjects. This is because as it is described earlier, the movie reviews and comments have been evaluated by set of movie reviewers, what we have implemented as functions are based on human subjects. So, in order to compare these two algorithms, human subject can be taken as the common factor for both existing and novel verb scoring functions. Two set of value series have to be derived for representing both verb scoring functions. Then both Verb Scoring function (VSk(r)) and Enhanced Verb Scoring function (EVSk(r)) will be plotted in a graph with respect to human subjects, for this domain the human subject is the movie reviewer. The goal is to see the behavior of two functions when the value k gets vary. Then by evaluating the k value, the best performing algorithm can be determined. Following graph shows the behavior of two algorithms in Pearson correlation coefficient with respect to the k value.

B. Experimentation of Sentence Scoring The next approach is to measure the performance of the existing Sentence Scoring function and the Enhanced Sentence Scoring with respect to human subjects. Again, the aim is to determine the best performing algorithm when the k value gets vary. We measure the best performing algorithm based on performance and the accuracy level as well. The existing movie rating system is an automated system but still there is nothing related to sentiment scoring. So, our effort will be the reason to neglect the existing bias scoring mechanism. First, we have to prepare a value series for both Sentence Scoring function (SSk(r)) and Enhanced Sentence Scoring (ESSk(r)) with respect to human subjects. Then both have to be plotted in terms of Pearson correlation coefficient values to experiment the behavior of the algorithms when the k value gets vary.

Fig. 2. Pearson correlation coefficient of SS and ESS

The rounded marker plotted graph and the square marker plotted graph represent the Sentence Scoring function graph and the Enhanced Sentence Scoring function graph respectively. Both graphs vary when the k value gets vary. Both algorithms show their maximum performance when the k value becomes 0.4. It can be concluded that the novel Enhanced Sentence Scoring function performs better than the existing Sentence Scoring function with respect to human subjects.

Fig. 1. Pearson correlation coefficient of VS and EVS

The rounded marker plotted graph and the square marker plotted graph represent Verb Scoring function graph and Enhanced Verb Scoring function graph respectively. When k value gets vary, both algorithms take different values. But when k = 0.5 both algorithms perform in their maximum value in Pearson correlation coefficient and the Enhanced Verb Scoring function performs better than the existing Verb Scoring function for any given k value. So, the Enhanced Verb Scoring function performs better than the Verb Scoring function in Pearson correlation coefficient.

In order to determine more information regarding these two sentence scoring functions, measuring accuracy will play a huge role. A rating scale which is the same as the existing movie review rating scale, 0 to 10, is defined in order to make sure that our objective of measuring precision of Enhanced Sentiment Scoring function is fulfilled. There are some definitions we have to define before measuring the accuracy of the novel function. Let RevCom(thr, alg) to represent set of all movie reviews and comments after computationally processing using the algorithm, Enhanced Sentence Scoring, which is also denoted as alg, which has a high value than a given threshold value called thr. Let RevMan(thr) to denote all considering

539

2017 IEEE/ACM International Conference on Advances in Social Networks Analysis and Mining [2]

movie reviews and comments which the existing bias system processes a value which is over the threshold value (thr). Then using those definitions, the precision of the Enhanced Sentiment Scoring function for the given considering threshold value can be implemented as follows.

[3] [4]

[5]

|𝑅𝑒𝑣𝐶𝑜𝑚(𝑡ℎ𝑟, 𝑎𝑙𝑔) ∩ 𝑅𝑒𝑣𝑀𝑎𝑛(𝑡ℎ𝑟)|×100 |𝑅𝑒𝑣𝐶𝑜𝑚(𝑡ℎ𝑟, 𝑎𝑙𝑔)|

(6) [6]

VII. DISCUSSION AND CONCLUSION

[7]

At the initial stage of this research, our idea was to come up with a solid solution to calculate the sentiment score of a given verb. Verb is the key part of speech in any given sentence. A verb can be considered as the anchor of a sentence. But still there is a very minimal work done in terms of sentiment scoring to show the importance of a verb. There are some researches have been conducted around the main verb categories. But there are two verb types such as main verbs and auxiliary verbs so that there should be a mechanism to deal with auxiliary verbs as well. Also, verb negation plays vital role to decide the overall sentiment score of a given sentence. Therefore, verb negation has to be considered as well. If a solid sentiment verb scoring mechanism is going to be implemented, these categories have to be taken into account. We have developed verb semantics based on all the verb categories and then verb axioms based on the implemented semantics. New set of verb scoring functions are formulated based on those axioms. The sentence level scoring function is an enhancement of the existing adjectiveverb-adverb approach where we have accomplished our goal of formulating a sentiment scoring mechanism without limiting to an adjective centric method. Our novel approach can be considered as a hybrid scoring mechanism where any given sentiment scoring functionality can be plugged for accomplishing various other computational objectives.

[8] [9]

[10]

[11]

[12] [13]

[14]

[15]

[16]

[17] [18]

This novel approach has been tested in terms of performance and accuracy. We have selected IMDB movie reviews and comments as the data domain since those comments have a nature of both unstructured and structured human perception. 2000 vs 1000 movie reviews have been used as the training data set and as the testing data set respectively. First, we measure both functions performance in terms of Pearson correlation coefficient with respect to human subjects. This is done for both word and sentence level scoring formulations. Then we measure the accuracy in terms of precision. In both cases, our approach performs better results over the existing mechanism.

[19]

[20]

[21] [22] [23]

[24]

REFERENCES [1]

[25]

K. Kipper et al, "A Large-scale Extension of VerbNet with Novel Verb Classes," Proceedings of the EURALEX, 2006.

540

V. Hatzivassiloglou and K. McKeown, “Predicting the Semantic Orientation of Adjectives”, ACL-97, 1997. J. Wiebe, "Learning subjective adjectives from corpora," Proceedings of the 17th Conference of the AAAI, 2000. P. Chesley, B. Vincent, R.K. Srihari, "Using Verbs and Adjectives to Automatically Classify Blog Sentiment", American Association for Artificial Intelligence 2005. C. Cesarano, B. Dorr, A. Picariello, D. Reforgiato, A. Sagoff and V. Subrahmanian, "OASYS: An Opinion Analysis System," Proc. AAAI Spring Symposium on Computational Approaches to Analyzing Weblogs, 2006, pp. 21–26. J. Kamps, R. J. Mokken, M. Marx, and M. de Rijke, "Using WordNet to measure semantic orientation of adjectives," In Proceedings LREC-2004 Paris. S. Kim and E. Hovy, "Determining the sentiment of opinions," In Proceedings of COLING, Geneva, Swtizerland 2004. G. A. Miller, "WordNet: An on-line lexical database," International Journal of Lexicography, Special Issue, 3(4):235–312, 1990. M. Hu and B. Liu, "Mining and Summarizing Customer Reviews," Proceedings of the ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, 2004. J. Klavans and M. Kan, "Role of Verbs in Document Analysis," Proceedings of the Conference COLING-ACL, Montreal, 1998, pp.680– 686. A. Esuli and F. Sebastiani, "SentiWordNet: a publicly available lexical resource for opinion mining," Proceedings of the 5th International Conference on Language Resources and Evaluation, 2006, pp. 417–422. A. Neviarouskaya, H. Prendinger, M. Ishizuka, "SentiFul: Generating a Reliable Lexicon for Sentiment Analysis", 2009. I. Maks, P. Vossen, "A verb lexicon model for deep sentiment analysis and opinion mining applications," Proceedings of the 2nd Workshop on Computational Approaches to Subjectivity and Sentiment Analysis, ACLHLT 2011, pp. 10–18. I. Maks and P. Vossen, "Modeling Attitude, Polarity and Subjectivity in Wordnet," In Proceedings of Fifth Global Wordnet Conference, Mumbai, India, 2010. S. Goldensohn et al, "Building a sentiment summarizer for local service reviews," InWWWWorkshop on NLP in the Information Explosion Era, 2008. N. Hollenstein, M. Amsler, M. Bachmann, M. Klenner, "SA-UZH: Verbbased Sentiment Analysis," Proceedings of the 8th International Workshop on Semantic Evaluation, 2014, pp. 503–507. B. Levin, "English Verb Classes and Alternations: A Preliminary Investigation," Univ. of Chicago Press, 1993. V.S. Subrahmanian, D. Reforgiato, "AVA: Adjective-Verb-Adverb Combinations for Sentiment Analysis," Published by the IEEE Computer Society, 2008. S. Sarkar, P. Mallick, T.K. Mitra, "A Novel Machine Learning Approach for Sentiment Analysis Based on Adverb-Adjective-Noun-Verb (AANV) Combinations," Int. J. on Recent Trends in Engineering and Technology, Vol. 7, No. 1, July 2012. M. Sokolova, G. Lapalme, "Verbs Speak Loud: Verb Categories in Learning Polarity and Strength of Opinions," Canadian AI 2008, LNAI 5032, pp. 320–331, 2008. G. Leech, "Meaning and the English Verb," Longman 2004. Roget’s interactive thesaurus (2006), http://thesaurus.reference.com/ A. Neviarouskaya, H. Prendinger, M. Ishizuka, "Semantically Distinct Verb Classes Involved in Sentiment Analysis," IADIS International Conference Applied Computing 2009. Y.H.P.P. Priyadarshana et al, "Sentiment Analysis: Measuring Sentiment Strength of Call Centre Conversations," In Proceedings of the 2015 IEEE International Conference on Electrical, Computer and Communication Technologies (ICECCT), 2015, pp. 1-9. Y.H.P.P. Priyadarshana, L. Ranathunga, P.M. Karunaratne, "Sentiment Negation: A Novel Approach in Measuring Negation Score," In Proceedings of the IEEE Future Technologies Conference, 2016.