Pre-trained Word Embeddings for Arabic Aspect

Download PDF
1 downloads 0 Views 594KB Size Report
Comment
tweets in Arabic, manually labeled for ABSA with imbalanced aspect categories. ..... sentiment analysis: Lexicon-based and corpus-based," , 2013. [Online].
Pre-trained Word Embeddings for Arabic Aspect-Based Sentiment Analysis of Airline Tweets Mohammed Matuq Ashi, Muazzam Ahmed Siddiqui, Farrukh Nadeem Department of Information Systems Faculty of Computing and Information Technology King Abdulaziz University Jeddah, Saudi Arabia [email protected], [email protected], [email protected]

Abstract. Recently, the use of word embeddings has become one of the most significant advancements in natural language processing (NLP). In this paper, we compared two word embedding models for aspect-based sentiment analysis (ABSA) of Arabic tweets. The ABSA problem was formulated as a two step process of aspect detection followed by sentiment polarity classification of the detected aspects. The compared embeddings models include fastText Arabic Wikipedia and AraVec-Web, both available as pre-trained models. Our corpus consisted of 5K airline service related tweets in Arabic, manually labeled for ABSA with imbalanced aspect categories. For classification, we used a support vector machine classifier for both, aspect detection, and sentiment polarity classification. Our results indicated that fastText Arabic Wikipedia word embeddings performed slightly better than AraVec-Web. Keywords: data mining, NLP, machine learning, word embeddings, sentiment analysis, aspect-based sentiment.

1

Introduction

Word embeddings trained on large unlabeled corpora can be very useful in various Natural Language Processing (NLP) applications [1]. Word embeddings are a mathematical model representing a word in space, mostly a vector. Each component or dimension is a feature of this word that can have semantic or syntactic meaning. In fact, they very useful for the Arabic language being morphologically rich. That is, it does not ignore words morphology, through appointing a distinct vector to each word presented in a large dataset such as Wikipedia. Therefore, this is very important especially for the Arabic language having a diverse vocabulary, and many rarely used words [2]. In this paper, we have utilized vector-based features instead of hand-crafted features. That is we conduct separate experiments using two distributed word embeddings models which attempt to capture both semantic and sentiment information from the large corpus used in training. Namely, they are the fastText Wikipedia Arabic content model [2] and AraVec skip-gram model [3] trained on top of Arabic web pages. We use them both to perform the two defined sub-tasks of ABSA being Aspect Detection and Aspect-Based Sentiment Classification. We propose a data mining, more specifically, a text mining methodology that employs the microblogging site Twitter, identifying the most crucial components of airlines industry customer service. We use Twitter for it being the most frequently used data source in Arabic sentiment analysis articles [4]. We mine Saudi Airlines customers public Twitter commentaries unveiling their customer's’ opinions regarding their products/services to determine which has caused customer satisfaction or dissatisfaction. The rest of this research paper is arranged in the following order. The related work is described in section 2. The data collection and labeling are discussed in Section 3. In Section 4, the experiment design and implementation are presented. Later, we discuss the obtained results for both sub-tasks of ABSA in Section 5. Eventually, we end this paper in Section 6 with a summary and a brief discussion of future work.

2

Related Work

According to [5], the opinion mining, sentiment analysis, or subjectivity analysis research field, is the body of work which deals with the computational treatment of opinion, sentiment, and subjectivity in the text. Sentiment analysis has often been studied at three primary levels of classification either a document-level, a

sentence-level, or at an entity- or aspect-level [6]. Aspect-level sentiment analysis (ABSA) is not only concerned with finding the overall sentiment related to an entity but the polarity associated with the specific feature/aspect of an entity [4]. The Arabic language is classified by its morphology, syntax, and lexical combinations into three different categories either classical Arabic, modern standard Arabic (MSA), and dialectal Arabic [4]. Arabic has a very complex morphology and structure. Currently, sentiment analysis is prevalent in the English language, while sentiment analysis on the Arabic language is still recognized at its early [7]. However, research on Arabic sentiment analysis is getting more attention from the research community recently [8]. Most of the studied works in the literature on Arabic sentiment analysis have adopted hand-crafted features for their classification with prominent use of sentiment lexicons. Example of the reviewed lexicon-based sentiment analysis works are [9], [10], [11], [12], [13]. Word representations are critical to lexical features being employed for sentiment analysis of Twitter data [14]. Most of these techniques represent each word of the vocabulary by a distinct vector. The process of manually extracting features is such a laborious task that heavily required time and efforts. Specifically in morphologically rich languages such as Arabic with a large vocabulary and many distinct forms and rarely used words. On the other hand, word embeddings add useful information to the learning algorithms for achieving superior results in various NLP applications [15]. Arabic word representations have been used by many researchers as features for different Natural Language Processing tasks [3]. To name a few, [16] who studied word embeddings effectiveness being used as a feature in named entity recognition in Arabic presenting that this novel representation scheme can be used in substitution of lexicon-based features. That is they produced better performance although they used the dataset [17] that was a small dataset containing only 3,646 Egyptian Dialectal tweets with only about 40K tokens. Some works have been completed for Arabic sentiment analysis using word embeddings are as follows. In [8] trained their word embeddings model on top of a large corpus, they compiled of publicly available Arabic text collections such as newspaper articles and consumer reviews as well as others. Then, they trained and generated word vector representations (embeddings) from the corpus. They also trained several binary classifiers employing their word embeddings model using such vector-based features as feature representation detecting sentiment and subjectivity in three different datasets for dialectical Arabic using tweets and book reviews datasets and for subjectivity classification they used MSA with news articles as a dataset. They achieved an accuracy of 77.87% that is slightly better than the major methods of manual feature extraction. Moreover, [18] compared the performance of different base classifiers and ensembles for polarity determination in highly imbalanced datasets of tweets in dialectical Arabic using features learned by word embedding rather than hand-crafted features. Their results show that applying word embedding with the ensemble and synthetic minority over-sampling technique (SMOTE) can achieve more than 15% improvement on average in the F1 score. Also, [19] they used word embeddings to perform ABSA covering the tasks of aspect term extraction, aspect category detection, and aspect sentiment prediction achieving F1 scores of 79.9% for aspect term extraction, 86.7% for category detection, and 72.3% for aspect sentiment prediction.

3

Dataset

We started this project by creating a Twitter API user to leverage Twitter's Search API obtaining the bulk of data ‘tweets’ required for this research. We used many keywords such as ‘Saudia’, ‘Saudi Airlines’, ‫"الخطوط‬ "‫السعودية‬or the studied airline customer service Twitter account ‘Saudia_Care’. We used Python as the sole programming language; creating a script making the API calls, then, appending and storing the data into a csv file format. Abstracting data ’Tweets’ from Twitter was iterated over a period of five months starting from May 2017. After removing 100k of duplicate tweets or retweets at every iteration, we ended up with a 5k tweets dataset containing the most relevant tweets. Our Arabic tweets dataset has the majority of tweets being in the Saudi dialect than MSA since the studied airline being the national carrier airline of Saudi Arabia. Moreover, as seen in Fig. 3 of the overall architecture of the project, we have performed tweets preprocessing removing all URLs, user mentions, and unrelated tweets, getting rid of the numerous irrelevant tweets such as advertisement and so on. After observing the full dataset, the researcher, as well as annotators,

were able to identify 13 aspect categories that encompass the various tweeted about topics of airline service. The dataset labeling was performed manually dividing the task with a group of native Arabic speakers who are working in related fields. Aspect categories with topics discussed in each aspect category are presented in Table 1. We ended up with a distribution plot of the final dataset of 5k tweets for the 13 specified aspects categories related to the airline service as seen in Fig 1. Table 1: Description of all Aspect classes used to label the dataset. Aspect Schedule Destinations Luggage and Cargo Staff & Crew Airplane Lounges Entertainment Meals Booking Services Customer Service Refunds Pricing Miscellaneous

Tweets Topics flight schedule, rescheduling, timing, delays. airline destination and routes. luggage, air cargo, luggage allowance, luggage delays. all staff such as pilots and flight attendants. airplane seating, cabin features, maintenance. first-class and frequent flyer service and airport lounges in-flight entertainment, other media, and wi-fi. in-flight meals and in-flight services. airline website, mobile app, and self-service machines. customer communications and complaint management. refund and compensations. ticket pricing and seasonal airline offers. represents all tweets with gratitude and thanking or complaints about the airline as general with no mention of the other 12 aspect classes.

Fig. 1. A distribution plot of the dataset with every airline service aspect and its tweet counts.

Also, for the other part of the study sentiment polarity detection, the collected tweets are distributed concerning sentiment polarity. In this part of ABSA, we are only classifying the predicted label tweets as being either positive or negative, but we did not consider the neutral category. For the 5k dataset, the majority of the tweets belonged to the negative class with the total of 3590 tweets and only 1410 positive tweets. Thus, some of the aspect classes are represented with very few positive examples as in Fig. 2. The reason is that most of Twitter users use it as a medium to channel their complaints and negative energy obtained from negative experience and failed service. On the other hand, very few use it for positive commentaries, discussing their pleasant experiences or showing their constant gratitude to the airline. The Miscellaneous class was exceptional having an almost equal number of tweets for both polarity classes since it included all tweets that were general in their polarity as presented in Fig 2.

Fig. 2: Distribution of sentiment polarity for each of the airline service aspect.

4

Experiment Design and Implementation

Python is used as the sole programming language for this project because of its well-renowned machine learning library known as Scikit-learn. After loading the dataset, it was split into two sets, 80% of the data for training, and the other 20% for testing the performance of the classifier. In this research, we opted the Support Vector Machines (SVM) classifier due to its well-known superiority on other machine learning techniques as it has shown exceptional performance in text mining tasks in many works in the literature. The overall architecture of the proposed aspect-based sentiment analysis (ABSA) is described in Fig. 3. 4.1 Method Applying the two pre-trained word embedding models, we used simple vector-based features to classify the 5k collected airline-related tweets measuring the semantic distance between Arabic words and phrases utilizing the Vector Space Model to perform ABSA two defined sub-tasks that are Aspect Detection and Aspect-Based Sentiment detection. We used a similarity measure defining similar related words or n-grams of one classification label. In this research, we utilized two distinct word embedding models as the lexical resource for the two sub-tasks of Arabic aspect-based sentiment analysis. Such word embedding models use the Vector Space Model (VSM) and cosine similarity between words. Vector space models (VSMs) are one of the oldest and most renowned schemes for text representations where words are represented in continuous space where semantically similar words have a high similarity measure in that space [20].

Fig. 3. The overall architecture of the proposed aspect-based sentiment analysis (ABSA), with the research utilized two Arabic pre-trained word embedding models.

With the creation of the Word2Vec toolkit, Mikolov et al. [15] has introduced the idea and brought considerable attention to the research community contributing to word embedding widespread use for it being easy to implement and tuned generating embeddings. Word2Vec models are shallow two-layer neural networks that have been presented for taking as input a large corpus of text to efficiently build word embeddings in a vector space where semantically similar words in the corpus have been assigned a corresponding vector in the space. Word2Vec use a probabilistic prediction approach which captures syntactic and semantic word relationships from huge datasets. The skip-gram model as seen in Fig. 4, was introduced by [15] for representing words in a multidimensional vector space as it predicts the surrounding context words given the center word.

Fig. 4. The Skip-gram models [15]

Since we have the entire corpus of data labeled upfront, thus, we run Word2Vec to generate word vectors and then use them as input representation of words to our SVM classifier. Feature vectors were generated for the input document of labeled airlines tweets dataset as a bag of words. With the mapping of the n-grams is being used to generate features with the model word vector embeddings being associated to each n-gram obtaining the simple vector-based features. We were able to load and utilize the word embedding models using the gensim [21] open-source vector space modeling toolkit implemented in Python; the applied two models are:

a) AraVec, Web skip-gram pre-trained word embedding model AraVec [3] is an Arabic language pre-trained distributed word embedding models where it was produced using the Word2Vec skip-gram technique trained on World Wide Web pages Arabic content with a dimension of 300 and a vocabulary size of 145,428. b) fastText Arabic Wikipedia skip-gram pre-trained word embeddings model. This model [2] was trained on top of Wikipedia Arabic articles using fastText with a vector embeddings of 300 and vocabulary size of 610,977. FastText is a library for learning of word embeddings created by Facebook's AI Research lab. [2] proposed a new approach based on the Word2Vec Skip-gram and CBOW architectures, where each word is represented as a bag of character n-grams to learn word representations by considering subword information [2]. 1.1 Aspect Detection We trained a supervised SVM linear classifier using simple vector-based features on the labeled 5k tweets dataset to train it on unseen tweets to predict on which aspects of airline service they fall. Having a multiclass dataset, it required a multiclass classification task which refers to classifying data with more than two classes. Here, it assumes that each tweet is assigned to one and only one of the aspect classes, and not to two aspect classes at once. Therefore, we applied a one-vs-rest (one-vs-all) Support Vector Machine classifier for Aspect detection sub-task of ABSA. It is about fitting one classifier per class, and for each classifier, the class is fitted against all the other classes, which means training a classifier for each possible class. Hence, since only one classifier is representing each class solely, gaining knowledge regarding each class was feasible through inspecting its corresponding classifier. Then, using the Word2Vec model, the target word/label is the aspect category and that words in the tweet represented through the word vectors are the input to the classifier. In the simple setting, if one tweet contains the words, for example, "‫ "تأخرت رحلتي للرياض‬with "‫ "الرياض‬or Riyadh being the Arabic word for Saudi Arabia capital city. We use pre-trained word embeddings so that during training of our classifier, every word's feature vector is its Word2Vec or fastText word embeddings' vector. Therefore, during testing when a new word such as "‫"الدمام‬or Dammam another Saudi city is inferred which may or may not have occurred in training. We set its feature vector to its Word2Vec vector and predict its class since it is similar to "‫"الرياض‬, both are locations and cities. We can expect the SVM classifier to capture the features that are more specifically indicative of any Airline service aspect class or polarity class and predict words in a tweet being similar and more common in which class for both the Aspect Detection or the Aspect-Based Sentiment Classification sub-tasks of ABSA. 1.1 Aspect-Based Sentiment Detection We used linear support vector classification for the aspect-based sentiment classification sub-task being a binary classification task. Every collected and labeled tweet consists of not only aspect specific word but also many positive and negative emotional words as these since we are tackling an ABSA problem, the aspects in the tweets were labeled with their sentiment polarity. That is, each aspect classes was manually labeled according to their sentiment polarity having either positive or negative but not neutral polarity. Using the Word2Vec, we can classify similar polarity words for both polarity classes as being pre-trained in the word embedding models since it represents semantically similar words a corresponding vector in a vector space. As such simple vector-based features capture both semantic and sentiment information and this has improved the sentiment classification task, as seen in Table 5. In other words, the sentiment words are computed automatically through the selection of nearest word vectors neighboring the vectors of the word “‫ ”ممتاز‬for positive or “‫ ”سيء‬for negative seeds, for example, see in Table 3. For both parts of the study aspect detection and the aspect-based sentiment detection, we trained the SVM classifier on the produced training vectors along with the training labels. Then predicted and tested the classifier with the test vectors only for it to predict the labels for the test vectors to see how well it will generalize to measure its performance as a last step in the process as shown in Fig. 3.

Table 3. The fastText model cosine distance of words closest to the sentiment expressing words "‫"ممتاز‬and "‫" سيء‬. Word Cosine Distance ‫ممتاز‬ ‫وممتاز‬ ‫جيد‬ ‫وجيد‬ ‫رائع‬ ‫وراقي‬ ‫ومريح‬ ‫ورائع‬ ‫مميز‬ ‫ممتازين‬ ‫ممتازه‬

5

0.665062 0.568468 0.549050 0.524772 0.501781 0.499864 0.489222 0.482590 0.481457 0.475665

‫سيء‬ ‫سيئ‬ ‫سئ‬ ‫السيء‬ ‫اسواء‬ ‫لسوء‬ ‫جيد‬ ‫يسيء‬ ‫قبيح‬ ‫رديء‬ ‫سيئا‬

0.714792 0.611662 0.496258 0.485051 0.464293 0.463055 0.461216 0.459320 0.458123 0.456819

Results

We compare the two word embedding models’ performances using various evaluation metrics. Namely, accuracy, precision, recall, and F1 with the macro-averaged precision, recall and F-measure at the bottom of Table 4 and Table 5. In addition to the graphical area under the ROC curve plot for the two models as seen in Fig. 5. The fastText Arabic Wikipedia word embeddings model uses character level information, thus, containing many word forms that are rarely occurring in the training corpus improving word vector representations for Arabic and on the two sub-tasks of ABSA results. Besides, it has four times the vocabulary size as the AraVec model. However, we can see the two models having almost closer results, and that can be due to the nature of Wikipedia having many words that carry a negative or a positive meaning in that in a spoken language having neutral meaning [3]. For the Luggage & Cargo aspect, the aspect-based sentiment polarity detection for the positive class results are zeros, as seen in below Table 5. That is due to the positive polarity class having very minimal examples as compared to the number of negative tweets for this same aspect class. Nevertheless, looking at the same Table 4 and Table 5 to compare the overall performances using the two word embedding models in terms of evaluation metrics. The adopted pre-trained word embeddings model utilized for both ABSA subtasks has performed comparably well in comparison to existing techniques it brings extra semantic features that help in text classification as it relies not only on the word count but also saves all required semantic and syntactic information [15]. However, one limitation that we face was that our 5k tweets dataset was imbalanced with a significant difference in tweets counts between the 13 aspect classes as well as for the positive and negative polarity classes.

Table 4. The experimental results of Aspect Detection sub-tasks using the two word embedding models.

Aspect Schedule Destinations & Routes Luggage & Cargo Staff & Crew Airplane and Seating Frequent Flyer Lounges In-flight Entertainment In-flight Meals & services Booking Services Customer Service Refunds and Compensations Pricing & Offers Misc. avg / total Accuracy

One-vs-the-rest (OvR) multiclass strategy (Linear Support Vector Classification) AraVec, web skip-gram word fastText skip-gram Arabic Wikipedia embeddings model [3] word embeddings model [2] precision recall F1 precision recall F1 0.66 0.74 0.70 0.74 0.84 0.79 0.56 0.45 0.50 0.74 0.62 0.68 0.63 0.63 0.63 0.65 0.56 0.60 0.60 0.58 0.59 0.63 0.71 0.67 0.63 0.69 0.66 0.68 0.74 0.71 0.64 0.39 0.48 0.69 0.50 0.58 0.63 0.59 0.61 0.95 0.69 0.80 0.70 0.51 0.59 0.70 0.58 0.63 0.64 0.67 0.65 0.79 0.70 0.75 0.47 0.51 0.49 0.64 0.61 0.63 0.67 0.51 0.58 0.73 0.59 0.65 0.75 0.68 0.71 0.82 0.75 0.78 0.59 0.65 0.62 0.64 0.70 0.67 0.62 0.62 0.62 0.74 0.84 0.79 0.62 0.70

Table 5. The experimental results of Aspect-based Sentiment Polarity Detection step using the two word embedding models.

Aspect Schedule Destinations & Routes Luggage & Cargo Staff & Crew Airplane and Seating Frequent Flyer Lounges In-flight Entertainment In-flight Meals & services Booking Services Customer Service Refunds Ticket Pricing & Offers Misc. avg / total

Polarity Positive Negative Positive Negative Positive Negative Positive Negative Positive Negative Positive Negative Positive Negative Positive Negative Positive Negative Positive Negative Positive Negative Positive Negative Positive Negative Positive Negative

Linear Support Vector Machines Classifier AraVec, web skip-gram word embeddings fastText skip-gram Arabic Wikipedia model [3] word embeddings model [2] precision recall F1 accuracy precision recall F1 accuracy 0.67 0.33 0.44 0.67 0.22 0.33 0.96 0.94 0.97 0.99 0.98 0.95 0.99 0.97 0.67 0.57 0.62 0.73 0.67 0.70 0.76 0.83 0.79 0.85 0.82 0.87 0.90 0.88 0.00 0.00 0.00 0.00 0.00 0.00 0.98 0.98 0.98 1.00 0.99 0.98 1.00 0.99 0.85 0.87 0.86 0.83 0.93 0.88 0.89 0.93 0.92 0.90 0.91 0.97 0.93 0.95 0.80 0.65 0.71 0.84 0.73 0.78 0.81 0.89 0.81 0.90 0.85 0.91 0.95 0.93 1.00 0.86 0.92 0.86 0.75 0.80 0.94 0.83 0.92 1.00 0.96 0.82 0.90 0.86 0.73 0.92 0.81 0.82 0.93 0.87 0.82 0.86 0.92 0.75 0.83 0.91 0.77 0.83 0.56 0.56 0.56 0.94 0.65 0.77 0.69 0.80 0.76 0.76 0.76 0.72 0.95 0.82 1.00 0.62 0.76 0.83 0.62 0.71 0.90 0.92 0.88 1.00 0.94 0.93 0.98 0.95 0.93 0.78 0.85 0.86 0.67 0.75 0.94 0.90 0.94 0.98 0.96 0.91 0.97 0.94 0.67 0.40 0.50 1.00 0.60 0.75 0.89 0.95 0.91 0.97 0.94 0.94 1.00 0.97 0.80 0.25 0.38 0.88 0.58 0.70 0.78 0.90 0.78 0.98 0.87 0.90 0.98 0.94 0.76 0.80 0.78 0.87 0.81 0.84 0.80 0.84 0.83 0.80 0.81 0.80 0.86 0.83 0.73 0.59 0.63 0.78 0.63 0.68 0.86 0.89 0.88 0.91 0.89 0.89 0.94 0.91

Fig. 5. The overall area under the ROC Curve of Sentiment Polarity Detection applying the two models; left: the AraVec, web skip-gram model, right: fastText Wikipedia skip-gram model.

6

Conclusion

We employed word embeddings to perform Aspect Based Sentiment Analysis on Arabic airline-related tweets. We employed two effective pre-trained word vectors models producing vector-based features and showing their effectiveness for both aspect detection and aspect-based sentiment detection. Our vector space approach using these features and no hand-crafted features performs comparably well in comparison to techniques adopting manually engineered features. Nonetheless, our system results have presented that such word embedding features yet simple are powerful enough to achieve state of the art performance in performing ABSA for the Arabic language. For Future work, we would look at performing multi-label classification ABSA using Twitter where each tweet can fall into one or more classes/labels.

References [1] X. . Ma and E. H. Hovy, "End-to-end Sequence Labeling via Bi-directional LSTM-CNNs-CRF," arXiv: Learning, vol. 1, no. , pp. 1064-1074, 2016.

[2] P. . Bojanowski, E. . Grave, A. . Joulin and T. . Mikolov, "Enriching Word Vectors with Subword Information," Transactions of the Association for Computational Linguistics, vol. 5, no. 0, pp. 135-146, 2017.

[3] A. B. Soliman, K. Eissa and S. . R. El-Beltagy, "AraVec: A set of Arabic Word Embedding Models for use in Arabic NLP," Procedia Computer Science, pp. 256-265, 2017.

[4] M. . Biltawi, W. . Etaiwi, S. . Tedmori, A. . Hudaib and A. . Awajan, "Sentiment classification techniques for Arabic language: A survey," , 2016. [Online]. Available: http://ieeexplore.ieee.org/document/7476075. [Accessed 14 3 2018].

[5] B. . Pang and L. . Lee, Opinion Mining and Sentiment Analysis, ed., vol. , , : Now Publishers Inc, 2008, p. . [6] K. . Schouten and F. . Frasincar, "Survey on Aspect-Level Sentiment Analysis," IEEE Transactions on Knowledge and Data Engineering, vol. 28, no. 3, pp. 813-830, 2016.

[7] S. R. a. A. A. El-Beltagy, "Open issues in the sentiment analysis of Arabic social media: A case study," Innovations in information technology (iit), 2013 9th International Conference, 2013.

[8] A. A. a. L. T. Altowayan, "Word embeddings for Arabic sentiment analysis," in Big Data (Big Data), 2016 IEEE International Conference on. IEEE, 2016.

[9] M. . Abdul-Mageed, M. T. Diab and M. . Korayem, "Subjectivity and Sentiment Analysis of Modern Standard Arabic," , 2011. [Online]. Available: http://dblp.uni-trier.de/db/conf/acl/acl2011s.html. [Accessed 14 3 2018].

[10] M. . Abdul-Mageed, S. . Kuebler and M. T. Diab, "SAMAR: A System for Subjectivity and Sentiment Analysis of Arabic Social Media," , 2012. [Online]. Available: http://dl.acm.org/citation.cfm?id=2392971. [Accessed 14 3 2018].

[11] A. Mourad and K. Darwish, "Subjectivity and sentiment analysis of modern standard arabic and arabic microblogs," in the 4th workshop on computational approaches to subjectivity, sentiment and social media analysis, 2013.

[12] N. A. Abdulla, N. A. Ahmed, M. A. Shehab and M. . Al-Ayyoub, "Arabic sentiment analysis: Lexicon-based and corpus-based," , 2013. [Online]. Available: http://ieeexplore.ieee.org/xpls/abs_all.jsp?arnumber=6716448. [Accessed 14 3 2018].

[13] R. Bouchlaghem, A. Elkhelifi and R. Faiz, "A Machine Learning Approach For Classifying Sentiments in Arabic tweets," in Proceedings of the 6th International Conference on Web Intelligence, Mining and Semantics. ACM., 2016.

[14] Y. R. W. a. D. J. Ren, "A topic-enhanced word embedding for twitter sentiment classification," Information Sciences, vol. 369 , pp. 188-198, 2016.

[15] T. . Mikolov, K. . Chen, G. . Corrado and J. . Dean, "Efficient Estimation of Word Representations in Vector Space," arXiv: Computation and Language, vol. , no. , p. , 2013.

[16] A. a. M. D. Zirikly, "Named entity recognition for arabic social media," in Proceedings of the 1st Workshop on Vector Space Modeling for Natural Language Processing, 2015.

[17] K. . Darwish, "Named Entity Recognition using Cross-lingual Resources: Arabic as an Example," , 2013. [Online]. Available: http://dblp.uni-trier.de/db/conf/acl/acl2013-1.html. [Accessed 15 3 2018].

[18] S. Al-Azani and E.-S. M. El-Alfy, "Using word embedding and ensemble learning for highly imbalanced data sentiment analysis in short arabic text.," Procedia Computer Science, pp. 359-366, 2017.

[19] A. Alghunaim, "A vector space approach for aspect-based sentiment analysis," Diss. Massachusetts Institute of Technology, 2015.

[20] P. D. Turney and P. . Pantel, "From frequency to meaning: vector space models of semantics," Journal of Artificial Intelligence Research, vol. 37, no. 1, pp. 141-188, 2010.

[21] R. Rehurek and P. Sojka, "Software framework for topic modelling with large corpora," in the LREC 2010 Workshop on New Challenges for NLP Frameworks, 2010.