Annotating Relations between Named Entities with ...

Annotating Relations between Named Entities with Crowdsourcing Sandra Collovini1 , Bolivar Pereira1 , Henrique D. P. dos Santos1 and Renata Vieira1 [email protected], [email protected], [email protected], [email protected] 1

Pontif´ıcia Universidade Cat´ olica do Rio Grande do Sul Porto Alegre – Rio Grande do Sul – Brazil

Abstract. In this paper, we describe how the CrowdFlower platform was used to build an annotated corpus for Relation Extraction. The obtained data provides information on the relations between named entities in Portuguese texts. Keywords: Crowdsourcing, Semantic relations annotation, Portuguese

1

Introduction

The task of extracting relations is usually given as a problem in which sentences that have already been annotated with entity mentions are additionally annotated with relations between pairs of those mentioned entities. In general, the performance of this task is measured against gold datasets, such as ACE 2008 RDC1 for English, and HAREM2 for Portuguese. These gold collections were created through a manual annotation process based on guidelines, which took extensive time and effort to be developed by experts. To tackle this problem, Crowdsourcing platforms became an alternative, broadly used by researchers [11], to build annotated corpus. Amazon Mechanical Turk and Crowdflower are some of the tools that could be used to handle the annotation task. Those systems are able to publish a task to hundreds of annotators spread through the globe, in several countries [10], to solve problems manually. In this work, we describe an annotation task using Crowdsourcing, which consists of annotating any type of semantic relations between pairs of Named Entities (NE) in sentences of Portuguese text, based on the proposed annotation instructions. This work is organized as follows: In Section 2, we present the background. The task description is detailed in Section 3. In Section 4, we discuss the results. One of the contributions of this work is briefly presented in Section 5. Finally, Section 6 presents the conclusions and future works. 1 2

http://projects.ldc.upenn.edu/ace http://www.linguateca.pt/harem

2

2

Sandra Collovini, Bolivar Pereira, Henrique D.P. Santos and Renata Vieira

Background

Relation Extraction (RE) is the task of identifying and classifying the semantic relations between entities from natural language texts, such as placement (Organization, Place) or affiliation (Person, Organization). It focuses on extracting structured relations from unstructured sources using different approaches. Thus, depending on the application and on the resources available, the RE task can be studied for different settings. Many approaches to RE use supervised machine learning, but these methods require human-annotated training datasets that may be unavailable [12]. For Portuguese, there are few annotated data for RE compared to other languages such as English [1]. One of the obstacles to create high quality annotated data is the lack of detailed guidelines for executing manual annotation of relations. Unfortunately, for works in Portuguese, it is not possible to use resources and databases developed for English. Also, a major difficulty is the availability of experts to perform the annotation. One way to address these problems is to apply Crowdsourcing3 annotation to RE. The basic workflow of Crowdsourcing in all platforms is similar. First, a requester (a human or computer) creates a task for workers to complete and posts that task on a platform of its choice. Generally, a requester also specifies certain characteristics a worker must meet to perform the task. Next, workers find the task, complete it, and return the results to the requester. If the results meet a requester’s approval criteria, it compensates the worker. Many NLP tasks have successfully used Crowdsourcing approaches, such as Named Entities [9], Relation Extraction [12], Ontology [14], Text Classification, Sentiment Analysis [13], Topic Detection [8], among others. However, these works treat languages other than Portuguese. The present work has the challenge of applying a Crowdsourcing platform to build annotated corpus quickly and efficiently from Portuguese texts, aiming at the extraction of semantic relations between named entities. There are online platforms that enable crowdsourcing, such as Amazon’s Mechanical Turk (MTurk)4 and CrowdFlower5 .

3

Task Description

In this work, we describe an annotation task of semantic relations between named entities of Summ-it++ corpus [3], using the CrowdFlower platform. Therefore, annotation instructions were developed to serve as a guide for annotators, as well as test questions to evaluate the annotators’ knowledge for the proposed task. The input data and the construction/design of the job in CrowdFlower platform are presented below. 3

4 5

Crowdsourcing has been defined as the act of taking a job traditionally performed by a designated agent and outsourcing it to an undefined, generally large group of people in the form of an open call. http://www.mturk.com/mturk http://www.crowdflower.com

Annotating Relations between Named Entities with Crowdsourcing

3.1

3

Annotation Instructions

We provided the workers with annotation instructions containing an overview of the annotation task. In this task, the workers should check whether or not there is an explicit relation located between named entities of Organization, Person or Place in a sentence. If it occurs, the workers should identify all the words that describe the relation. Table 1 shows the example (1), where a placement relation occurs between “Marfinite” (Organization) and “Brasil” (Place), identified by the elements “fica em” (verb + preposition). The relations identified in the sentence are represented as a triple: (NE1, relation, NE2), in the case of this example we have the triple (Marfinite, fica em, Brasil). If a relation does not occur, one must inform the incident, as in example (2) of Table 1 which shows no relation between “Turquia” (Place) and “Pent´ agono” (Organization). In general, the annotators should follow these instructions: – Annotate only the words that occur between the pair of named entities in the sentence (see example (1) in Table 1); – Annotate the smallest number of elements required to describe the relation, as in example (3) of Table 1, where “abre perspectivas em” (open perspectives in) is sufficient to express the relation between ”Marfinite” (Organization) and “Brasil” (Place). We highlight the difficulty to determine which elements between named entities are in fact part of the relation [7]. Thus, our guidelines were described as clearly as possible. The list of elements that form a relation and illustrative examples are presented below. – Relations must be delimited/considered up to the preposition, if it occurs. However they are dismembered in preposition (”de”, “em”) plus article (”o”, ”a”), and the article should not be included (see example (4) in Table 1); – Relations composed by nouns, such as nouns expressing titles/jobs (see example (5) in Table 1); – Relations composed by verbs (predicates of the sentence), as in example (6) of Table 1; – Relations composed only by preposition, as in example (7) of Table 1; – Relations formed by punctuation such as: parentheses, dashes, commas etc. (see example (8) in Table 1); There are elements that should not be included in the relation, such as adjectives: “exerc´ıcio”, “excelente” (see example (9) in Table 1); and pronouns: “seu”, “sua” (see example (10) in Table 1). 3.2

Dataset

In order to accomplish the task, we used the Summ-it++ corpus [3], which originated from the Summ-it corpus [5]. Summ-it is one of the first corpora of the Portuguese Language gathering annotations from various levels of discourse.

4

Sandra Collovini, Bolivar Pereira, Henrique D.P. Santos and Renata Vieira Table 1. Input examples.

Examples (1) A Marfinite fica em o Brasil. (Marfinite is located in Brasil.) Relation: fica em (located in) Triple: (Marfinite, fica em, Brasil) (2) Os aparelhos regressaram a ` base na Turquia, acrescenta o comunicado do Pent´ agono. (The equipment returned to the base in Turquia, added the statement of Pent´ agono.) Relation: no relation (3) A Marfinite abre perspectivas de neg´ ocios atrav´es de novos distribuidores em o Brasil. (A Marfinite opens business perspectives through new distributors in Brasil). Relation: abre perspectivas em (opens perspectives in) Triple: (Marfinite, abre perspectivas em, Brasil) (4) Ronaldo Goldone continua atuando em as atividades de o Niter´ oi Rugby. (Ronaldo Goldone continues to work in the activities of Niter´ oi Rugby.) Relation: atuando em (to work in) Triple: (Ronaldo Goldone, atuando em, Niter´ oi Rugby) (5) Hugo Dom´ enech, professor de a Universidade Jaume de Castell´ on. (Hugo Dom´ enech, teacher of Universidade Jaume de Castell´ on.) Relation: professor de (teacher of) Triple: (Hugo Dom´enech, professor de, Universidade Jaume de Castell´ on) (6) Em 1956, Am´ılcar Cabral criou o Partido Africano. (In 1956, Am´ılcar Cabral created the Partido Africano.) Relation: criou (created) Triple: (Am´ılcar Cabral, criou, Partido Africano) (7) Ant´ onio Fontes de a AIPAN. (Ant´ onio Fontes of the AIPAN.) Relation: de (of) Triple: (Ant´ onio Fontes, de, AIPAN) (8) A USP (Universidade de S˜ ao Paulo) aprovou a iniciativa dos alunos. (USP (University of S˜ ao Paulo) approved the students’ initiative.) Relation: ( ao Paulo) Triple: (USP, ( , Universidade de S˜ (9) O Presidente em exerc´ıcio de o Conselho. (The current Presidente of the Conselho.) Relation: de (of) Triple: (Presidente, de, Conselho) (10) A Legi˜ ao da Boa Vontade comemora o anivers´ ario da sua implanta¸ca ˜o em Portugal. (Legi˜ ao da Boa Vontade celebrates the birthday of its establishment in Portugal.) Relation: implanta¸ca ˜o em ( establishment in) Triple: (Legi˜ ao da Boa Vontade, implanta¸ca ˜o em, Portugal)

Summ-it++ consists of fifty journalistic texts from the Science section of the Folha de S˜ ao Paulo newspaper and has the following annotations: morphosyntactic annotation provided by CoGrOO6 ; coreference annotation from original Summ-it [5]; named entities recognition by system NERP-CRF [2]; and annotation of some relations between pairs of named entities (Organization and Person 6

http://cogroo.sourceforge.net/

Annotating Relations between Named Entities with Crowdsourcing

5

or Place) by system RelP [7]. These last annotations (named entities and relations) were automatically annotated and manually revised by humans. 3.3

Pre-annotation

In the context of this work, we selected 3 representative texts from the Summit++ corpus to be annotated manually by 3 annotators (students of Computer Science). These 3 texts resulted in 43 selected sentences, that is, sentences containing the pair of named entities (Organization and Person or Place). To measure the agreement of annotation between the 3 annotators, we calculated the kappa coefficient [4] which reached the result “K = 0.625” - “substantial agreement”, resulting in the annotation of 26 sentences containing relations between named entities in focus and 17 without a relation. These gold sentences served as test questions for the training/evaluation of annotators in the CrowdFlower platform, presented in Section 3.4. 3.4

Annotation with CrowdFlower

The CrowdFlower platform was used to perform our annotation task, which involved 50 texts from Summ-it++. In order to develop the job, it was necessary to customize the task for application in the platform. First, the texts were pre-processed to serve as input to the workers, and only the sentences composed by the pair of named entities in focus were selected. However, each input sentence can have only one pair of entities (parameters). Because of that, if a sentence had more than one pair, the number of times it was selected equals it’s number of pairs. This process of sentence selection produced an output of 243 sentences/input question. In addition, we selected 10 gold sentences for test questions taken from 3 texts of Summ-it ++ corpus (see Section 3.3). Along the course of the task, we turned 9 regular questions that had 100% consensus among the workers into test questions, resulting in a total of 19 test questions. To avoid mistakes by the annotator and ensure a quality annotation, we develop some features at CrowdFlower Design Step. Using Javascript/jQuery functions and CML (CrowdFlower Makeup Language), we display the text to the annotator and he is able to click at each word he believe that belong to the relation between the entities. Only the words between the entities are clickable, avoiding annotator mistakes. Those words and its indexes are used in the responses to compute agreement and build the final corpus. For the application of the job, the first step was to present the task to the workers. Figure 1 (a) provides a test question of the task as presented to a worker. We can verify that the named entities involved are highlighted in bold, indicating which parameters are being considered and which part should be analyzed. The worker must click on the words that compose the relation between these highlighted pairs. If there is no relation, the worker must select the option: “Senten¸ca sem rela¸c˜ ao” (no relation). There is also the option “Corrigir” (Correct) to correct the annotation, if necessary. Figure 1 (b) also shows the annotation

6

Sandra Collovini, Bolivar Pereira, Henrique D.P. Santos and Renata Vieira

resulting from the task performed by the worker, where a affiliation relation occurs between “Cassius Vinicius Stevani” (Person) and “USP” (Organization), identified by the elements “qu´ımico de” (chemist of).

Fig. 1. Example task presented to a worker (a), before and (b) after the annotation.

Each input question was annotated by three workers, who should be Brazilian, Portuguese speakers and Level 2 qualified - “Higher Quality: Smaller group of more experienced, higher accuracy contributors”. The 19 test questions were annotated by at least three workers and interspersed with other input questions (243 sentences). If any annotator has a wrong test question, he is immediately notified of it. If one’s accuracy on the test questions falls below 70%, they are rejected and all of their annotations are discarded [13]. However, the worker may comment the test question that has been reported. A total of 55 annotators participated in the CrowdFlower platform job, each one receiving 2 cents per annotation.

4

Results and Discussion

In this Section, we present the result of our task on CrowdFlower platform, which considered the judgments per sentence of each worker. Table 2 shows the agreement between annotators. In the first column, we illustrate the inter-annotator agreement, and in the second column the number of sentences annotated. We can see that 69 sentences reached an agreement between 2 annotators and 167 sentences resulted in an agreement between 3 annotators, in a total of 236 annotated sentences. After, the cases with relations between named entity pairs and the unrelated cases, which obtained agreement between 2 and 3 annotators are quantified in Table 2, respectively. We obtained a total of 127 relations annotated among the named entity pairs in focus and 109 cases containing no relation.

Annotating Relations between Named Entities with Crowdsourcing

7

Table 2. Results of the annotation task. ORG-ORG (relation between Organizations), ORG-PER (relation between Organization and Persons), ORG-PLC (relation between Organization and Places), NO-REL (no relation between the named entities) Annotation Sentences ORG-ORG ORG-PER ORG-PLC NO-REL Agreement - 2 annotators 69 8 14 7 40 Agreement - 3 annotators 167 25 40 33 69 Total 236 33 54 40 109

We measure the quality/reliability of the annotations by the Kappa coefficient, evaluating whether there is a relation or not in each input sentence. For this, 52 input sentences were selected, the biggest subset annotated by the same three workers, resulting in a kappa coefficient “K = 0.815”, considered “almost perfect agreement”.

5

Portuguese Data Enrichment

The resulting data from the annotation task (see Table 2) will be used to enrich an annotated corpus for RE from Portuguese described in [6, 7]. The authors present a subset of the Golden Collections from the two HAREM7 conferences, to which they added manual annotation of relations expressed between particular named entities (ORG, PER or PLC), in a total of 516 relation instances. We unified both corpus and added annotations made in Summ-it ++, totalizing 752 relation instances.

6

Conclusion

In this work we show how to build a task for annotation of semantic relations between named entities (ORG, PER or PLC) for Portuguese language. Also, we make available a corpus to train and evaluate algorithms for RE in Portuguese texts. This corpus is the starting point to enrich the NLP community that research the field of RE. All the content of this work is available in PUCRS-PLN Group website: http://www.inf.pucrs.br/linatural/re-annotation. As future work, we intend to consider other categories of named entities and relations between them in the annotation task. We also intend to apply and evaluate different supervised machine learning techniques for RE to Portuguese using the resulting corpus.

Acknowledgments We thank the CNPQ, CAPES and FAPERGS for their financial support. 7

https://www.linguateca.pt/harem

8

Sandra Collovini, Bolivar Pereira, Henrique D.P. Santos and Renata Vieira

References 1. S. C. Abreu, T. L. Bonamigo, and R. Vieira. A review on relation extraction with an eye on portuguese. Journal of the Brazilian Computer Society, 2013. 2. D. O. F. d. Amaral and R. Vieira. Nerp-crf: uma ferramenta para o reconhecimento de entidades nomeadas por meio de conditional random fields. Linguam´ atica, vol. 6, pages 41–49, 2014. 3. A. Antonitsch, A. Figueira, D. Amaral, E. Fonseca, R. Vieira, and S. Collovini. Summ-it++: an enriched version of the summ-it corpus. In Proceedings of 10th edition of the Language Resources and Evaluation Conference (LREC 2016), 2016. 4. J. Carletta. Assessing agreement on classification tasks: the kappa statistic. Computational linguistics, 22(2):249–254, 1996. 5. S. Collovini, T. I. Carbonel, J. T. Fuchs, J. C. Coelho, L. Rino, and R. Vieira. Summ-it: Um corpus anotado com informa¸co ˜es discursivas visando a sumariza¸ca ˜o autom´ atica. In Proceedings of V Workshop em Tecnologia da Informa¸ca ˜o e da Linguagem Humana , Rio de Janeiro, RJ, Brasil, pages 1605–1614, 2007. 6. S. Collovini, G. Machado, and R. Vieira. Extracting and structuring open relations from portuguese text. In Computational Processing of the Portuguese Language 12th International Conference, PROPOR 2016, volume 9727 of Lecture Notes in Computer Science, pages 153–164, Tomar, Portugal, 2016. Springer. 7. S. Collovini de Abreu and R. Vieira. Relp: Portuguese open relation extraction. Knowledge Organization, 44(3):163–177, 2017. 8. H. D. P. dos Santos, V. Woloszyn, and R. Vieira. Portuguese personal story analysis and detection in blogs. In Web Intelligence (WI), 2017 IEEE/WIC/ACM International Conference on, Leipzig, Germany, August 2017. 9. T. Finin, W. Murnane, A. Karandikar, N. Keller, J. Martineau, and M. Dredze. Annotating named entities in twitter data with crowdsourcing. In Proceedings of the NAACL HLT 2010 Workshop on Creating Speech and Language Data with Amazon’s Mechanical Turk, CSLDAMT ’10, pages 80–88, Stroudsburg, PA, USA, 2010. Association for Computational Linguistics. 10. A. Kittur, E. H. Chi, and B. Suh. Crowdsourcing user studies with mechanical turk. In Proceedings of the SIGCHI conference on human factors in computing systems, pages 453–456. ACM, 2008. 11. P. Kucherbaev, F. Daniel, S. Tranquillini, and M. Marchese. Crowdsourcing processes: a survey of approaches and opportunities. IEEE Internet Computing, 20(2):50–56, 2016. 12. A. Liu, S. Soderland, J. Bragg, C. H. Lin, X. Ling, and D. S. Weld. Effective crowd annotation for relation extraction. In NAACL HLT 2016, The 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 897–906, San Diego California, USA, 2016. 13. S. M. Mohammad and F. Bravo-Marquez. WASSA-2017 shared task on emotion intensity. In Proceedings of the Workshop on Computational Approaches to Subjectivity, Sentiment and Social Media Analysis (WASSA), Copenhagen, Denmark, 2017. 14. J. M. Mortensen, M. A. Musen, and N. F. Noy. Crowdsourcing the verification of relationships in biomedical ontologies. AMIA Annual Symposium Proceedings, 2013:1020–1029, 2013.