Overview of the 2014 ALTA Shared Task - Identifying ...

Overview of the 2014 ALTA Shared Task Identifying Expressions of Locations in Tweets

Diego Moll´a

Sarvnaz Karimi

Macquarie University

CSIRO

ALTA 2014, Melbourne, Australia

The 2013 Task

The Tweet Data

Kaggle in Class

Evaluation Results

Contents

The 2014 ALTA Shared Task The Tweet Data Kaggle in Class Evaluation Results

2014 ALTA Shared Task

Diego Moll´ a, Sarvnaz Karimi

2/21

The 2013 Task

The Tweet Data

Kaggle in Class

Evaluation Results

Contents

The 2014 ALTA Shared Task The Tweet Data Kaggle in Class Evaluation Results

2014 ALTA Shared Task

Diego Moll´ a, Sarvnaz Karimi

3/21

The 2013 Task

The Tweet Data

Kaggle in Class

Evaluation Results

The 2014 Shared Task Task: Identify Expressions of Locations in Tweets Categories: student, open Prize: $500 (IBM Research Shared Task Student Prize) Framework: Kaggle in Class

Student Category

Open Category

I

All members are university students.

I

No members are full-time employed.

I

No members have a PhD. 2014 ALTA Shared Task

I

Any other teams.

Diego Moll´ a, Sarvnaz Karimi

4/21

The 2013 Task

The Tweet Data

Kaggle in Class

Evaluation Results

Identify Expressions of Locations in Tweets Tweet France and Germany join the US and UK in advising their nationals in Libya to leave immediately http://bbc.in/1rVmrDJ Dutch investigators not going to MH17 crash site in eastern Ukraine due to security concerns, OSCE monitors say Seeing early signs of potential flash flooding with stationary storms near St. Marys, Tavistock, Cambridge #onstorm pic.twitter.com/BtogIxgQ5G

2014 ALTA Shared Task

Location France, Germany, US, UK, Libya MH17 crash site, eastern Ukraine St. Marys, Tavistock, Cambridge

Diego Moll´ a, Sarvnaz Karimi

5/21

The 2013 Task

The Tweet Data

Kaggle in Class

Evaluation Results

Motivation 1. When people discuss events, often they mention the location. 2. In the case of emergencies, such locations are very useful. 3. Recommender systems can use location information to improve their recommendations.

2014 ALTA Shared Task

Diego Moll´ a, Sarvnaz Karimi

6/21

The 2013 Task

The Tweet Data

Kaggle in Class

Evaluation Results

Motivation 1. When people discuss events, often they mention the location. 2. In the case of emergencies, such locations are very useful. 3. Recommender systems can use location information to improve their recommendations.

http://rt.com/usa/new-jersey-flooded-sandy-575/

2014 ALTA Shared Task

Diego Moll´ a, Sarvnaz Karimi

6/21

The 2013 Task

The Tweet Data

Kaggle in Class

Evaluation Results

Motivation 1. When people discuss events, often they mention the location. 2. In the case of emergencies, such locations are very useful. 3. Recommender systems can use location information to improve their recommendations.

http://static.echonest.com/DukeListens/event_mapping_at_last_fm.html

2014 ALTA Shared Task

Diego Moll´ a, Sarvnaz Karimi

6/21

The 2013 Task

The Tweet Data

Kaggle in Class

Evaluation Results

Location Expressions in Tweets What is a location? Any specific mention of a country, city, suburb, or POI. I

Macquarie Centre.

I

Ryde Hospital.

Where can we find location mentions? I

In the text.

I

In hashtags: #Australia.

I

In URLs: http://abc.net.au/melbourne/.

I

In mentions: @Australia.

2014 ALTA Shared Task

Diego Moll´ a, Sarvnaz Karimi

7/21

The 2013 Task

The Tweet Data

Kaggle in Class

Evaluation Results

Location Expressions in Tweets What is a location? Any specific mention of a country, city, suburb, or POI. I

Macquarie Centre.

I

Ryde Hospital.

Where can we find location mentions? I

In the text.

I

In hashtags: #Australia.

I

In URLs: http://abc.net.au/melbourne/.

I

In mentions: @Australia.

2014 ALTA Shared Task

Diego Moll´ a, Sarvnaz Karimi

7/21

The 2013 Task

The Tweet Data

Kaggle in Class

Evaluation Results

Related Work Named entity recognition in Twitter I

LabelledLDA for NER and PoS on tweets (Ritter et al. 2011).

I

TwiNER: Unsupervised, using external sources (e.g. Wikipedia) for NER on tweets (Li et al. 2012).

Location extraction I

Twitcident: Using NER to identify location information on tweets (Abel et al. 2012).

I

Ensemble classifiers to predict home locations of tweets (Mahmud et al. 2012).

I

NER tools, used out of the box vs. re-trained on tweets (Lingad et al. 2013). 2014 ALTA Shared Task

Diego Moll´ a, Sarvnaz Karimi

8/21

The 2013 Task

The Tweet Data

Kaggle in Class

Evaluation Results

Related Work Named entity recognition in Twitter I

LabelledLDA for NER and PoS on tweets (Ritter et al. 2011).

I

TwiNER: Unsupervised, using external sources (e.g. Wikipedia) for NER on tweets (Li et al. 2012).

Location extraction I

Twitcident: Using NER to identify location information on tweets (Abel et al. 2012).

I

Ensemble classifiers to predict home locations of tweets (Mahmud et al. 2012).

I

NER tools, used out of the box vs. re-trained on tweets (Lingad et al. 2013). 2014 ALTA Shared Task

Diego Moll´ a, Sarvnaz Karimi

8/21

The 2013 Task

The Tweet Data

Kaggle in Class

Evaluation Results

Contents

The 2014 ALTA Shared Task The Tweet Data Kaggle in Class Evaluation Results

2014 ALTA Shared Task

Diego Moll´ a, Sarvnaz Karimi

9/21

The 2013 Task

The Tweet Data

Kaggle in Class

Evaluation Results

Tweet Collection Source I

From Lingad et al. (2013).

I

Tweets from late 2010 to late 2012.

I

Augmented with additional tweets.

I

Several annotations, only location mentions were used for the ALTA shared task.

Size I

Originally, 3,220 tweets.

I

Available for the ALTA shared task: 3,047.

I

After removing duplicates: 3,003. 2014 ALTA Shared Task

Diego Moll´ a, Sarvnaz Karimi

10/21

The 2013 Task

The Tweet Data

Kaggle in Class

Evaluation Results

Data Contents Data for training and development I

Tweet IDs.

I

Location mentions.

I

Tweet download script.

Copyright restrictions I

Twitter does not allow the distribution of tweets.

I

The shared task participants were asked to download the tweets themselves.

I

Depending on the network status and changes by Twitter and Twitter users, specific tweets might not be available for download. 2014 ALTA Shared Task

Diego Moll´ a, Sarvnaz Karimi

11/21

The 2013 Task

The Tweet Data

Kaggle in Class

Evaluation Results

Data Format Format of location mentions I

All multi-word terms split into their single words.

I

Word duplicates are numbered.

I

All punctuation marks are removed, including #.

I

Words are lowercased.

I

Data in a CSV file.

Examples I

Tweet ID1, france germany us uk libya

I

Tweet ID2, australia australia2 australia3 2014 ALTA Shared Task

Diego Moll´ a, Sarvnaz Karimi

12/21

The 2013 Task

The Tweet Data

Kaggle in Class

Evaluation Results

Contents

The 2014 ALTA Shared Task The Tweet Data Kaggle in Class Evaluation Results

2014 ALTA Shared Task

Diego Moll´ a, Sarvnaz Karimi

13/21

The 2013 Task

The Tweet Data

Kaggle in Class

Evaluation Results

Kaggle in Class Kaggle I

Kaggle offers a Web-based framework for data-driven competitions.

I

A large base of potential participants.

I

Potentially large prizes for the participants.

I

Fee-based for the organisers; free for the participants.

Kaggle in Class I

Free for organisers and participants.

I

Limited user support by Kaggle.

I

Used by course-based competitions. 2014 ALTA Shared Task

Diego Moll´ a, Sarvnaz Karimi

14/21

The 2013 Task

The Tweet Data

Kaggle in Class

Evaluation Results

Alta Shared Task in Kaggle in Class

2014 ALTA Shared Task

Diego Moll´ a, Sarvnaz Karimi

15/21

The 2013 Task

The Tweet Data

Kaggle in Class

Evaluation Results

Alta Shared Task in Kaggle in Class

2014 ALTA Shared Task

Diego Moll´ a, Sarvnaz Karimi

16/21

The 2013 Task

The Tweet Data

Kaggle in Class

Evaluation Results

Features of Kaggle in Class

I

Public leaderboard: all participants can submit and compare with other participants.

I

Automated evaluation: organisers can choose among several evaluation metrics. Public and private partitions: A private partition of the test data is held private for the final ranking

I

I I

I

Public: 501 tweets. Private: 502 tweets.

Discussion forum: for communication among participants.

2014 ALTA Shared Task

Diego Moll´ a, Sarvnaz Karimi

17/21

The 2013 Task

The Tweet Data

Kaggle in Class

Evaluation Results

Contents

The 2014 ALTA Shared Task The Tweet Data Kaggle in Class Evaluation Results

2014 ALTA Shared Task

Diego Moll´ a, Sarvnaz Karimi

18/21

The 2013 Task

The Tweet Data

Kaggle in Class

Evaluation Results

Evaluation Metric Mean F1-Score I

Compute recall and precision of each individual word.

I

This allows evaluation of partially correct location mentions. pr F1 = 2 p+r

Example I

Target: senegal senegal2

I

System output: senegal christchurch brighton

I

p = 1/3

I

r = 1/2

I

F 1 = 0.4 2014 ALTA Shared Task

Diego Moll´ a, Sarvnaz Karimi

19/21

The 2013 Task

The Tweet Data

Kaggle in Class

Evaluation Results

Conclusions

Conclusions I I

Kaggle in class, a useful means to run the shared task. Few participants, but very active. I

I

168 runs in the combined 4 teams.

Participants (read the Proceedings!) used a combination of: 1. sequence labellers, 2. feature engineering, and 3. combined classifiers.

2014 ALTA Shared Task

Diego Moll´ a, Sarvnaz Karimi

20/21

The 2013 Task

The Tweet Data

Kaggle in Class

Evaluation Results

Results

Team MQ AUT NLP Yarra JK Rowling

Category Student Open Student Open

2014 ALTA Shared Task

Public 0.781 0.748 0.768 0.751

Private 0.792 0.747 0.732 0.726

Diego Moll´ a, Sarvnaz Karimi

21/21