Validating search protocols for mining of health and disease events on Twitter Aditya Lia Ramadona1,2*, Lutfan Lazuardi3, Sulistyawati1,4, Anwar Dwi Cahyono5, Åsa Holmner6, Hari Kusnanto3, Joacim Rocklöv1 Department of Public Health and Clinical Medicine, Epidemiology and Global Health, Umeå University, Umeå, Sweden 2 Center for Environmental Studies, Universitas Gadjah Mada, Yogyakarta, Indonesia 3 Department of Public Health, Faculty of Medicine, Universitas Gadjah Mada, Yogyakarta, Indonesia 4 Department of Public Health, Universitas Ahmad Dahlan, Yogyakarta, Indonesia 5 District Health Office, Yogyakarta, Indonesia 6 Department of Radiation Sciences, Umeå University, Umeå, Sweden 1
principal author Phone: +6285229394370, E-mail:
[email protected] (AR) *
Abstract Background: Twitter is a free social networking and micro-blogging service that enables its users to read and share information with user and media communities in messages no longer than 140-character. In the year of 2016, there were more than 24 million Indonesian twitter users sharing news, events, as well as personal feelings and experiences on Twitter. This study seeks to validate a search protocol of health related terms using real-time Twitter data which can later be used to understand if, and how, twitter can reveal information on the current health situation in Indonesia. In this validation study of mining protocols, we: 1) extracted geo-located conversations related to health and disease postings on Twitter using a set of pre-defined keywords, 2) assessed the prevalence, frequency and timing of such content in these conversations, and 3) validated how this search protocol was able to detect relevant disease tweets. Subjects and Methods: Groups of words and phrases relevant to disease symptoms and health outcomes were used in a protocol developed in the 1
Indonesian language in order to extract relevant content from geo-tagged Twitter feeds. A supervised learning algorithm using Classification and Regression Tree´s (CART) was used to validate search protocols of disease and health hits comparing to those identified by a team of human experts. The experts categorized tweets as positive or negative in respect to health events. The model fit was evaluated based on prediction performance. Results: We observed 390 tweets from historical Twitter feeds and 1,145,649 tweets from Twitter stream feeds during the period July 26th to August 1st, 2016. Only twitter hits with health related keywords in the Indonesian language were obtained. The accuracy of predictions of mined hits versus expert validated hits using the CART algorithm showed good validity with AUC beyond 0.8. Conclusion: Our study shows that monitoring of public sentiment on Twitter, combined with contextual knowledge about the disease, can detect health and disease tweets and potentially be used as a valuable real-time proxy for health events over space and time. Keywords: social networking; disease detection; disease early warning; digital epidemiology; big data analytics
2
Background Twitter is a free social networking and micro-blogging service that enables its users to read and share messages no longer than 140-characters. As of May 2016, there are 24.34 million Indonesian, or around 10% of population being active monthly on Twitter (Statista 2016), sharing news, events, as well as their personal feelings and experiences including health-related information. Twitter offers a potential for data mining of public information flows (Twitter 2016b). These massive data sources may be exploited for public health monitoring and surveillance purposes (Paul et al. 2016). Previous studies have explored the use of Twitter, for example, to track levels of disease activity (Signorini et al. 2011), to predicts heart disease mortality (Eichstaedt et al. 2015), and for measuring health-related quality of life (Strom et al. 2013). However, the validity of twitter mining protocols to correctly detect health and disease events is one methodological challenges of this media. This study seeks to validate a search protocol of health related terms using real-time Twitter data which can later be used to understand if, and how, twitter can reveal information on the current health situation in Indonesia. In this validation study of mining protocols, we: 1) extracted geo-located conversations related to health and disease postings on Twitter using a set of pre-defined keywords, 2) assessed the prevalence, frequency and timing of such content in these conversations, and 3) validated how this search protocol was able to detect relevant disease tweets.
Subjects and Methods Groups of words and phrases relevant to disease symptoms and health outcomes were used in a protocol developed in the Indonesian language in order to extract relevant content from geo-tagged Twitter feeds. The search terms were run through a correlation model to reveal relationships. A supervised learning algorithm using Classification and Regression Tree´s (CART) was used to validate search protocols of disease and health hits comparing to those identified by a team of human experts. The experts categorized tweets as positive or negative in respect to health events. The model fit was evaluated 3
based on prediction performance according to AUC, and positive and negative predicted value.
Results Text Analysis Using Twitter Searches API (Twitter 2016a), we filtered tweets in the Indonesian language (Bahasa) using health risk-related keywords (i.e., "rumah OR sakit OR rawat OR inap OR demam OR panas -cuaca OR berdarah OR pendarahan OR tombosit OR badan OR muntah OR badan OR tua OR ':('"). Following some development and modification of the mining protocol, such as removing retweets and eliminate some noise, we collected 390 tweets in the Bahasa. Furthermore, we obtained 1,632 words from such tweets after removing punctuation, numbers, capitalization, and the Bahasa stop-words (e.g. kamu and aja). Example of text preprocessing [107] "@XYZ kamu izin aja, bilang kamu sakit :(("
Many words in the protocol appeared infrequently and we considered only the 22 highest words frequencies, namely words that appear at least 10 times (Figure 1). We found that hati (heart), rasa (feel) and perut (stomach) are the most highly correlated words with sakit (sick, ill, pain), with a correlation of 0.23, 0.13 and 0.12 respectively. Other words showed a low correlation to sakit (correlation
