big data monitoring and evaluation - UN Global Pulse

BIG DATA MONITORING AND EVALUATION A theoretical framework, tools, and lessons learned from practice

December 2015 (Draft v2.0)

Sally Jackson. Consultant in Monitoring and Evaluation at Pulse Lab Jakarta, United Nations Global Pulse

 

1  

Preface “Big Data” is an umbrella term that refers to the large and ever increasing amounts of digital data that we continually generate as we use digital devices, and to new technology and methods that are now available to analyze large and complex datasets. It is an emerging and experimental field in the development sector, and much of the information that is publicly available about big data for development is ill defined, lacking in detail, and has its basis in theory rather than in practical application. In applying big data for monitoring and evaluation, a wide interdisciplinary gap currently exists: many monitoring and evaluation practitioners do not know what big data is; and many data scientists underestimate the complexities involved in appropriately applying data for decision- making in the development sector. This gap in understanding and lack of information presents a challenge for constructive debate over the potential of big data. There appears be a largely one-sided narrative as neither discipline is in a position to be able to present a balanced argument, but one has an incentive to promote its cause. This report aims to: •

• • • •

Further constructive discussion about what the potential for big data in the development sector is by adding some structure: Concrete definitions, categorizations and frameworks for big data practice in the development sector are proposed. Although these are not likely to be perfect, the hope is that they will provide anchors for debate. Suggest areas where big data is most likely to be successfully and unsuccessfully applied to monitoring and evaluation (defined by type of data, type of question, and stage in the project cycle). Provide basic information about a variety of concepts and tools in the approximate order that they would need to be considered in the project cycle. Describe practical examples of big data projects. Provide some recommendations for future practice.

Overall this report found that the reality for the potential of big data probably sits in between the ‘revolutionary’ views held by the strongest protagonists and the ‘nonsense’ claims expressed by the harshest critics. As with all data there are many limitations in the application of big data. It has potential in a few specific areas but it is by no means a ‘silver bullet’.

 

2  

Content summary This report is structured into six sections. Not every section will be relevant for all readers. A summary of the content and purpose of each section is given below. Pages Section A. An introduction to big data monitoring and evaluation Provides an introduction to the concept of big data for monitoring and evaluation. This aims to improve the understanding of the basics for new practitioners and further constructive discussion about what the potential for big data in the development sector is by adding some structure: Concrete definitions and categorizations for big data practice in the development sector are proposed.

4-9

Section B. Delving a little deeper: thinking through the options for a big data M&E approach Builds on the introduction in section A by introducing some more detailed concepts and some practical examples for consideration by project managers about to embark on a big data project. Emphasizes the importance of a mixed-method approach, and the importance of incorporating contextual information to adequately interpret data.

10-19

Section C. Developing a framework for the application of big data for monitoring and evaluation Describes the methodology that was used to develop a framework for the application of big data in the development sector.

20-22

Section D. A toolkit for big data experimental design Provides tools that may be useful for practitioners in the approximate order of the big data project cycle. These tools will require practical testing by the Pulse Labs to determine their utility in different situations before this section is released as an external toolkit or as recommendations for practice.

23-56

Section E. Can social media analytics provide insights relevant for communicable disease control in Indonesia? A rapid assessment. Due to challenges with access to data sources, all projects conducted at Pulse Lab Jakarta to date have been based on social media analytics. This section describes a rapid assessment that was conducted to assess whether Twitter analytics have any potential to provide information for communicable disease control in Indonesia.

57-72

Section F. Research strategy: some observations from practice at Pulse Lab Jakarta. Observations of systematic issues and systemic challenges facing big data practice at Pulse Lab Jakarta are described in this section, along with suggestions of strategies that may improve research quality and impact

73-75

 

3  

A. An introduction to big data monitoring and evaluation Big data is rising…

…while our existing monitoring and evaluation tools are becoming increasingly inadequate

Digital data is being produced in large and increasingly higher volumes. Many have suggested it has a role in a ‘data revolution’ for sustainable development. As a result of technology, we are producing digital data in higher volumes, and from increasingly diverse sources than we ever have before. We are also able to process that data in new ways and at ever-increasing speeds. “Big Data” is an umbrella term that refers to these large and ever increasing amounts of digital data that we continually generate as we use digital devices, and to new technology and methods that are now available to analyze large and complex datasets. As well as increasing the amount of data that we are producing, this rapid growth in technology has also had another effect. The world is becoming an increasingly interconnected and complex place. An event in one part of the world can have a large and rapid impact in another. The monitoring and evaluation (M&E) tools we currently use are not responsive, interconnected, or adaptive enough to react to this rate of change. As a result of this, many people have called for a ‘data revolution’ for sustainable development defined in Box A1. Box A1. Defining the data revolution Since the phrase was coined in May 2013 in the report of the High-Level Panel of Eminent Persons on the post-2015 Development Agenda, the “data revolution” has come to mean many things to many people. The United Nations Secretary-General’s Independent Expert Advisory Group on a Data Revolution for Sustainable Development takes it to mean the following: The data revolution is: • An explosion in the volume of data, the speed with which data are produced, the number of producers of data, the dissemination of data, and the range of things on which there is data, coming from new technologies such as mobile phones and the “internet of things”, and from other sources, such as qualitative data, citizen-generated data and perceptions data; • A growing demand for data from all parts of society. The data revolution for sustainable development is: • The integration of these new data with traditional data to produce high-quality information that is more detailed, timely and relevant for many purposes and users, especially to foster and monitor sustainable development; • The increase in the usefulness of data through a much greater degree of openness and transparency, avoiding invasion of privacy and abuse of human rights from misuse of data on individuals and groups, and minimizing inequality in production, access and use of data; • Ultimately, more empowered people, better policies, better decisions and greater participation and accountability, leading to better outcomes for people and planet

Source: A world that counts, mobilizing the data revolution for sustainable development

 

4  

Big data is an umbrella term that refers to new digital data sources, technology, and innovative approaches Big data does not just refer to new sources of digital data. It also refers to new technology and innovative approaches: Figure A1. The components of big data

New data sources

Big data

Technology

Big data sources, technologies and innovative approaches have the potential to provide complementary, actionable information for decision-making in the development sector

Innovative approaches

Big data’s new data sources, technology, and innovative approaches may have a role in monitoring and evaluation Decision-makers gain an understanding of a situation through interpreting and extrapolating the pieces of information they have. This information is incomplete, ‘asymmetric’ (decision-makers and beneficiaries have different information to each other) and comes from a combination of diverse sources. M&E aims to: • •

Provide information that closes this knowledge gap with the goal of targeting available resources to the areas of greatest need for beneficiaries Improve accountability Big data sources, technologies and approaches have the potential to provide complementary, actionable information for decision-making in the development sector.

Big data sources, technologies and innovative approaches have the potential to provide complementary, actionable information for decision-making in the development sector.

 

5  

New data sources Digital data sources are diverse and data generated in different ways

Digital data comes from diverse sources and is generated in different ways (Table 1). As it is usually secondary data it is not usually generated for the specific purpose of M&E. Table A1. New data sources Technology users’ actively generated content

Online news sources, publicly accessible blogs, forum posts, comments and public social media content, online advertising, e-commerce sites and websites created by local retailers that list prices and inventory.

Passive generation of signals from digital devices

Transactional data from the use of digital services such as financial services (including purchases, money transfers, savings and loan repayments), communications services (such as anonymized records of mobile phone usage patterns) or information services (such as anonymized records of search queries).

Citizen reporting or crowd-sourced data: information generated by citizens through mobile phone-based surveys, user-generated maps, etc.

Physical sensors: satellite or infrared imagery (e.g. changing landscapes, traffic patterns, light emissions, urban development and topographic changes), sensors on infrastructure (e.g. to monitor usage patterns) etc. Adapted from Global Pulse’s Primer - Big Data for Development In development practice, existing M&E methods use data that is either: • •

They present us with a new type of information – digital signals

Continually, actively, routinely collected through public service systems during implementation Non-routinely, actively collected at intervals in time e.g. through surveys.

New digital data sources present us with a new type of information (as described in Table 1) – digital signals. This refers to digital traces that are continually generated through both active and passive activities. Like routine collection, data is generated continuously. However digital data differs because it is generated outside of public sector information systems (Figure A2). Figure A2. Basic categorization of information subsets

Routine collection (continuous; generated through public sector systems)

Information subset Non-routine collection (noncontinuous; outside of public sector systems)

 

Digital signals (continuous; outside of public sector systems)

6  

Technology New technology enables us to gain new insights from data and process it at faster speeds

New computational techniques, typically using algorithms, enable us to reveal trends, patterns, and correlations in data, and visualize it to turn it into actionable information. Examples include machine learning, data mining, and sensemaking: • • •

Machine learning: a type of artificial intelligence focused on using computers to develop and implement algorithms that can learn from experience i.e. when they are exposed to new data Data mining: an interdisciplinary field (including artificial intelligence, machine learning, statistics, and database systems) that explores large data sets with the aim to discover patterns Sensemaking: an interdisciplinary field that focuses on using intelligent systems to find insights in (make sense of) large amounts of information by interpreting them in context

Advances in technology also enable us to store greater volumes of data, and process it at faster speeds. It requires new approaches

Innovative approaches Approaches to science have adapted over time through people’s drive to understand the world better in different circumstances. When many of the methods that underpin how we currently generate M&E data were developed, they were developed before electronic storage or processing was available, and they were based on active data collection. Data collection frames were designed for purpose, and primary data stored on one medium, in one format. Big data analytic methods enable insights to be gained from across diverse sources. A big data project could mix-and-match data from several different information sub-sets. A funding announcement for a ‘big’ data project may look something like this (Box A2): Box A2. A big data project funding announcement “To combine high-quality data from fisheries, Coast Guard, commercial shipping, and coastal management agencies for a growing data collection that can be used to support a variety of governmental and commercial management studies in the Lower Peninsula” Source: Jules Berman1 Much of ‘big data’ practice involves using data that was generated because of another activity or for another purpose. It is secondary data that already exists so we don’t design the data collection frames, and before we have it we don't know what it contains. Berman1 also gives some comparisons between ‘big’ vs. ‘small’ data. An adapted extract of these is given in Table A2. Table A2. Differences between ‘small’ and ‘big’ data Small data Location* Usually contained on one computer within one organization Data structure Usually highly structured (e.g. and content* spreadsheet) and in one format Data generation

Data collection frames are usually designed for the specific purpose of M&E

Big data Can be spread across multiple Internet servers in any location Diverse. Data can be structured and unstructured (e.g. free text, images, videos, audio) Data is from diverse sources, not designed for the purpose of M&E, and generated by many people * Source: Jules Berman

                                                                                                                1  Berman, Jules J. (2013). Principles of Big Data: preparing, sharing, and analyzing complex information. Elsevier.

   

7  

If you implemented the project described in Box A2, you would probably find that some of the data collection may come from new digital ‘big’ data sources (e.g. financial transaction data), but other data would come from traditional, ‘small’ data sources. As ‘small’ data sources could be included in a big data project, and traditional M&E also utilizes secondary data sources where possible - what is the difference between big and small data? And does the definition matter? The more complex or large the data, the less likely it is that traditional data processing applications can be used. But in reality the ‘big’ vs. ‘small’ data sources distinction is not clear-cut. Data sources fall somewhere along a continuum between simple structured data to complex unstructured data, and depending on the size of the data, or the way its integrated and analyzed, or the questions you want to ask of the data, it can transition from ‘small’ to ‘big’ and vice versa (Box A3). Box A3. From small to big, and big to small Consider a fictional big data project where some data was structured (information in a set format like a spreadsheet) and entered using forms, some data was unstructured free text, and some was financial transaction records. If you analyzed each data type in isolation with traditional methods, it would be considered as ‘small’ data analysis. If you took the data sources and integrated them; then analyzed the data as a whole using techniques that could pull insights from both structured and unstructured information simultaneously, then this could be considered the realm of ‘big’ data analytics. Why? Because it is complex enough that traditional analysis techniques would not be able to analyze the data as a whole. For an example of using ‘big’ data but analyzing it with ‘small’, analytic methods consider the social media analytics for communicable diseases project described in Section E. In this case, a ‘big’ data source was used – the Twitter firehose. Although an algorithmic approach was used to extract and conduct basic sorting of the Tweets, a manual, qualitative research approach was used to analyze the data. This was because a computer was not able to extract meaning from the intricacies of language in context (detect jokes or sarcasm for example). Big’ data therefore cannot be considered a separate entity to ‘small’ data – it is all data and it is not a completely new field of endeavor. In practice, the name of the data (‘big’ or ‘small’) does not matter and when using data to gain insights for development as the same statistical considerations need to be applied regardless of how the data is defined – and this seems to have a tendency to be overlooked by new practitioners. The areas that through subjective observation seem to be frequently overlooked are data quality, inference, causality, and bias. Extensive collaboration is essential

As ‘big’ data projects use diverse data sources and this requires input from multiple organizations and disciplines (including those within the often-untapped private sector), extensive collaboration is required to pull together information from multiple sources, as is a legal and regulatory environment that supports the sharing of data. Because new ‘big’ data sources and analytic methods are emerging, their application for M&E also requires experimentation, and iteration (Figure A3).

 

8  

Figure A3. Iterative loops in the project cycle allow for rapid project adaptation …as is an iterative approach to experimentation

n'1' a4o Iter

Design' Evaluate' and'learn'

Implement' and'monitor'

2' Itera4 4on' a r e t I

Design' Evaluate' and'learn'

on'3'

Design'

Evaluate' and'learn'

Implement' and'monitor' Implement'

and'monitor'

In many ways, traditional M&E project cycles have always involved iteration: a project is designed, implemented, monitored, and evaluated; and then a decision is made to adapt/continue/scale/close it etc. However, big data iterative practice refers to smaller iterations – it can be considered like a series of compressed project cycles within the main frame of a typical project cycle. Iteration expects a project to change course – there is no pre-defined ‘end game’ – so how do we measure success?

This iterative approach expects the project to change course: there is no end game (as you might have with a goal as defined in a log-frame), and the idea is that the project adapts to circumstance. This requires some courageous and innovative funders who are willing to experiment: To put money into a project where the outcome is unknown is not the status quo – how can we measure our success if we don’t know where we are going? But then, is an open question (an experimental project where predefined targets have not been set) more likely to be reciprocated by honest reporting than a project where there is a static target and a ‘right’ or ‘wrong’ answer?

 

9  

B. Delving a little deeper: thinking through the options for a big data M&E approach The way that big data is applied in the development sector is likely to differ from the way it is applied in many commercial organizations Development interventions are likely to have a relatively poorly defined feedback loop when compared to commercial organizations

Incorporation of contextual information using mixed-methods will be essential to make sense of automated analytics…

In many commercial organizations, big data is applied on a closed, relatively simple feedback loop. For example, when looking at shoppers’ transactions, there is a direct connection between product design, sales and revenue: a company has the opportunity to experiment with product specifications which gives rise to positive or negative consumer feedback via increased or decreased revenue. Development interventions address social problems where the relationship between interventions and target beneficiaries is usually more complex. It is likely that any feedback loop between development interventions and beneficiaries will not be so well defined. In addition, many of the variables that are relevant to a development intervention will not be captured by digital data, and where data does exist, it is unlikely that the data ecosystem will be continuous - it will most likely be fragmented: Information is often held in ‘silos’ – isolated units of information – in different formats and different systems that differ both by organization and the units within them. These silos of information don’t just exist on one horizontal ‘layer’ e.g. different units within the local government. They also exist vertically. For example, information from citizens may be held in silos in Civil Society Organizations, but not fed into silos in the local government – and then not linked to silos in the national government. The types of automated analytics that may be appropriate to gain rapid insights for relatively simple feedback loops in the private sector may have some applications in the development sector - an infrastructure sensor for example (the feedback loop is clearly defined). However, in many cases automated analytics will be insufficient in isolation and appropriate interpretation of data will be reliant on incorporating contextual information into the analysis. In these instances it is likely that the use of mixedmethods will be essential to make sense of ‘big data’.

Having a strong theoretical basis behind big data research, and using qualitative techniques to incorporate contextual information for data interpretation will be essential to gain actionable insights

…as without context and a strong theoretical basis, a correlation could be interpreted inappropriately

“Information becomes knowledge only when it is placed in context. Without it, we have no way to differentiate the signal from the noise, and our search for truth might be swamped by false positives” Nate Silver Data scientists look at digital signals, some are which are relevant and some of which are not. Unwanted, irrelevant signals are called noise, and they distract from relevant signal. Often, when noisy data is analyzed, statistical relationships are found in the data, and care has to be taken not to come to the wrong conclusions when they do. Many statistical relationships are spurious: just because variables correlate does not mean that they are causally related. More details about this are given in Box B1.

 

10  

Box B1. Spurious relationships – correlation does not imply causation In statistics, a spurious relationship (not to be confused with spurious correlation) is a mathematical relationship in which two events or variables have no direct causal connection, yet it may be wrongly inferred that they do, due to either coincidence or the presence of a certain third, unseen factor (referred to as a “common response variable”, “confounding factor”, or “lurking variable”. Suppose there is found to be a correlation between A and B, Aside from coincidence, there are three possible relationships: Where A is present, B is observed (A causes B) Where B is present, A is observed (B causes A) OR Where C is present, both A and B are observed (C causes both A and B) In the last case there is a spurious relationship between A and B. In a regression model where A is regressed on B but C is actually the true causal factor for A, this misleading choice of independent variable (B instead of C) is called specification error. Because correlation can arise from the presence of a lurking variable rather than from direct causation it is often said that “correlation does not imply causation” Source: Wikipedia (accessed 09/12/2014) The likelihood of inappropriate data interpretation is amplified if data is interpreted remotely as contextual information can be lost

Using information from qualitative, participatory methods will help incorporate contextual information and close the ‘distance gap’

To separate signal from noise, a strong basis in theory is essential (a hypothesis prior to analysis), as is the incorporation of contextual information for its interpretation. If data is analyzed remotely, the likelihood of making incorrect assumptions will be amplified because of the loss of contextual information - illustrated in Figure B1. Figure B1. Incorrect assumptions

Source: Julie Smith, IFRC M&E Guide Qualitative, participatory methods (like interviews, focus groups, or direct interpretation of the data itself by stakeholders including the community) should facilitate a better understanding of the data – as oppose to interpreting data from a distance. Box B2 contains a project scenario where it was proposed that quantitative big data analytics could be combined with qualitative research methods to gain richer insights on the situation.

 

11  

Box B2. Using quantitative big data analytics methods alongside qualitative methods “To move from data to usable information, data interpretation will be conducted in a participatory manner (e.g. focus groups) with the youth team involved in the project. It is hoped that by engaging youth with props like visualizations of topics discussed on social media, and using qualitative analysis techniques to integrate insights from multiple communication channels including the social media data and the opinions of the youth themselves; that new insights will be gained on youth perspectives around sexual and reproductive health.” Source: Project documentation at Pulse Lab Jakarta

The incorporation of contextual information alongside ‘big data’ is essential to develop a relevant solution Box B3. An anecdote for consideration “I recently rushed to the airport through Jakarta’s Friday night traffic. I set off early and managed to get there well in advance, only to realize I’d gone to the wrong airport. I had two options: to forget about my planned trip and book another flight – or rush through Jakarta’s rush hour traffic with 1 hour 15 minutes until boarding time. Google maps estimated that it would taken 1 hour 30 to get there, the taxi driver outside the terminal said an hour. He seemed so convinced about that, that with little hesitation I jumped in the taxi and we went at surprisingly high speed – until we hit a solid traffic jam exactly where Google maps said we would. In the end, the taxi driver was absolutely right, and to my relief we made it to the airport on time. As I went past queuing traffic, it occurred to me that Google maps didn’t have the local knowledge to account for all the local solutions that the driver had, and employed in certain circumstances. Although it captured the majority of the traffic flow, it didn’t capture the actions of local ‘innovators’ – those who have created solutions to the traffic by using the bus lane, breaking the speed limit, taking side roads, skipping red lights – it definitely didn’t capture all the motorbikes driving on the pavements. All hallmarks of Jakarta’s traffic. So although it could capture the overall flow, it was not able to capture the full reality – the reality that affects individual circumstances.” Source: Pulse Lab Jakarta In this case, although the events described ‘from a distance’ on Google maps were completely correct, they did not reflect the actual reality for an individual in a situation on the ground. Traffic data was presented by Google maps in aggregate, the decision-maker was an individual, and local solutions to the traffic problem existed that were not captured in the data. Big data does not capture the complete picture, but enables you to see patterns in part of it. From just looking at these patterns, it is not possible to fully understand all the complexities of a situation on the ground, understand the local solutions to a problem that may already exist (“necessity is the mother of invention”), or understand how new big data solutions may help individuals facing a given problem on a daily basis. Contextual design is therefore essential and the focus of section D of this report - the ‘toolkit’.

 

 

12  

Mixed-methods approaches are likely to be more robust than one method used independently Information gained from different research methods is complementary

Different types of data have different strengths and weaknesses and one type does not replace the other. Information gained from different sources and methods is complementary. If you need data that is a highly representative sample of your population to answer your questions, then you may choose a household survey. If you want secondary data from an additional communication channel to gain insights on the community perception of an intervention, you might go for Twitter. It is not an either/or scenario and each type of data provides an additional piece of information. Would you look at a jigsaw piece by piece in isolation, when you could put it together and see the big picture? If you imagine a scenario where you want to better understand the mobility of a population: In the particular community you are working in, half the population has mobile phones. Here are two possible ‘purist’ approaches to understanding mobility: 1. The big data purist: Using mobile phone detail records you would be able to monitor mobility over time for half of the people (those with phones) in the population 2. The small data purist: Using a survey (with a sample size that was determined as being the best trade off between information obtained and cost - say 1% of the population), you were able to find out the mobility patterns of representative sample of the population over the past month

Combining information from different sources delivers new insights and makes findings robust…

…Triangulating sources can strengthen data collection systems as a whole

 

 

In making inferences from these samples to the full population the big data purist would face the issue that those with and without phones might move in different ways (people in the two groups are likely to differ socioeconomically and demographically for example). The small data purist would face the limitation that although the sample was representative of the population, it was only a snapshot and was not representative of an extended time period. Issues like seasonality could change mobility in the longer term. Neither method would provide a comprehensive picture of the mobility of the entire population over time. The mixed-methods researcher would be in a win-win situation. ‘Big’ data methods could be used to understand the movement of half of the population in detail. ‘Small’ data methods could be used to answer the specific question of how mobility differed between the group of people with phones, and the group of people without phones (at a smaller sample size and therefore lower cost than a survey representative of the entire population). Through combining the information, an estimate of the movement of the whole population could be made that would be cheaper and more robust over time than either approach taken in isolation. As well as providing new information, comparing new data sources with existing data sources facilitates better validation of information and could be used to develop stronger data collection systems: for example, identifying gaps that require strengthening in data collection systems that function heterogeneously. Triangulation (Figure B2) can be either quantitative e.g. looking at the concordance of new digital data sets existing data sets from routine data collection (e.g. a spreadsheet with outcome frequency data); or qualitative – systematic analysis of several different ‘anecdotal’ information sources.

13  

Figure B2. Triangulation of various sources of information

Source Julie Smith, IFRC M&E Guide

Fast decisions are not necessarily good decisions “Technology magnifies human intent and capacity” Kentaro Toyama Technology facilitates the rapid availability of data but interpreting data appropriately takes time, capacity and willingness

One benefit of big data analytics is that technology facilitates the rapid availability of data. However, data, information, and knowledge are not the same entity. To interpret digital data appropriately and gain information and knowledge from it, contextual information needs to be incorporated - and this takes time. Even if data from initial analytics are available rapidly, that does not automatically mean that evidence is rapidly available for policy. Interpreting data appropriately requires a willingness to critically assess data, present information in a balanced way, and make decisions based upon that evidence as oppose to other interests (e.g. political, or financial). When data is used for policy-making, the party holding data determines its use and as data can be interpreted and potentially twisted in many ways, it can be used to provide ‘evidence’ for a variety of intentions.

For balanced So for balanced decision-making, the availability of data is not sufficient in isolation. The capacity and decision-making, willingness for its appropriate interpretation also needs to exist. Is there a way that we can help ensure that data alone is big data has a positive impact upon policy? insufficient • How can we be transparent about the strength of evidence the data provides? • Where does capacity already exist and how can we improve capacity to interpret insights appropriately in context? • Does appropriate interpretation of data require independence from political or funding influence? • What factors will change through the introduction of more technology? Will it make a difference?

 

14  

The ‘most appropriate’ big data approach will vary by context From the perspective of the ecosystem Data ecosystems are often fragmented, particularly in developing countries

Fragmented ecosystems are most likely to occur in developing countries where institutions and systems do not function well. More continuous data ecosystems are most likely to occur in highly developed countries. When thinking about the type of analyses to conduct in any given context, thinking about the approach in relation to the fragmentation of the ecosystem and the scale of continuous data may be helpful. For example, in a highly fragmented ecosystem, a ‘dual’ approach may be relevant to:

• Address data fragmentation on a large, system wide scale (in the long-term) e.g. strengthening Different research legal, regulatory, policy, and institutional frameworks; advocating for open data approaches will • Take a localized problem oriented approach to analysis whereby focal ‘pockets’ of data are be appropriate to analyzed to gain local-level insights (in the shorter-term) e.g. Examples C5 or C6 where data is different extents linked to either a service or an outcome. in ecosystems with different As a data ecosystem becomes more continuous, then the focal data ‘pockets’ should merge to become levels of larger fragments of information that have greater utility and facilitate analyses on a larger scale. It should fragmentation become increasingly possible to address more complex problems. Some analysis types that require more continuous data include: • • •

Exploratory analyses dependent on multiple disparate sources (e.g. Example C2) Predictive analyses that depend heavily on measuring the right variables (e.g. Example C4). Pulling together diverse data sources to present a big picture (e.g. Example C7)

From the perspective of digital data A digital divide means that those who are the poorest and most marginalized are not likely to be represented in digital data The relevance for any given data source is likely to depend on its ubiquity and homogeneity

A digital divide exists – a divide between those who have access to technology and those who do not. The people who are least likely to be captured in digital data or able to access technology to benefit from it are the poorest and most marginalized – the people that the development sector primarily aims to reach. Penetration of technology is increasing at a rapid rate in developing countries. In these contexts it is likely that the potential for data to reflect development project target groups in the future will be greater than it currently is. The precise utility, however, remains unknown. One potential way of looking at the utility of digital data is by looking at its ubiquity (being everywhere at the same time) and its homogeneity (uniformity in generation). A framework developed for this purpose is given in the ‘toolkit’ in Section D (the methodology behind the framework is given in Section C) and some practical examples of big data projects categorized by ubiquity are given below (pages 16-17). As is illustrated in the framework in section D, few digital data sources can currently be considered ubiquitous and homogenous; and the homogeneity (and therefore utility) of non-ubiquitous digital data is highly dependent on the strength of the system within which it functions. In which contexts is big data likely to have the most utility? By the time in the development process that data from technology becomes useful through its increased ubiquity and homogeneity, will the country in which the data is generated still be considered as developing? Or will the utility of big data be primarily recognized in middle- to high- income countries?

In which contexts is big data likely to have the most utility?

 

15  

For reference: examples of projects where data is considered ‘ubiquitous’ Mobile phone Call Detail Records have been considered ubiquitous (for “ubiquitous sensing”) in several developing country contexts. Some examples of these are listed here (B1-B4).

Example B1 Using mobile phone Call Detail Records (CDRs) to better understand population movement and quantify mobility dynamics.

Wesolowski AP, Eagle N. Inferring human dynamics in slums using mobile phone data http://www.santafe.edu/media/cms_pa ge_media/264/AmyWesolowskiREUFin alPaper.pdf

Example B2 Using mobility information from mobile phone CDRs layered with other sources of data on malaria risk to better target interventions.

Tatem AJ et al. (2014) Integrating rapid risk mapping and mobile phone call record data for strategic malaria elimination planning. Biomed Central 13:52. http://www.malariajournal.com/content /pdf/1475-2875-13-52.pdf

Example B3 Using descriptive historical data to predict future patterns: Predicting where people will move to in the event of a disaster based upon descriptive analysis of their past movement.

Lu et al. (2012). Predictability of population displacement after the 2010 Haiti earthquake. PNAS 109 (29) 1157611581. http://www.ncbi.nlm.nih.gov/pmc/artic les/PMC3406871/pdf/pnas.1203882109 .pdf

Example B4 Using new data sources as a proxy for an existing indicator type. Mining mobile phone CDRs to derive proxies for poverty indicators. Poverty maps as established by traditional methods (the Multidimensional Poverty Index image on the left), and mobile phone CDRs (image on right):

 

Smith C. et al. (2013). Ubiquitous sensing for mapping poverty in developing countries. (http://www.cities.io/wpcontent/uploads/2012/12/d4d-chrissubmitted.pdf

16  

For reference: examples of projects using data that is not considered ubiquitous Other data sources are not ubiquitous. Example B5 describes a project where data is linked to a service, and Example B6 where data was linked to an outcome

Where data is not ubiquitous and generated by those using specific services e.g. attending hospital or linked to a specific outcome (e.g. fire), its most likely use is in service optimization. Example B5. Predicting hospital admissions “Evidence-based research demonstrates that overcrowding in emergency departments causes ambulance diversion, increased hospital lengths of stay, medical errors, increased patient mortality, financial losses to hospital and physician, and medical negligence claims. Many hospitals still do not anticipate and prepare for the next day’s volume and admission through the emergency department. And yet, contrary to the conventional wisdom that emergency patient volume is highly unpredictable, the number of admissions per day can be predicted with remarkable accuracy. Forecasting presentations and admissions is a relatively easy solution. When implemented, it can protect everyone’s access to emergency care”. Boyle, J. (2010). PAPT – Patient Admissions Prediction Tool. ICU Management Spring 2010, EMC Consulting Group. p. 12-13, 16, 30. Example B6. Getting down to the root of the problem – fires in New York “Drawing on building information from many sources, the Risk Based Inspection System enables fire companies to prioritize the buildings that pose the greatest fire risk—and that means we’ll stop more fires before they can start”. NYC Mayor Bloomberg2. A video about the approach can be found on You Tube3

                                                                                                               

http://www1.nyc.gov/office-of-the-mayor/news/163-13/mayor-bloomberg-fire-commissioner-cassanonew-risk-based-fire-inspections-citywide#/1 3 https://www.youtube.com/watch?v=_M_20UjRvr0   2

 

17  

For reference: pulling together fragmented data sources to present a big picture The data ecosystem is fragmented, to different extents in different contexts. If big data is viewed from the data ecosystem perspective (as oppose to digital data sources), then big data methods may offer an opportunity to pull together information from diverse sources

Although this fragmentation could be considered to preclude some types of big data analysis e.g. predictive analysis, as its success is dependent on measuring the right variables (more difficult when data is fragmented); is it possible that other big data analytic methods could present an opportunity in these settings? Does big data have a role in pulling together and gaining insights from diverse, fragmented data so that the ‘big picture’ is more apparent? And beyond that in rapidly presenting data in a cohesive way for decision-makers Figure B3. Big data – a way to gain insights from diverse, fragmented sources of data?

Policy*brief* Budg et

*

ditu Expen

re*

Rou;ne*public*service* data*collec;on* Complaints*to*local*g overnment*

Civil*Society*Organiza;on*websites**

The ability to do this is dependent on whether the data ecosystem is sufficiently ‘healthy’. Example B7 describes an example of this from a highly developed country – the Netherlands

Online*job*boards* Video*

Sensors*

Surve ys*

Satellite*imagery* NonDgove rn Organiza mental* ;on*websi tes** Ci;zens*requ ests*for* h s* informa;on* rap g o ot Radio* Crowdso Ph urcing* GP Social*media* S*taggin g*

s* ource ews*s n l* a c Lo Mobile*phone*use*

Financial*transac;ons*

Example B7: The Dutch “virtual census” The Dutch “virtual census” did not conduct any additional data collection activities but brought together all existing available sources of data and linked them to get an up-to-date picture of the whole population. Information about the project is available in a report4 and can also be found on You Tube5.

                                                                                                                Nordholt E.S., Hartgers M, Gircour R. Statistics Netherlands, Voorber/Heerlen (2004). The Dutch Virtual Census of 2001, Analysis and Methodology. http://www.cbs.nl/NR/rdonlyres/D1716A60-0D134281-BED6-3607514888AD/0/b572001.pdf 5 Dutch Virtual Census video https://www.youtube.com/watch?v=SLpDkcyenf0 4

 

18  

For reference: visualizing and communicating data “We think about the world in all the ways that we experience it. We think visually, we think in sound, we think kinesthetically, we think in abstract terms, we think in movement.” Sir Ken Robinson, How schools kill creativity6 Big data offers new opportunities and is likely to have a large role in visualization (e.g. interactive charts, infographics, deep zooming). Some methods of displaying data are illustrated in the social media project described in Section E (page 69). Another example from the UK Police is given below: Example B8: UK Police The UK Police website supports an interactive crime map7. It is possible to select a geographical area, visualize numbers of crimes occurring in different locations and then click on the crimes to get more information about each one (Figure B5). Figure B5. A Numbers of crimes occurring in an area of London (image on left), and deep zooming to gain more information (image on right)

As is the case with this example; the majority of the time, visualizations are descriptive (as oppose to being relevant to other question types). Stephen Few8 proposes some future directions for visualization (listed below) that are all being pursued to some degree but “could be exploited more quickly if more researchers focused on solving real problems that we face in the world today”: • • • • • •

Building data visualization best practices right into the tools, such as in the form of defaults, thereby making it easier and less time-consuming to do what works and harder and more costly to do what doesn’t Integration of geo-spatial and network displays (such as node and link diagrams) with other forms of display for seamless interaction and simultaneous use Technological support for collaborative data sensemaking to bring the complementary advantage of multiple brains together The application of data visualization beyond descriptive statistics to the realm of predictive analytics, such as through the use of interactive predictive visual models Tighter integration of data mining algorithms to find meaningful patterns with data visualization to find a better way to review and explore these patterns Improved human-computer interface devices for interacting with data visualization in a more rapid and seamless manner

                                                                                                                http://www.ted.com/talks/ken_robinson_says_schools_kill_creativity#t-801967 http://www.police.uk/metropolitan/00BK17N/crime/ 8 https://www.interaction-design.org/encyclopedia/data_visualization_for_human_perception.html 6 7

   

19  

C. Developing a framework for the application of big data for monitoring and evaluation This section describes the methodology that was used to develop a framework for big data monitoring and evaluation. The purpose of the framework is to provide approximations of the areas where big data might be most successfully and unsuccessfully applied. To do this, data was considered in the context of its ‘type’, the proposed question, and the needs at each stage of a theoretical project cycle. Tables that break down information about data application by data ‘type’ can be found in Section D of this report – the ‘toolkit’ pages 44-46, Tables D3-D5. Step1: Data was categorized into five types Data was categorized into five types (Table D1, page 32)

As was discussed in Section B (page 15), the utility of digital data is dependent on its ubiquity and its homogeneity. Data was categorized into five types dependent on these characteristics, and through looking at whether the data was linked to a specific system or outcome. Table D1 on page 32 contains the classification. Step2: Question types were defined

Question types were defined (pages 35-43)

A pre-existing question type definition (Table D2 p.35) was used. A detailed description of each question type is given on pages 35-43. Step 3: The project cycle was characterized (Figure B1). Three stages of the project cycle were broadly defined as 1) Design; 2) Implementation and Monitoring; and 3) Evaluation and Learning. The likely needs for each stage were determined as illustrated in Figure C1. Figure C1. A theoretical project cycle with broadly defined needs proposed for each stage.

The needs of three stages of the project cycle were broadly defined

 

20  

Step 4: A matrix for cross-examination by data type, question type, and needs in the project cycle was developed A matrix was developed that cross-tabulated the categorizations developed in steps 1-3. The purpose of this was to consider which research questions and types of data would be relevant for addressing the different needs of the project cycle. The matrix, color-coded for relevance is given in Table B1, and a written description of the table’s coding provided below. Table B1. Matrix for cross-examination: data type, question, and needs at each stage of the project cycle Ubiquitous Type 1: Information generated everywhere at the same time

A matrix was developed that cross-tabulated data type, question type, and needs of the project cycle

Design New sources of contextual information & strategic planning Implementati on and Monitoring Real-time actionable information Learning and Evaluation Additional and broader insights

Key:

Type 1A Descriptive Exploratory Inferential Predictive Causal Mechanistic Descriptive Exploratory Inferential Predictive Causal Mechanistic Descriptive Exploratory Inferential Predictive Causal Mechanistic

Type 1B Descriptive Exploratory Inferential Predictive Causal Mechanistic Descriptive Exploratory Inferential Predictive Causal Mechanistic Descriptive Exploratory Inferential Predictive Causal Mechanistic

Non-ubiquitous Type 2: Information that is generated within a specific system or as a result of a specific outcome Type 2A Type 2B Descriptive Descriptive Exploratory Exploratory Inferential Inferential Predictive Predictive Causal Causal Mechanistic Mechanistic Descriptive Descriptive Exploratory Exploratory Inferential Inferential Predictive Predictive Causal Causal Mechanistic Mechanistic Descriptive Descriptive Exploratory Exploratory Inferential Inferential Predictive Predictive Causal Causal Mechanistic Mechanistic

Relevant for new sources of contextual information

Relevant for real-time actionable information

Relevant for strategic planning

Irrelevant for real-time actionable information

Type 3: Information generated ‘outside of the system’: Descriptive Exploratory Inferential Predictive Causal Mechanistic Descriptive Exploratory Inferential Predictive Causal Mechanistic Descriptive Exploratory Inferential Predictive Causal Mechanistic Relevant for gaining additional and broader insights

Description: Design: It was deemed likely that questions related to new sources of contextual information would be descriptive and exploratory; and that strategic planning could potentially involve all of the research question types (descriptive, exploratory, inferential, predictive, causal, and mechanistic). Implementation and monitoring: It was determined that descriptive analyses were the most relevant question type for delivering real-time actionable information. All other types of analyses (exploratory, inferential, predictive, causal, mechanistic) were deemed irrelevant because these analyses demand relatively more time and capacity to deliver insights. Learning and evaluation: It was concluded that all research question types would be relevant for this phase.

 

21  

Step 5: The matrix was refined Through examining Table B1 it became apparent that there was substantial overlap between the type of The matrix was then simplified… insights that were relevant at the design and the learning and evaluation phases. Essentially relevance had been determined by categorization into two types of information: 1. Descriptive insights that were available quickly - ‘fast insights’, relevant for rapid response in the implementation and monitoring stage 2. Deeper insights that would take more analysis time - ‘slow insights’, relevant for preparation in the design phase, and the learning and evaluation stage A new, simplified classification was therefore developed (Table B2). Table B2. Simplified matrix for cross-examination: data type, question, and the need for fast and slow insights

Fast insights Slow insights

Ubiquitous Type 1: Information generated everywhere at the same time

Non-ubiquitous Type 2: Information that is generated within a specific system or as a result of a specific outcome

Type 1A: Homogenous (complete data) - reflects reality on the ground in its entirety

Type 1B: Heterogenous (incomplete data) - reflects only partial reality on the ground

Type 2A: Homogenous (complete data) - reflects data about the users of a specific system or outcome of interest in its entirety

Type 2B: Heterogenous (incomplete data) - only partially reflects data about the users of a specific system or outcome of interest

Descriptive Descriptive Exploratory Inferential Predictive Causal Mechanistic

Descriptive Descriptive Exploratory Inferential Predictive Causal Mechanistic

Descriptive Descriptive Exploratory Inferential Predictive Causal Mechanistic

Descriptive Descriptive Exploratory Inferential Predictive Causal Mechanistic

Type 3: Information generated ‘outside of the system’: data is not generated as a direct result of a specific system or the occurrence of a specific outcome, although elements of data may be related to it Descriptive Descriptive Exploratory Inferential Predictive Causal Mechanistic

…and considered to determine the areas where big data is likely to be Step 5: The matrix was applied to approximate how big data is likely to be applied successfully applied successfully and and unsuccessfully. unsuccessfully These considerations are provided in Tables D3-D5 pages 44-46. (Tables D3-D5, pages 44-46) Note: Data application is complex and this framework is not considered to be all encompassing. It works on the basis of generalizations and assumptions. These are not likely to be perfect, but the hope is that they provide insights on research direction, and anchors for constructive discussion.

 

22  

D. A toolkit for big data experimental design Design is a multi-stage iterative, interactive process Big data experimental design requires communication and collaboration

Box D1. Design and public services – a sound bite from the design council A number of governmental representatives, designers and academics weigh in about how design can help us to deliver better public services: https://soundcloud.com/designcouncil/what-role-can-design-play-in

 

The aim of the design phase is to help different partners in communicating and developing a solution together. Locally relevant solutions = local needs + locally informed design

Locally relevant solutions = local needs + locally informed design • •

What are the right tools? Literacy, capacity, connection, electricity etc. What are the communities’ needs and what do they want to track?

Design is an iterative, and interactive process. Many data problems and ideas for research will arise, but not all of these will be appropriate for development: investment should be selective and many ideas will need to be excluded along the way. Therefore rapid, low resource approaches should always be considered as a first step before more comprehensive assessments or concept development is conducted. Figure D1 illustrates the different steps of the design process over time (before, and after the data problem arises), including a very approximate indication of the relative level of resources required at each stage. Figure D1. Steps of the design process and roughly approximated resource requirements

Preparedness:"data" mapping"and" collec/on"in"the" public"and"private" sectors"(census," surveys,"geospa/al" data,"telecoms"data" etc.)"

The&design&phase&

Resource&requirement&

Design is iterative and conducted in several stages. Rapid low resource approaches should be considered as a first step before more comprehensive assessments are conducted or concepts developed

Full&Assessment&of& problem" Concept& development"

Rapid&Assessment& of&problem"

Before&specific& Problem&arises&

?&

Rapid&Prototyping" Time&

Data&problem& arises&

The information in the following Toolkit sections is structured in an approximate order for consideration in a project cycle, although it not a definitive guide. It is likely that every big data project will have different needs and follow a slightly different path.

 

23  

I want to…

 

Prepare for my project by better understanding the context

Mapping data sources Stakeholder mapping Identify entry points

23 24 25

Rapidly assess the problem

Interviews Observation Focus Group Discussions Producing a report

26 26 27 27

Fully assess the problem

Data source mapping (comprehensive, targeted) The type of data you have and what it represents The type of questions that you want to ask The limits of how the data type can be used to answer the question Determine who the target group for the intervention is Understand users better Understand the problem better Clarify priorities moving forwards

28 28-31 31-43 44

Make some rapid prototypes

Engage with people to come up with new ideas Develop and communicate your vision of the end product

47-48 48-50

Turn the concept into reality

Clarify your plan Put together an interdisciplinary team

50-51 52

44 44-45 46 46

24  

Prepare for my project by better understanding the context Consideration of context is essential to produce individualized and localized solutions. The preparatory stage of big data research should be an ongoing effort: understanding and building relationships with different organizations, understanding the regulatory and legislative context, and developing agreements to share data requires a long-term investment. Box D2. Contextual Design “Contextual Design is rooted in the observation that any technology or system is always situated in a larger environmental context – and that introduction of new solutions invariably changes the environment for its users”. It is a “structured, well-defined user-centered design process that provides methods to collect data about users in the field, interpret and consolidate that data in a structured way, and use the data to create and prototype product and service concepts, and iteratively test and refine those concepts with users. This is the core of the Contextual Design philosophy – understand users in order to find out their fundamental intents, desires, and drivers. But these are invisible to the users – so the only way to glean them is to go out in the field and talk to people”. Implications for the designer: To create a successful product, first be aware of users’ work practice and design for it implicitly Hotzblatt & Beyer9

                                                                                                                9Interaction

Design Foundation Encyclopedia of Human-Computer Interaction http://www.interactiondesign.org/encyclopedia/contextual_design.html

   

25  

Mapping data sources When mapping the landscape, information should be collected systematically for each data source encountered and stored in a database. Figure D2 gives some suggestions about the types of questions that should be asked about each data type. Figure D2. Getting information about the attributes of different data sources

 

! Name!and!type!of!data!source!

! What!is!the!reason!for!data!genera5on!and! who!generates!the!data?!(e.g.%users%of%a% specific%digital%service)!

! Where!is!the!data!stored?!(device,%loca6on)!

! Is!there!any!legisla5on!surrounding!the!use! of!this!data!type!in!this!context?! (interna6onal,%na6onal,%sub9na6onal?)! !

! ! How!can!permission!be!gained!for!access?!Do! How!is!the!data!accessed?!(by%whom,%what% partnership!agreements!need!to!be!made?!! methods?)%

! Are!there!any!privacy!concerns!or!risks!with! using!this!type!of!data?!(par6cularly%related% to%the%iden6fica6on%of%individuals)! !

! How!usable!is!the!data?!(does%any%addi6onal% work%need%to%be%done%to%get%the%data%in%the% right%format?)%% !

! What!indicators!does!the!data!contain?! (variable%names,%informa6on%on%format%or% validity%of%data)% !

26  

Stakeholder mapping This mapping exercise should include stakeholders in both the public and private sectors. The aim is to clarify the relationships between the stakeholders that you are working with. DIY Toolkit’s people and connections tool may be of use (Figure D3). Figure D3. Stakeholder mapping tool10

                                                                                                                10  http://diytoolkit.org/tools/people-connections-map/    

27  

Identify entry points Big data analytics needs big data. Big data can refer to either new sources of data, or new analytics, technologies or approaches. It adds value where new sources of data exist, or where information cannot be processed using traditional methods (most likely because data exists either in large volumes, or in diverse formats). New sources of digital data tend to be located fairly centrally (usually private sector, national/international offices), and existing data sources (routine systems or surveys) are likely to grow in scale and complexity with distance from the ground. Conversely, application of insights from big data requires interpretation of the data by people who know the context well. Those who know the context best work close to the ground, and in the country context localized contextual information is lost the further up the hierarchy you go towards the central government. In any given context determining where the best entry point is an essential first step. It is likely to be the points at which there is sufficient capacity (data, human resources) to implement big data analytics, and sufficient knowledge to interpret and respond to the data appropriately (Figure D4). This will involve a judgment call and some trial and error.

Capacity'

Figure D4. Determining potential entry points for big data analytics – a trade-off between localized central knowledge and big data availability

l oc aliz

ed 'kn o

wle d

Concept development ge'

ta'

'd a big

cs' ly7 a n

a

Entry&point?&

Grassroots'

 

Central'government'

28  

Rapidly Assess the Problem Potential data problems’ may not actually be about the data In this rapid assessment step it is essential to identify exactly where the problem lies

Three possible techniques for rapid assessment are interviews, observation, and group discussions

If presented with a broad problem statement (e.g. we need more data about child mortality), the first step is in finding out exactly where the issue lies: many ‘data’ problems may not actually be about the data. They may be about willingness or capacity for example, or about the political or legislative context. At this point it is important to screen the issues that can and cannot be addressed with big data. Figure D5 provides some basic examples of the type questions that need to be answered before proceeding further and investing more resources in a full assessment. Figure D5. Some basic examples (not an exhaustive list) of the type of questions that should be answered in a rapid assessment.

Various methods can be used to get to the root of problems, including interviews, observation, and group discussions.   Interviews ODI’s guide to the Rapid Outcome Mapping Approach has some useful tips for this: the ‘five whys’ technique to make a first approximation, delving into the detail using fishbone diagrams, and analyzing stakeholders using the influence and interest matrix11 Observation Learning about the problem at hand through observing those who need to overcome it can be very instructive. Figure D6 shows DIY Toolkit’s basic tool for observation: it is designed to help you think generally through the types of things you should think about when observing others.

                                                                                                                11  http://roma.odi.org/defining_the_problem.html    

29  

Figure D6. Basic tool for observation12

Group discussions Focus Group Discussions are often a good way to generate ideas. A variety of props can be used to stimulate ideas during focus group discussions, and many M&E sites contain information about methodology for conducting focus groups well. An example of an innovative approach that practitioners could potentially employ in this rapid assessment phase is affinity diagrams. These may help you identify “the major themes affecting a problem by generating a number of ideas, issues or opinions”13. It is a good idea to document all of the findings before continuing to the next phase of research

Producing a report Before deciding to move on to the next stage, writing a report that summarizes your knowledge to date is likely to be useful for idea consolidation, accountability, and to keep all project stakeholders on the ‘same page’. This report should include a summary of the key question(s) that need to be addressed and available resources to do it with.

                                                                                                                12  http://diytoolkit.org/tools/shadowing-2/ 13  http://www.improvementandinnovation.com/features/article/what-affinity-diagram/    

30  

Full Assessment of the Problem This stage aims to define the issue and the context in a more comprehensive and targeted way

The full assessment stage considers all of the different sources of data that are available, and how new data sources, analytic methods, or approaches may or may not answer the question you want to answer.

Data held in different organizations and sectors should be mapped

Data source mapping (comprehensive, targeted)

Conducting a ‘data dive’ may help in doing this

This builds on the work conducted in the rapid assessment stage (the role of which was to establish that the problem was a ‘data problem’ that could potentially be addressed with big data). Two primary steps need to be conducted: 1) A comprehensive, targeted, data source mapping exercise, and 2) establishing exactly which questions big data can answer.

As a continuation of the work done in the preparation phase, establish what data sources are available in your specific project area in both the public and private sectors. The data-mapping tool (as illustrated in the preparation phase – page 23) may be useful when you meet to discuss data sources with representatives from different organizations. Another approach that could be used to explore the data is a ‘data dive’: Specialists working on the problem area delve into the questions identified in the mapping phase with people from different sectors and backgrounds (e.g. private sector, social scientists, community workers, NGOs, etc.). Together they aim to identify sources of data that could potentially provide new light into the issue. This phase also provides the means to explore the willingness of potential data providers to release data. The type of data you have and what it represents

Data characteristics needs to be considered

To apply data appropriately for M&E, the characteristics of the data need to be taken into account. The proposed framework for typifying all data types (including both big and small data) is illustrated in Table D1. A description of the method that was used to develop this framework is given in Section D.

 

31  

Table D1. Proposed framework for typifying data Ubiquitous (generated ‘everywhere’) Type 1: Information generated everywhere at the same time Some characteristics apply to data regardless of its information subset. This proposed framework takes account of these:

Type 1A: Homogenous (complete data) reflects reality on the ground in its entirety

Type 1B: Heterogenous (incomplete data) - reflects only partial reality on the ground

In reality, it is likely that no data source will fully fit ‘Type 1A’ criteria (completely homogenous and ubiquitous), but satellite data is likely to be the closest example of it (depending on the frequency that satellite images are taken).

Mobile phone Call Detail Records. Although currently classed as a type 1B data type), mobile phone Call Detail Records could potentially be considered as type 1A data if mobile phone penetration was high enough: a judgment would need to be made as to whether the characteristics of those who did not generate data biased the outcome of the answer to the research question (see section ‘validate and interpret’).

Non-ubiquitous (not generated ‘everywhere’) Type 2: Information that is generated within a specific system or as a result of a specific outcome Type 2A: Type 2B: Homogenous Heterogenous (complete data) (incomplete data) - reflects data - only partially about the users reflects data about of a specific the users of a system or specific system or outcome of outcome of interest in its interest entirety

Data generated within a specific system e.g. hospital user visits detailed in well-kept Electronic Medical Records; infrastructure sensors. Data collected as a result of a specific outcome e.g. data collected about every recorded incident of fire or flood, an information and complaint service

Same data sources as Type 2A data, but incomplete records. Could include data passively generated through the use of mobile app or web services; or systems where digital data is actively generated by users (e.g. mobile reporting systems for health workers) are particularly likely to be hold incomplete (heterogeneous) data considering staff turnover, incentives, variation in skill levels etc.

Type 3: Information generated ‘outside of the system’: data is not generated as a direct result of a specific system or the occurrence of a specific outcome, although elements of data may be related to it Social media (e.g. Twitter) Online news reports or other website content Internet transactions (e.g. search engine queries)

When conducting a big data analysis it is important to establish whether the data reflects the target population. The following tools (D7-D9) may help in thinking through what the data represents.

 

32  

Figure D7. Thinking through who is generating the data

Who$is$genera0ng$the$ data?$What$are$their$ characteris0cs?$

wh o

e$ er h w

$

What$informa0on$can$ the$data$provide?$

Where$is$the$data$ generated?$

$ en wh

wh a

t$

Data$source$

When$is$the$data$ generated?$

Figure D8. Thinking through who is affected by the problem the question is trying to address (the target population)

Who$is$affected$by$the$ problem?$What$are$ their$characteris7cs?$

wh o What$informa7on$do$ you$need$about$the$ target$popula7on$to$ answer$your$ques7on?$

e$ er wh

$

Where$does$the$ problem$occur?$

$ en wh

wh at $

Target' popula,on'

When$does$the$problem$ occur?$

 

33  

Figure D9. Are the people generating the data the same as the target population. Does it represent them?

COMPARISON$3$target$popula8on$vs.$data$source$popula8on$$ Overlapping+characteris0cs+

Differences+

Who$

Where$

When$$

What$

Consider the type of questions you want to ask It is essential to define exactly what types of questions you can realistically gain insights on from the data

To know what the right question is to ask of the data, you need to know what the exact problem is, and what the data reflects. “The Answer to the Great Question… Of life, the Universe and Everything… Is… Forty-two, said Deep Thought, with infinite majesty and calm… …“Forty-two!” yelled Loonquawl. “Is that all you have to show for seven and a half million years’ work?” “I checked it quite thoroughly” said the computer, “and that quite definitely is the answer. I think the problem… is that you’ve never actually known what the question is.” “But it was the great question!...” howled Loonquawl.

The first step (as described in earlier sections of the toolkit) is in knowing exactly what the problem is and what the data reflects

“Yes” said Deep Thought with the air of one who suffers fools gladly, “but what actually is it?” “Well, you know, it’s just Everything…” offered Phouchg weakly. “Exactly! Said Deep Thought. “So once your know what the question actually is, you’ll know what the answer means”. Source: Douglas Adams, Hitchhikers Guide to the Galaxy

 

34  

Table D2. Experimental research question type definitions (listed in approximate order of difficulty) Approximate level of difficulty

Types of experimental research questions and a brief definition of each question type are given here. More information about each question type is given on pages 35-43

Question Type Descriptive Exploratory Inferential Predictive Causal Mechanistic

Question Use Describe a set of data e.g. a trend To find relationships you didn’t know about Use a relatively small sample to say something about a bigger population Use data on some objects to predict values for another object To find out what happens to one variable when you make another variable change Understand the exact changes in variables that lead to changes in other variables for individual objects Source: Professor Jeffrey Leek, The Data Scientists Toolbox, Coursera

Additional information about each question type is detailed on the following pages:

Descriptive analysis tends to be the first, most basic step in analysis. It aims to describe trends or patterns in data

Descriptive analysis (steps described from pages 35-41) Descriptive analysis aims to describe trends or patterns. It may be used to describe: 1. An individual event of interest 2. The distribution of aggregate events of interest Descriptive analysis tends to be the first, and the most basic step in any analysis: Many projects are likely to take descriptive insights as a starting-point, and then use other analysis methods to look into the reasons behind the trends. The utility of different types of data for descriptive analyses is dependent on the representativeness and timeliness of the data insights. ‘Fast’ analyses that aim to rapidly identify and respond to events of interest are more likely to prioritize timeliness over representativeness (within reason), when compared with ‘slow’ analyses that aim to gain contextual information through describing a trend or pattern. Descriptive insights form one-step in a chain of events that is required to initiate a response (Figure C10).

Several steps occur between selecting data to capture and mounting a response to the information gleaned from the data – described in pages 36-41

Figure D10. Typical steps in the rapid response process and the phase at which they are conducted Design

 

Implementation and monitoring

Select event to capture

Capture data

Validate

Describe

Interpret

Respond

Page 36

p.37

p.38

p.39

p.40

p.41

35  

Descriptive analysis step 1: Select event to capture Three important rules to consider:

It is essential that you maintain the confidentiality of individuals. Following three important rules facilitates this

• • •

Never analyze personally identifiable information Never analyze confidential data Never seek to re-identify data

When capturing data, the type of event captured (expected or unexpected) will influence the type of analysis that can potentially be conducted (qualitative or quantitative) and whether it cab be used for a ‘formal’ indicator or as an informal source of information. Figure D11. Capturing data, most appropriate analysis method, and most probable use

Design'

in'combina.on'with'

Qualita.ve' approaches'

'e F

ec ted

sou rma rce l' '

Quan.ta.ve' measurements'of''change'

Rapid' monitoring' system' mobiliza.on'

De tec t'e xp

&' ng' rni on' Lea lua. Eva

Im ple mo men nit ta. or on i n g '& ' '

Define'data'signatures'and' indicators'for'expected'events''

orm ve RE RAPI source al' nts' S P D' ' ON Inf SE' o

When considering which type of event to capture, it is essential to consider the type of event; whether the data required is for use as a formal source (e.g. indicator) or informal source (e.g. anecdote) of information; and whether the type of analysis that is necessary is feasible in the time frame required for decision-making

Unexpected' event'

Box D3. Theory behind ‘data capture’ (as illustrated in Figure D11) Indicators can only be generated for ‘expected’ events, not ‘unexpected’ events because: • Event selection is dependent on the problem question: The choice of the events for detection is made at the design phase of the project cycle • Data signatures and indicators can only be defined when the event that they intend to capture is known Once an unexpected event has been recognized, digital event detection systems have the potential to be rapidly mobilized (rapid relative to mobilization through routine public service systems or active data collection through surveys). Data signatures and indicators can then be defined - essentially an unexpected event becomes and expected event. Quantitative measurements of change can be only used for expected events (formal data source), not unexpected events • When indicators can be generated, the data can be considered to represent a ‘formal’ source of information • Formal sources of information have the potential to be analyzed quantitatively (in addition to qualitative analysis) Qualitative analysis should be used for any unexpected events captured in digital signals (informal data source), for expected events where formal indicators cannot be defined, and in cases where interpretation of data for multiple events is required (rather than direct response to individual events). Qualitative analysis of this type is likely to be time consuming.

 

36  

Careful consideration of scale is also essential during this stage: You need to ensure a match between the data source, event, and intervention; as well as consider the variance of the data source relative to the magnitude of the event that you want to capture

Other important considerations at this stage could include the following: Are there sufficient user characteristics in the data to facilitate adequate disaggregation (both to the scale of the event and the scale of the intervention)? With digital data it is often unclear as to whom the data represents. Lack of disaggregation compromises our ability to understand inequality and reflect development priorities at the sub- national level (e.g. geographic location, sex, age...). Will the intended tool and data source enable you to detect a sufficient magnitude of change to capture the event (and its potentially its change over time for base/end-line evaluation) in this context? As development is dynamic and heterogeneous, an algorithmic tool’s sensitivity and specificity (see Step 2 of descriptive analysis) for event detection is likely to vary in different contexts and at different points in time. The ability of a tool to capture an event will be heavily dependent on whether the scale of the event matches the scale of the generation/disaggregation of the data; and on the variance of the data captured. Will you be able to separate the event you want to capture (the signal) from the noise? Changes in population size and structure will occur alongside changes in equality and access to and use of technology. Changes in age-structure, incomes, and geographic distribution will need to be considered so that diverse starting points in different contexts can be accounted for, and so that change can be measured in absolute or relative terms. As characteristics of technology users are frequently absent, this may not be feasible with many digital data sources.

Descriptive analysis step 2: Capture data The sensitivity and specificity of a data capture tool will determine the noisiness of the data

The signal to noise ratio is affected by the sensitivity and specificity of the data capture tool. If the sensitivity is increased, then specificity decreases (see Figure D12): more false positives are captured. False positives can be viewed as being akin to ‘noise’. The less specific the tool is at capturing intended events (true positive), the noisier the data (implications for data accuracy and therefore for validation) Figure D12. Tool sensitivity and specificity

 

37  

Descriptive analysis step 3: Validate When validating data, there is a trade off between timeliness, cost, and accuracy…

Data needs to be accurate so that resources are not used chasing false leads in the response phase. Resources for data validation are often limited and in validation there is a trade-off between accuracy, timeliness, and cost. Figure D13. Trade-offs for real-time data collection

…For example, the noisier the data is (less accurate) then the higher the cost and time required to validate it

 

38  

Descriptive analysis step 4: Describe Descriptive analysis most likely has an application in detection/description of individual events, or rapidly identifying very obvious shocks or shifts in aggregate data e.g. sudden spikes in a trend. It is not likely to be as appropriate in application for the detection of smaller absolute or incremental changes. Figure D14. The potential of descriptive analysis for different samples of digital data

A B C In the development sector Type A analysis is likely to be the most appropriate use-case for describing either individual or aggregate events of interest. For descriptive analysis to hold utility, the data needs to be representative and clean

The utility of the data will be dependent upon the: • • • • • •

Event being expected or not (Step 1) Event of interest occurring on a spatiotemporal scale that can be detected by the resolution of the data (Step 1) Variance of the data (Step 1) Noisiness of the data – dependent on the sensitivity/specificity of the tool to capture the data (Step 2) Capacity to validate each event (Step 3) Representativeness of the data relative to the analysis question (Step 5)

It will always be easier to detect obvious spikes or shifts as oppose to gradual or small changes. Although obvious changes may be detected using fairly automated quantitative methods, interpretation of smaller changes is likely to involve more extensive qualitative work.

 

39  

Descriptive analysis step 5: Interpret Descriptive statistics describes trends and patterns in digital data. The meaning of these trends will be determined by what the data represents. As descriptive analysis describes trends and patterns, the meaning and value of the data is dependent upon what the data reflects. This is determined by type of data. 1A and 2A are the preferable data types for this type of analysis as they are homogenous

Data Type 1A 1B 2A 2B

3

What the data reflects Ubiquitous and homogenous so descriptive statistics reflect the total population. Ubiquitous but heterogenous so descriptive statistics only reflect part of the population. Further analyses (inferential statistics) would be required if the problem at hand needs the data to reflect the total population. Reflects the users of a specific system or outcome of interest in its entirety. Reflects the users of a specific system or outcome of interest but is heterogenous. Therefore descriptive statistics only reflect part of the outcome of interest. Further analyses (inferential statistics) would be required if the problem at hand needs to be generalized to reflect the entire system or outcome. Not generated as a direct result of a specific system or the occurrence of a specific outcome, therefore its application will be dependent upon knowledge of its characteristics. If characteristics are known (e.g. the age, sex and location of those generating the data) then methods of statistical inference (e.g. Bayesian statistics) could be used to gain insights that could be generalized to the population. A range of example characteristics that could influence data generation are given in Figure D15. More of these characteristics are likely to apply for actively generated content vs. passively generated signals (Table A1) which makes interpretation of data types involving actively generated content (e.g. social media posts) problematic. With this data type, the characteristics of the data are often unavailable (e.g. reliant on user generation or anonymized for privacy reasons).

Figure D15. The meaning of descriptive statistics for data types 1 and 2

 

40  

Descriptive analysis step 6: Respond To trigger a response, the data has to have met the needs of the problem

For data to trigger a response, a thorough needs assessment prior to project implementation is essential. If a thorough needs assessment is not conducted or insufficient resources exist to process the information adequately then: • Opportunity-costs could occur e.g. the introduction of technology diverts resources from the capacity for response, or new data methods have less utility than existing ones in terms of accuracy, time and cost • Poor decisions could be made e.g. poor investment of resources • Resources could be invested in developing tools that are not used The new data’s value will be determined by other comparisons to existing sources, for example the timeliness, quality and quantity of information obtained, and the cost of obtaining it. As existing information cycles can potentially be very long (e.g. the census is 10 years), much of the value of new data sources is likely to be around improved timeliness of insights. It is implicit that for a response to occur, the processes and capacity for the response must exist. Exploratory analyses

Exploratory analyses seek to find relationships you didn’t know about

Exploratory analyses seek to find relationships that you didn’t know about. Essentially, exploratory analyses only give you a lead that you can then explore further using additional analyses. New sources of digital data and new computational techniques, typically using algorithms, are used to reveal trends, patterns, and correlations in data, and visualize it to turn it into actionable information. Methods that could be used to gain exploratory insights include machine learning, data mining, and sensemaking. Although correlation can be established from this type of analysis, causality cannot. Care should be taken not to make inappropriate inferences about the cause of trends, patterns, and correlations (in addition to the considerations described in the descriptive analysis section). If inference needs to be made to a population, inferential analyses should be conducted. A risk of this type of analysis is finding false associations by exhaustively testing hypotheses until something fits. A method to minimize risk is testing only for those possible relationships which seem compelling a priori to domain experts, or explicitly using multiple hypothesis testing. Inferential analyses Inferential analyses use a relatively small sample to say something about a bigger population.

Inferential analyses use a relatively small sample to say something about a bigger population

Inequalities in the generation of and access to digital data mean that digital signals often represent information from a sub set of the ‘population’ (or signals if not user generated) of interest. Different people in the population have different characteristics and populations don’t mix evenly – people ‘cluster’ with others with similar characteristics to themselves e.g. socioeconomically, demographically, or geographically. Different social groups use technology to different extents and in different ways. Unless a very high proportion of the entire population generates a source of digital data, one digital data source will most likely only provide information about people with a particular set of characteristics. The ability to infer from digital data to the general population is dependent on the digital divide for any given data source; and on the way that those who can access the technology choose to use it in any given context. This can be dependent on many factors (broadly demographic, cultural, economic, political and geographic), including those indicated in Figure D16.

 

41  

Figure D16. Characteristics that could potentially influence data generation

Where data is technology user generated, many potential characteristics of users could influence data generation

Age'

Physical'health:'vision,'hearing,' thinking,'dexterity,'mobility'

Gender'

DEMOGRAPHIC'

Mental'health' Knowledge' of'languages'

Behavior'

Ethnicity'

GEOGRAPHIC'

SOCIAL' FACTORS'

Service'delivery' and'access'

Educa,on' Occupa,on'

Transparency' Trust' Freedoms'

Religion'

CULTURAL'

POLITICAL'

ECONOMIC'

Stability'

Income' Wealth'

Predictive analyses Predictive analyses use the data on some objects to predict values for other objects. An example of this could be the development of proxy indicators. The ability to conduct this type of analysis is wholly dependent on the data ecosystem and regulatory environment. It is challenging where data is fragmented and inaccessible as the method depends heavily on measuring the right variables.

Predictive analyses use data on some objects to predict values for other objects

In a situation where lots of data is available, techniques like data mining could potentially be used to detect unanticipated correlations. To get this type of data, data needs to be publicly available, or data access needs to be granted with a relatively undefined research protocol. In situations where data is relatively closed – or where a specific research question is needed to access data (e.g. I need this exact variable to predict this one), then application is limited to situations based on human generated theory - a precise research question is formed a priori. This is the realm of traditional research rather than big data and it limits the potential of big data methods. Unless the data ecosystem is ‘healthy’, predictive analyses are likely to remain in the realm of simple, known, direct causal pathways with priors; as oppose to unanticipated computer generated hypotheses that might lead to fresh insights on complex problems. Causal analyses Many problems in the development sector are complex – they involve dynamic interacting factors. This makes causal analyses difficult: A change in an outcome (‘outcome’ in Figure D17) is not necessarily caused by one factor (independent variable in Figure D17). The reasons for a change in a variable could be multifactorial, or, as illustrated in Figure D17, other factors (the confounder or intermediate variable) could confound (a) or modify (b) the outcome. Determining the cause behind any given change is often difficult so randomized studies are often used – non-randomized studies are sensitive to assumptions. As big data analysis uses retrospective data (a function of it usually being secondary, and not designed for purpose), and randomization is a function of prospective design; it is unlikely that randomized studies and big data will be a good match for big data analysis unless the outcome that the data source measures happens to be the outcome of interest.

 

42  

Figure D17. Confounding and effect modification

Mechanistic analyses Mechanistic analyses don’t seek to identify the cause; but to understand the exact changes in variables that lead to changes in other variables for individual objects. It is usually modeled by a deterministic set of equations in which the data’s only random component is measurement error (a deterministic system is one in which no randomness is involved in the future state of the system). This means that the cause already needs to be known; and the causal pathway needs to be direct and without confounding or effect modification (Figure D17).

Consider the limits of how the data type can be used to answer the question Type 1 data Appears to have the most potential in ‘slow insights’ such as gaining contextual information, than in gaining ‘fast insights’ for rapid response such as detecting events (Table D3).

 

43  

Table D3. Type 1 data summary Question Descriptive (fast)

Summary for Type 1 data Timeliness: Challenges related to capture and validation have the potential to compromise timeliness of event detection. Data is not captured as a result of the event of interest so the scope of the data signal is likely to be broad, and the relevance of the data that is captured will be heavily dependent on the sensitivity and specificity of the data capture tool. Most likely to meet with success where the event is expected, well defined, and the data capture tool is both sensitive and specific - manual validation has the potential to be resource intensive and time consuming which compromises the data’s utility for rapid response. Scale: Observation of any unexpected perturbations in the data will be dependent on a match between the geographical scale that the data is generated on, and the scale of the event. As an example, if mobile phone signals were captured from a wide geographic area (e.g. a country) it is likely that only large-scale catastrophic effects would be observable in the signal (e.g. a large volcanic eruption that affected populated areas on a large geographic scale, not a more focal event like a landslide). In which case, events are likely to be recognized on the ground before in the data. As responses are usually relatively localized, data must be available at (aggregated to) a resolution where information is available at a scale that matches the needs of the response. Representativeness: 1A data (homogenous) is likely to be better suited than 1B data (heterogenous). 1A could potentially be used for either individual or aggregate event detection. 1B is only suitable where getting a complete picture is not essential. 1B is likely to work best as an informal, complementary data source for individual event detection where parallel systems for data collection exist for triangulation (extensive internal and external validation necessary). It is not likely to work as a formal indicator with aggregate trends unless detailed information about the reasons behind the heterogeneity are known. It is likely that underlying heterogeneity will shift over time and this needs to be taken into account with timeseries trends.

Descriptive (slow)

Exploratory

Inferential

Same limitations as above relating to representativeness and scale. Timeliness not as much of a limiting factor. Most likely use is in strategic planning (design phase) e.g. better understanding of mobility in a community. Insights from this type of analysis could be used to facilitate a rapid response (e.g. by knowing from historical events where the population is likely to move to in the event of a disaster), but not in detecting the events themselves (challenges for event detection discussed above). 1A data has more utility than 1B because of its homogeneity. Likely to play a large role with homogenous type 1A data. If conducted at the design phase of a project, it could be used to gain contextual information for example layering it with other data sources (e.g. multivariate analyses), building mobility patterns into models for disease transmission etc. The application of type 1B data for exploratory analyses will be dependent on its heterogeneity in relation to the question. Although these analyses may detect correlations, correlation is not the same as causation. Type 1A data is considered ubiquitous and homogenous so it reflects, and inference can be made to the population (designating it as ‘ubiquitous & homogenous requires a careful judgment call on representativeness). Type 1B data can only be used for inferential analyses if sufficient characteristics about the data are available in the sample to enable statistical inference to the larger population. To evaluate change over time e.g. evaluating the effect of an intervention on a population (e.g. how population mobility is affected through building a bridge), the generation of the data needs to remain consistent.

Predictive

Causal Mechanistic

 

Type 1 data could be appropriate for the development of proxies if the data ecosystem is healthy (feasibility is dependent on availability of data sources). An example could be in the development of proxies for poverty mapping. Generation of type 1 data is likely to be multifactorial and not likely to be suitable for either causal or mechanistic analyses.

44  

Type 2 data

 

Information is related specifically to an outcome of interest or specific system. Because of this, this data type is likely to have the least ‘noise’ in the data for the detection of individual events of interest (relative to other types) so rapid validation is likely to be relatively feasible. Appears to hold high potential in service optimization: either through rapid detection of events of interest or through slow insights; although slow insights seem to hold more scope in application. Table D4. Type 2 data summary

Question

Summary for Type 2 data

Descriptive (fast)

Should be possible to establish data type 2A’s variance based on historical trends, and conduct quantitative analyses: Automated quantitative descriptive analyses could include setting thresholds for the automated detection, and flagging of unusual activity. 2B data has limited utility for fast descriptive analyses: It is heterogenous so should only be used for fast analyses where getting a complete picture is not essential. Any further application would require more analyses.

Descriptive (slow)

Exploratory Inferential

Predictive Causal Mechanistic

 

Possible to describe trends in the outcome or use in the service with 2A data. Also possible to describe differences between two groups of exposure if subsamples are selected from within the digital sample using probability sampling (e.g. exposure to two different services, or positive vs. adverse outcomes). Statistical inference can then be used to describe the differences between two groups. Insights gained from aggregate trends in 2B data will not be usable without knowledge about exactly how data generation is heterogenous. Getting this information is likely to be particularly problematic for data sources that are actively generated. 2B data will require extensive internal and external validation from other sources. Likely to work best as an informal, complementary data source as oppose to a formal indicator where parallel systems for data collection exist for triangulation (either triangulating with external data to see gaps in data on the digital system, or for externally evaluating a data different collection system). Possible and dependent on the metadata collected along with the outcome data. The more detailed the metadata, the higher the potential for analysis. Can only be inferred to the sample (e.g. a specific service or outcome): 2A data reflects the service or the outcome in its entirety. For 2B, data characteristics regarding its heterogeneity (e.g. details about events that are included and excluded in the sample) need to be known to generalize findings to the full service/outcome. Limited as data refers to a specific outcome/service. Dependent on the metadata collected and the health of the data ecosystem. Unlikely unless the data was randomized through study design Possible if the causal pathway is direct e.g. physical sensors

45  

Type 3 data • • • •

•

•

Only represents the digital domain. Application limited due to poor data quality. Should only be used in situations where getting a complete picture is not essential Unlikely to be suitable as a formal monitoring and evaluation indicator. Has most potential as an informal, complementary data source where parallel systems for data collection exist for triangulation and there is a high level of capacity to critically analyze the data: extensive internal and external validation is required to know where gaps in information generation exist. Holds a high risk for decision-making due to misinterpretation. Misinterpretation of data is likely to occur if either the data analyst or the recipients of the data insights hold too much value in its meaning. To reduce risk, insights from this type of analysis should only be presented along with a comprehensive list of the limitations of the data. Most likely use-case is in gaining information about individual events where data does not need to be complete (see fast descriptive). Not likely to be useful for trend analysis, and only limited use for ‘slow insights’.

Table D5. Type 3 data summary

Question

Summary for Type 3 data

Descriptive (fast)

Most likely application of type 3 data. For example, rapid response in very situations where the data capture tool can identify individual events with a high enough sensitivity/specificity to make validation feasible; and where incomplete information is useful e.g. where gaining information about some emergency events (e.g. floods) is sufficient to improve a response to them, even if not all events are known.

Descriptive (slow)

As the data is likely to be noisy and incomplete, validation and interpretation will most likely require extensive qualitative research and contextual knowledge. Most likely to be useful in combination with other sources as a complementary data source.

Exploratory

For exploratory analyses to be meaningful, a substantial investment will need to be made in validating and interpreting the data. As this type of analysis does not involve inference to the population, insights will not be definitive. Use of this data for research will require triangulation with other data sources in addition to a strong basis in theory and knowledge of the context. Extensive use of qualitative research methods will be essential.

Inferential

Causal

Unsuitable. Data characteristics are likely to be unknown. Very difficult to determine what the data represents. Insights could be misleading when inferences are made outside of the digital dimension. Possible, but if information about the generation of data is unknown and patterns in its generation shift over time, the model is not likely to be sustained. No

Mechanistic

No

Predictive

 

46  

Determine who the target group for the intervention is Figure D18. Tool for better defining the target group14

Understand users better

Remember that “work practice is complex and varied, and that useful design data are hidden in everyday details. Many systems fall short of expectations because they fail to take into considerations seemingly insignificant details of work practice – details that are not consciously available to users when they are not engaged in the ongoing work. Contextual Design holds that design team members must go into the field and observe and talk with users in their natural work or life environments – their natural contexts – in order to understand work practice. This is the principle of context from which the process draws its name. This aspect of Contextual Design leverages the work of earlier ethnographic methodologies (Garfinkel 1967) but extends it in important ways. Implications for the designer: Use field interviews to reveal tacit aspects of users' work practice - the motivations, workarounds, and strategies that they may never articulate, but structure their work.” Source: Interaction Design Encyclopedia15

  Try to understand the situation from their perspective. Consider their intent and desires, and their key characteristics. What drives them? The character-profiling tool in Figure D19 may be useful for this.

                                                                                                                14  http://diytoolkit.org/tools/target-group/   15  Source: http://www.interaction-design.org/encyclopedia/contextual_design.html    

47  

Figure D19. Character profiling tool16

Consider how individuals work together to get work done: The flow model captures communication and coordination between people to accomplish work. It reveals the formal and informal workgroups and communication patterns critical to doing the work. It shows how work is divided into formal and informal roles and responsibilities. The cultural model captures culture and policy that constrain how work is done. It shows how people are constrained and how they work around those constraints to make sure the work is done. The sequence model shows the detailed steps performed to accomplish each task important to the work. It shows the different strategies people use, the intents or goals that their task steps are trying to accomplish, and the problems getting in their way. The physical model shows the physical environment as it supports or gets in the way of the work it shows how people organize their environments to make their work easier. The artifact model shows the artifacts that are created and used in doing the work. Artifacts reveal how people think about their work – the concepts they use and how they organize them to get the work done. Source: Interaction Design Encyclopedia17

                                                                                                                16  http://diytoolkit.org/tools/personas-2/   17  http://www.interaction-design.org/encyclopedia/contextual_design.html    

48  

Understand the problem better Figure D20. Clarifying the problem Figure D11. Tool to clarify the problem - break down a complex issue18

 

Clarify your priorities moving forward Figure D21. Define the problem more succinctly19

                                                                                                                18  http://diytoolkit.org/tools/causes-diagram/ 19  http://diytoolkit.org/tools/problem-definition-2/    

49  

Make some rapid prototypes Engage with people to come up with new ideas Figure D22. Engage with a group of people and work with them to generate new ideas20

Figure D23. Frame the discussion21

                                                                                                                20  http://diytoolkit.org/tools/creative-workshop-2/ 21  http://diytoolkit.org/tools/thinking-hats-2/    

50  

Figure D24. Think in a different way22

Consider holding a data challenge: During data challenges, solution solvers are given access to the datasets identified in the data dives, and are asked to either provide new analytics or working prototypes of potential solutions that can help fill the existing data gaps. As well as virtual competitions, face-to-face events gathering subject matter specialists and data scientists can also be organized in order to encourage the formulation of new insights. Develop and communicate your vision of the end product “In visioning, the team uses the consolidated data to drive conversations about how to improve users’ work by using technology to transform the work practice. This focuses the conversation on how to improve people’s lives with technology, rather than on what could be done with technology without considering the impact on peoples’ real lives”. Source: Interaction Design Encyclopedia23 Figure D15. Develop a plan for prototype development and testing24

                                                                                                                22  http://diytoolkit.org/tools/fast-idea-generator-2   23  http://www.interaction-design.org/encyclopedia/contextual_design.html   24  http://diytoolkit.org/tools/prototype-testing-plan/    

51  

“When you put all these things together, with elements from architecture, physical design, electronic technology from software, how do you actually prototype an idea for a service, and it seems that really, its about storytelling, its about narrative”. Source: Bill Moggridge25 Box D4. Methods to consider Storyboards Each storyboard describes how users will accomplish a task in the new system. They show the steps the user will take and the system function that supports each step. The task may be handed off between users, and may be supported by several systems operating together; the storyboard ensures the task remains coherent across these boundaries. User Environment Design The storyboards ensure coherence of individual tasks, but the new system must have the appropriate structure to support a natural flow of work through the system no matter what task the user is doing. Just as architects draw floor plans to see the structure and flow of a house, designers need to see the “floor plan” of their new system – the basic structure that will be revealed by the user interface drawing, implemented by an object model, and that responds to the customer work. This “floor plan” is typically not made explicit in the design process. The User Environment Design captures the floor plan of the new system. It shows each part of the system, how it supports the user’s work, exactly what function is available in that part, and how the user gets to and from other parts of the system – without tying this structure to any particular user interface. Paper prototyping Paper prototyping develops rough mockups of the system using notes and hand drawn paper to represent windows, dialog boxes, buttons, and menus. The design team tests these prototypes with users in their workplace, replaying real work events in the proposed system. When the user discovers problems, they and the designers redesign the prototype together to fit their needs. Rough paper prototypes of the system design test the structure of a User Environment Design and initial user interface ideas before anything is committed to code. Paper prototypes support iteration of the new system, keeping it true to the user needs. Refining the design with users gives designers a customer-centered way to resolve disagreements and work out the next layer of requirements. After several rounds of prototyping, the larger structure of the system design stabilizes. At this point, the design team can continue iterating areas of the user interface. Once the structure and interaction design are largely stable, the team can develop and test interaction and visual design options with users. Source: Interaction Design Encyclopedia26

                                                                                                                25 26

 

Source: http://www.180360720.no/index.php/archive/prototyping-services-with-storytelling/ http://www.interaction-design.org/encyclopedia/contextual_design.html  

52  

Other resources that may be useful can be found in the following locations: Resource

Location

LEGO serious play

http://www.servicedesigntools.org/tools/46

Role Play

http://hci.stanford.edu/dschool/resources/proto typing/RolePlayingCHI031.pdf

General prototyping tips

http://thesquigglyline.com/2009/01/15/how-toprototype-the-awesome-guide/

Communicating your service design (moodboards, posters, storyboards, system organizational maps, stakeholder motivation matrix)

http://www.slideshare.net/urijoe/visualizationtool-how-communicate-the-service-designconcepts-presentation

Turn the concept into reality This stage helps you ensure that what you are developing is what people want, technically and organizationally feasible, and financially viable. Clarify your plan Figure D26. Evaluate where I am and what my options are27

                                                                                                                27  http://diytoolkit.org/tools/swot-analysis-2/    

53  

Figure D27. Work out my business model28

Figure D28. Formulate a clear plan29

                                                                                                                28  http://diytoolkit.org/tools/business-model-canvas/ 29  http://diytoolkit.org/tools/theory-of-change/    

54  

Establish an interdisciplinary research team “Intelligence is dynamic. If you look at the interactions of a human brain, intelligence is wonderfully interactive. The brain isn’t divided into compartments. In fact, creativity which I define as the process of having original ideas that have value, more often than not comes about through the interaction of different interdisciplinary ways of seeing things.” Sir Ken Robinson, TED talk on ‘how schools kill creativity’30

Specific skillsets and interdisciplinary collaboration are essential for the success of a big data research project Careful judgments need to be made by people with specific skill-sets Skills are likely to come from a multidisciplinary team, rather than one individual

Although broad approximations for successful practice were given in this toolkit, considerations about the appropriate application of data require careful judgments on a case-by-case basis by people with specific skill-sets. So what are the required skill-sets? Required skillsets are diverse and likely to be drawn from a team, rather than be covered by one person. A selection of different quantitative data disciplines and the data that practitioners in these disciplines are trained to handle are illustrated in Figure D30. To put it simply, traditional research moves into the realm of data science with the addition of ‘hacking’ (computer programming/problem solving) skills. It is key that along with hacking skills, that the person leading the big data project has training in the analysis and application of data (discipline-specific) and strong contextual knowledge so that they can make appropriate judgment calls. Without this, a project is likely to fall into the ‘danger zone’ (misapplication) – or remain only in the digital realm (annotated as ‘machine learning’ in Figure D29). Note that ‘data science’ as it is referred to in Figure D30 (a mixture of hacking skills, math and statistical knowledge, and substantive expertise) does not have the same meaning as ‘data science’ as illustrated in Figure D29 (web analytics). Web analytics is a specific discipline that is focused on the digital realm and it involves training in a very different skill-set to those who are trained specifically in data application in the development sector.

                                                                                                                30  http://www.ted.com/talks/ken_robinson_says_schools_kill_creativity#t-801967    

55  

Figure D29. Some examples of different quantitative data disciplines

Different disciplines are specialized in different types of quantitative data analysis

Sta's'cal) process) control)) Industrial( process(data(

Econometrics) (Economic( data(

Business) analy'cs)) Customer(data(

Biosta's'cs) Medical(data( ‘Data)science’) Web(analy5cs(

Different( types(of( data(

Signal) processing)) Electrical( signals(

Machine) learning) Computer( science/vision(

Natural) language) processing) Text(analy5cs((

Figure D30. Skill requirements to make a judgment call on a big data analysis

Substantive expertise in combination with math and statistical knowledge and programming skills is essential

Although only quantitative disciplines are illustrated in this section, qualitative Source: Drew Conway expertise is likely to play an important role in turning data In addition to quantitative disciplines, qualitative research methods will play a large role in turning data into usable information. This is discussed in more detail in section B. into actionable information

 

56  

E. Can social media analytics provide insights relevant for communicable disease control in Indonesia? A rapid assessment Contributors: Sally Jackson1 (SJ), Sutawijaya IB1 (SIB), Theresia Tanzil1 (TT), [Luuk Schoonman2 (LS), James McGrane2 (JM)], Nirmal Kandel3 (NK) 1Pulse

Lab Jakarta, United Nations Global Pulse and Agriculture Organization Emergency Center for Transboundary Animal Diseases, Indonesia 3World Health Organization, Indonesia 2Food

Summary Pulse Lab Jakarta (PLJ) is a joint project of the United Nations and the Government of Indonesia (GoI), through the Ministry of National Development Planning (Bappenas), under the umbrella of Global Pulse. Global Pulse is a UN innovation initiative in the Executive Office of the Secretary-General that, through its Pulse Lab Network, aims to 1) promote Awareness of the opportunities Big Data, 2) forge publicprivate partnerships for data, tools and expertise, 3) conduct joint research projects to evaluate potential of new methodologies, 4) build innovative technology tools for real-time monitoring, and 5) drive adoption of new approaches across public sector. PLJ aims to collaborate with the GoI and UN country teams, and partner with private sector companies to explore the potential – and challenges - of big data analytics for policy making in Indonesia. This study aimed to rapidly assess whether social media analytics holds any potential to provide information for communicable disease control. We established that information from social media may hold some potential as an informal, complementary, data source but we were unable to assess its full value in the lab setting. Social media contains information that is related to communicable disease events and community perception of interventions. As Tweets tend to peak in volume in response to releases of information, primarily from online news sources, we found that the vast majority of Tweets were secondary and relatively uniform in content. The number of Tweets that contained primary information from individuals was very low. The value of information derived from any data source or analytic method is dependent on how it compares relative to existing sources of information (e.g. lower cost or time to gain insights, and metrics related to data quality). To be able to establish the value of information from social media analytics relative to existing sources and methods, use of social media analytic platforms needs to move out of the lab setting and into practice. Health units that process informal, online sources of information exist in Indonesia. These units present an ideal entry point for this technology. The social media analytics tool that was used for this study was a ‘plug and play’ dashboard, the ease of use of which would facilitate rapid capacity building. From a purely technical perspective, scale-up is possible. Integration however, presents a risk. The cost of implementing a commercially available social media analytics tool by the GoI has not been established and is likely to depend upon public-private partnerships. To the best of our knowledge, these do not currently exist between the GoI and social media analytic software providers. The value of the data (in terms of data quality, and time and cost savings) also remains unclear. It is therefore not possible to balance the cost of using the social media analytic platform (initial and ongoing) against its value, which does not make a strong case for investment. To move forward, more information about the process and costs for integration is required, as is a research plan that would provide more information about the value of social media analytics in this setting – particularly in terms of data quality, cost and time-saving, and opportunity cost.

 

57  

Indonesian context

The digital divide limits the representativeness of digital data in Indonesia, but not all applications of data require it to be representative  

That digital divide is slowly closing…

Indonesia is an emerging market economy that has made progress in reducing poverty over the past decade, but the gap between the rich and poor has grown due to a slow-down in the pace of poverty reduction, alongside a rapid rise in wealth31. This inequality means that there is a substantial digital divide: inequalities persist in the generation of, and access to, digital data. That divide is gradually closing as the use of the Internet is growing in Indonesia. Internet penetration (those who currently have access to the Internet at home via either a computer or mobile device) is currently estimated at 16.72% of the population (Figure E1). Figure E1. The growth of Internet penetration in Indonesia from year 2000-2014

Proportion of population with access to the internet (%)

Inequality in Indonesia leads to a marked digital divide

…but internet penetration in Indonesia remains relatively low…

18 16 14 12 10 8 6 4 2 0 2000 2001 2002 2003 2004 2005 2006 2007 2008 2009 2010 2011 2012 2013 2014 Year

…as does the use of smartphones relative to feature phones

Source: Internet Live Stats32 Of those using mobile phones, smartphones comprise approximately 23% of all handsets. This is low relative to many Asian countries, as well as Europe and the US (Figure E2).

                                                                                                                31 32

 

Indonesia Economic Quarterly, July 2014, The World Bank http://www.internetlivestats.com/internet-users/indonesia/  

58  

Figure E2. Proportion of smartphones vs. non-smartphones in different countries in the Asia-Pacific region, Europe and the USA 100% 90%

Proportion of handsets

80% 70% 60% 50% 40%

Non-Smartphone

30%

Smartphone

20% 10% 0%

Large volumes of Tweets originate from Indonesia but data from Twitter is likely to be heavily biased which limits its representativenes s, and potential for application…

Despite the relatively low proportion of the population with access to the Internet or smartphones from home, large volumes of Tweets originate from Indonesia. It is ranked as the third country worldwide in terms of Tweet volume, whereas Jakarta ranks first worldwide for cities34. This is likely to be due to Indonesia’s large population size (approximately 253 million), in addition to internet access from outside of the home (e.g. internet cafes).

…but not all applications of data require it to be representative

As this digital divide exists, signals from social media only represent a sub-set of the Indonesian population and are only likely to represent people with particular characteristics. Data from social media sources is likely to be biased in its generation by users according to a variety of social characteristics that could include economic, demographic, geographic, cultural, or political factors.

Country

Source: Nielsen33

Although this type of data cannot be used to gain representative insights about a population, not all applications of data require a complete picture of a situation. This paper focuses on the potential of social media data to provide informal, complementary insights relevant for the control of communicable diseases.

This report focuses on one potential area – as an informal and complementary information source for communicable disease control

It focuses on two main applications of information from social media: for surveillance (both early detection and continuous monitoring) of outbreaks, and for information about community perception in relation to Information Education Communication (IEC) campaigns.

                                                                                                                33 34

 

http://www.nielsen.com/us/en/insights/news/2013/the-asian-mobile-consumer-decoded0.html http://www.slideshare.net/OnDevice/indonesia-the-social-media-capital-of-the-world  

59  

Digital epidemiology

Research in digital epidemiology is growing, but in practice the application of social media analytic insights remains limited

Digital information sources will never replace traditional surveillance system but they play an increasing role in supporting them

Although internet-based approaches for disease surveillance do not have the capacity to replace traditional surveillance systems (Milanovich GJ, 2014), the use of digital data as a supplementary source of information for disease control is becoming increasingly well established globally. Salathe (2013) highlights four broad applications for digital data within the scope of communicable disease control: 1) early detection of disease outbreaks, 2) continuous monitoring of disease levels, 3) assessing disease-relevant health related behaviors and sentiments relevant to disease control, and 4) providing an additional method for examining the period before an outbreak came to light. One possibility is in the surveillance of infectious disease events. The World Heath Organization’s (WHO) perspective on informal sources of information for global infectious disease surveillance is given in box E1.

Surveillance of infectious disease events is one possibility…

Box E1. Informal sources of information for global infectious disease surveillance The rapid global reach in telecommunication, media and Internet access has created an information society permitting public health professionals to communicate more effectively. Many groups including health professionals, nongovernmental organizations, and the general public, now have access to reports on disease outbreaks, challenging national disease surveillance authorities which were once the sole source of such information. Public Internet sites are dedicated to disease news and include medicine and biology-related sites as well as those of the major news agencies and wire services. ProMed, an early electronic discussion site on communicable diseases occurrence on the Internet, provides an example. Electronic discussion sites, accessible through free and unrestricted subscription, are valuable sources of information. Their scope may be worldwide (ProMed, TravelMed), regional (PACNET in the Pacific region) or national (Sentiweb in France). They exemplify unprecedented potential for increasing public awareness on public health issues. The Global Public Health Information Network (GPHIN) is a second generation electronic surveillance system developed and maintained by Health Canada. It has powerful search engines that actively trawl the World Wide Web looking for reports of communicable diseases and communicable disease syndromes in electronic discussion groups, on news wires, and elsewhere on the Web. GPHIN has begun to search in English and French and will eventually expand to all official languages of the World Health Organization, to which it has created close links for verification. Other networks which are likewise sources for communicable disease reporting include nongovernmental organization such as the Red Cross and Red Crescent Societies, Medecins Sans Frontieres, Medical Emergency Relief International (Merlin), and Christian religious organizations such as the Catholic and Protestant mission networks.   World Health Organization35

…but surveillance using social media analytic tools remains primarily in the domain of academia

To the best of our knowledge, although informal information sources are commonly used to gain information about disease cases, tools for automated social media analysis are not. In Indonesia for example, online media sources are used as a source of informal information, but searching and extraction of relevant content is conducted manually. Through reviewing the literature (review method was not

                                                                                                                35

   

http://www.who.int/mediacentre/factsheets/fs200/en/

60  

systematic), it seems that studies relating to social media analytics for disease surveillance remain primarily in the domain of academia. It may also hold potential for better understanding community perception… This has been explored previously, for immunization in Eastern Europe, by the United Nations Children’s Fund This study explored its potential to support control efforts in Indonesia for three infectious diseases

Another potential approach could be supporting Information Education Communication (IEC). IEC refers to a public health approach “aiming at changing or reinforcing health-related behaviors in a target audience, concerning a specific problem and within a pre-defined period of time, through communication methods and principles”36. In countries like Indonesia where social media use is widespread, it is likely that messages relevant for public health efforts will be spread in the online space. Immunization is one example of an intervention for disease control that is widely affected by the dissemination of misleading information. As noted by UNICEF in their Central and Eastern European social media analytic study, it is increasingly the case that “a global, fast-paced communication environment makes it possible for negative publicity and antiimmunization positions to be disseminated quickly worldwide. Localized opposition (e.g., polio campaigns in India and Nigeria), negative publicity surrounding vaccine safety (e.g., MMR vaccination in the UK), and suspected or real adverse events following immunization are more likely to attract wide media coverage, and spread through the Internet (Clements & Ratzan 2003; Offit & Coffin 2003). Often “by the time the record is set straight, trust in immunization has been partly destroyed”. This project looked into the potential of information from social media to support control efforts in Indonesia for three infectious diseases, and considered the feasibility of moving social media analytics out of the lab setting and into practice

 

Research was conducted on avian influenza, rabies, and Middle East Respiratory Syndrome (MERS). All studies were all conducted in early 2014. To determine the research questions for social media analytic research, PLJ collaborated with infectious disease control experts at the Food and Agriculture Organization (FAO) Emergency Center for Transboundary Animal Diseases (ECTAD) (LS, JM), and the World Health Organization (WHO) (NK).

                                                                                                               

World Health Organization. http://www.emro.who.int/child-health/community/informationeducation-communication.html 36

 

61  

Methodology The proposed areas for research were in line with organizational priorities, and in areas in which they perceived that social media analytics might have the potential to add value. Table E1. Characteristics of disease agents, existing surveillance measures, and the aim of conducting social media analytics Disease

Avian influenza

MERS CoV

Rabies

Infectious agent

Avian influenza A - virus strains H5N1, H7N9

Novel coronavirus

Rabies virus

Transmission

Zoonotic

Zoonotic likely

Zoonotic

Reservoir of infection

Avian (chicken, ducks and quail)

Not yet fully determined; camels likely

Dogs

Epidemiology

H5N1 was first reported in humans in 1997, but re-emerged in 2003 and 2004. 15 countries worldwide have reported human cases including Indonesia. Between 2005 and December 2013, 195 cases with 163 fatalities were reported in Indonesia (WHO SEARO).

First identified in Saudi Arabia in 2012. Appears to be circulating throughout Arabian peninsula. All recent cases outside the Middle East appear to have been exported. No known cases in Indonesia (WHO accessed 9th July 2014).

Heterogenous across Indonesia. Introduced to Bali in 2008 where outbreak peaked in 2010 (~10-12 human deaths/month). Intensive control led to decrease in cases but control measures lessened, as it was perceived as less of a public health problem. More recently cases have increased in frequency.

“WHO encourages all Member States to enhance their surveillance for severe acute respiratory infections (SARI) and to carefully review any unusual patterns of SARI or pneumonia cases. WHO urges Member states to notify or verify to WHO any probable or confirmed case of infection with MERS-CoV (WHO accessed 9th July 2014)

Control measures in Indonesia have included mass dog catching/vaccination, humane euthanasia of rabid dogs, rapid response, post-vaccination surveillance and communication.

H7N9 was first found in China in March 2013 (WHO). Existing surveillance/ control

Active and passive. Reliant on local government structures, >2500 Participatory Disease Surveillance and Response (PDSR) officers trained. Community sensitized to report to PDSR officers. Officers report positive cases (by rapid antigen test) on paper (all reports usually received after ~6-7 weeks) or SMS gateway. Event location by centroid-based GPS and village ID.

Research aim - to use social media to gain descriptive insights on:

1

 

Cases of sudden death, high mortality in chicken, ducks and quail in all of Indonesia

2

Humans who have respiratory symptoms and have traveled in the Middle East and returned to Indonesia

3 Human exposure to dog bites in Bali. 4 Community perception about culling dogs.

62  

Analysis method Twitter data was used as it is accessible

For extraction and basic analysis, a commercially available data analytics platform was chosen both because of a pre-existing partnership, and its ease of use Infectious disease control experts from partner organizations determined the research question and selection criteria. PLJ developed the subject specific taxonomy for data extraction. The Tweet sample was refined for relevance using automated algorithmic classification Analyses varied by question according to the sample of Tweets extracted

Publicly available data from the micro-blog service ‘Twitter’ was used as it was easily accessible to us. Post content is more accessible than that of other social media platforms due to the characteristics of its use: Twitter users tend to set the privacy settings on posts to public, whereas users of another popular social network, Facebook, tend to opt for posts to only be visible within their network of mutually accepted ‘friends’. Tweets’ have a strict content limit of 140 characters. Users can post tweets containing their own text and/or image content and/or links to external online sites, forward information from other users to people within their own networks by ‘re-tweeting’ their tweets, mention other users in text content (e.g. @user_A what do you think about this?), and explicitly mention the topic of the tweet (e.g. #immunization is good). Flow of information through the social network is dependent on network relationships which can either be one-directional or bi-directional e.g. if user A follows user B but user B does not follow user A, the relationship is one directional. If users C and D follow each other, the relationship is bi-directional. A commercially available data analytics dashboard, Crimson Hexagon’s ForSight™ platform, was used to extract and conduct automated descriptive analyses on micro-blog posts, ‘Tweets’, from a stream of realtime data called the Twitter firehose. The ForSight™ platform was chosen due to United Nations Global Pulse’s (UNGP) pre-existing partnership with Crimson Hexagon, and its suitability for purpose: it was considered that the ease of use of the ‘plug-and-play’ dashboard could potentially facilitate rapid capacity building. Subject experts from FAO ECTAD (LS, JM) and WHO (NK) determined Tweet selection criteria based upon their a priori subject and context-specific knowledge. PLJ data engineers (SIB, TT) and a Monitoring & Evaluation (M&E) specialist (SJ) worked together to develop Tweet selection ‘taxonomies’. The taxonomy is determined by, and extraction based on a combination of Boolean operations (Boolean referring to a data type with only two possible values – true or false). For example, given “(A or B) AND (C or D)”, then “A AND C” is a true case. Taxonomies for each of the four research aims described in table 1 are given in table 2. An Indonesian speaking data engineer (SIB, TT) then entered the taxonomies into the ForSight™ software platform along with a language taxonomy that aims to only capture Bahasa Indonesian Tweets and exclude similar languages (e.g. Malay), and then ran the software. This extracted the Tweets. The automated language selection tool is ‘blackbox’ (the internal workings are not known to the user, only the inputs and outputs) and Bahasa Indonesia is listed by the ForSight™ software platform as an ‘unsupported’ language (unlike others which it supports). The validity of this language tool is therefore unknown. The data engineer (SIB, TT) then trained the ForSight™ platform to classify tweets as relevant or irrelevant: the software provides an automated selection of Tweets to the user. The user then responds by clicking a button that indicates whether that particular Tweet is relevant or not. The algorithm built into the software then uses the information input from the user about the subsample to classify the rest of the Tweets in the sample as relevant or irrelevant. The Tweet sampling method and the algorithm the software uses is blackbox (unknown). Figure 3 illustrates an overview of the data extraction and classification process. Different analysis methods were used for each research question. After the sample was extracted for each research question, a judgment call was made (by SJ) as to the type of insights that could be gained, and the most appropriate way to analyze the data. These analyses are described in the following text.

 

63  

Table E2. Taxonomies defined for each research question

A sensitive taxonomy was defined for avian influenza. This was aimed at getting a broad overview on events and looking into the relevance of the information A specific taxonomy was defined for rabies – our interest was only in bite cases The MERS taxonomy was designed to be specific – the aim was to detect cases, so selection criteria included symptoms, and travel to specific locations

The taxonomy for dog culling for rabies was also designed to be highly specific. It aimed to understand community perception the progression of events related to one specific public announcement

 

1. Avian influenza ("flu burung" OR "flu unggas" OR ( " bird flu " OR " avian flu " OR " avian "avian flu" OR "avian influenza" flu " OR " avian influenza " OR " dead OR "ayam mati" OR "unggas mati" chicken " OR " dead birds " OR H5N1 ) OR H5N1) AND -language:ms AND -language : ms AND language: id AND language:id 2. Rabies (cases of exposure to dog bites) ("rabies" OR "anjing gila" OR "digigit (" rabies " OR " mad dog " OR " dog bite anjing" OR "gigitan anjing") AND " OR " dog bites " ) AND -language : ms language:ms AND language:id AND language: id 3. MERS CoV (pulang OR kembali) AND ((umrah (return OR back ) AND ( ( Umrah OR OR umroh OR haji) OR ("Timur Hajj Umrah OR ) OR ( " Middle East " Tengah" OR Arab OR Jedah OR Arab OR OR OR Jeddah Jeddah Makkah Jeddah OR Mekah OR Makah OR OR OR OR Makah " holy land " OR OR "tanah suci" OR Riyad OR Riyadh Riyad Riyadh Tabuk OR OR OR Medina OR Tabuk OR Madinah OR Medina Najran OR OR OR Asjr UAE Medinah OR Najran OR Asjr OR OR " United Arab Emirates " OR Turkish UAE OR "Uni Emirat Arab" OR OR Egypt OR Tunisia ) ) AND ( fever, Turki OR Mesir OR Tunisia)) cough OR OR OR shortness of influenza AND (demam OR batuk OR sesak OR flu OR influenza OR ( ( not OR not OR flu OR influenza OR influensa OR not ) AND " malaise " ) OR sore OR ( (tidak OR gak OR nggak) Corona OR OR OR Mers OR corona AND "enak badan") OR sakit OR CoV AND -language:ms AND MERS OR Corona OR Korona language:id OR CoV) AND -language:ms AND language:id 4. Rabies (community perception around dog culling) (( "Anjing" OR "Cicing" OR (( " Dog " OR " Cicing " OR " kuluk " "Kuluk" OR "Hewan" ) AND ( OR " Animals " ) AND ( " Rabib " OR " "Rabib" OR "rabies" OR "Bali" OR rabies " OR " Bali " OR " Roaming " OR "Roaming" OR "jalanan" OR " street " OR " Stray " OR " wild " OR " "Stray" OR "liar" OR "Rescue" OR Rescue " OR " savior " OR " elimination " "penyelamat" OR "Elimination" OR " elimination " OR " kill " OR " OR "eliminasi" OR "membunuh" slaughter " OR " kill " OR " kill " OR " OR "membantai" OR "Kill" OR petition " OR " petition " OR " "membunuh" OR "Petition" OR vaccination " OR " vaccination " OR " "petisi" OR "Vaccination" OR Bali 's Governor " OR " Bali Governor " "vaksinasi" OR "Bali’s Governor" OR " Made Mangku " OR " Mangku OR "Gubernur Bali" OR "Made Pastika " OR " Pastika " OR " Mangku " ) Mangku" OR "Mangku Pastika" ) OR (" Savebawa " OR " Savebalidogs") OR "Pastika" OR "Mangku" ) ) OR AND -language:ms AND language:id ("Savebawa" OR "Savebalidogs") AND -language:ms AND language:id

64  

Figure E3. The data extraction process

Data extraction involves applying taxonomies, then training the software to algorithmically classify Tweets by relevance

• [Topic + Indonesian language] taxonomies applied Twitterverse

Selected Tweets

The software also produces automated analyses, but the majority of analyses conducted were manual and qualitative

Relevant tweets

• Training sample manually classified for relevance • Algorithmic methods used to classify full set

• Automated analysis: Visualization of overall trends • Manual analysis: Detection of cases

The ForSight™ social media analytics platform provides automated visualizations. The ‘metrics’ section of the dashboard provides descriptive visualizations of Tweet frequency in a time-series, and lists sites, hashtags, and retweets that have appeared in the Tweet sample’s metadata at high frequency. The ‘explore’ section provides visualizations of trending and clustered topics and words (blackbox but most likely a function of the frequency and co-occurrence of different words mentioned in Tweet content), in addition to a list of all of the posts and their content. This post list was downloaded and qualitative analysis conducted by an Indonesian-speaking data engineer for all research questions. Descriptive data about users was missing for the vast majority of Tweets

The dashboard is also designed to enable the visualization of descriptive information about those who generated the data - a list of authors who tweeted the most about a topic, their geographic location (if Tweets were sent from a GPS enabled phone and the user granted public access to their location data), and their demographics (where users volunteered the information publicly). As this descriptive metadata was missing for the vast majority of Indonesian language tweets, none of this information was considered in the analysis. The analysis methodology for each research question is described below: Avian influenza

Analyses differed for each topic of interest

Selection criteria were determined by a veterinary epidemiologist (LS) and a taxonomy developed (LS, SJ, SIB). This was run by a data engineer (SIB) and extracted 170,000 posts between 30 June 2011 and 08 March 2014. This time period was chosen as prior to 2011 Twitter use in Indonesia was low and there were few signals. The data engineer (SIB) then used a small sample (~100 Tweets) to train the algorithm to automatically classify Tweets as relevant or irrelevant. When this algorithm was run of the whole data set, it classified ~126,000 Tweets as relevant. To gain insights on the type of information that was captured, a random sample (n=20) was selected for each ‘theme’ automatically identified by the ForSight™ platform (SJ, SIB). 20 were selected because it is the closest round number to 19 - the minimum sample required to approximate data quality using Lot Quality Assurance Sampling. Relevant Tweet content was defined as events that had occurred in Indonesia that had syndromically resembled avian influenza (cases of sudden death, high mortality in chicken, ducks and quail) that had occurred in Indonesia). An Indonesian-speaking data engineer (SIB) then manually classified 360 Tweets for relevance (20 Tweets for each of 18 themes), and worked with the M&E specialist (SJ) to compare the findings for different categories to gain insights about the relevance of the sample.    

 

65  

MERS Selection criteria were determined by an infectious disease specialist (NK) and a taxonomy developed (SJ, SIB). This taxonomy was run by a data engineer (SIB) for the time period of January 2012 to May 2014. The time period was determined according to its epidemiological relevance – cases of MERS CoV have been reported to WHO since April 2012 (NK). MERS CoV and Indonesian language taxonomies were applied (SIB) to extract 6492 Tweets. A small training subset (62 Tweets) was then manually labeled for relevance (SIB) and the trained algorithm used to classify the full data set. 4936 Tweets were classified algorithmically as relevant, and these downloaded for qualitative content analysis by an Indonesianspeaking data engineer (SIB). A software limitation of Crimson Hexagon is that it is only possible to download content of 1000 tweets/day. As ‘relevant’ tweets exceeded 1000 on 7th May (1471), we were only able to manually check 2/3 of tweets on that day. The data engineer (SIB) sorted posts that were considered relevant (by syndrome and location) and worked with the M&E specialist (SJ) to extract content (number of cases and their location) about events linked to possible MERS cases in Indonesia. Rabies (dog bite cases) Selection criteria were determined by a veterinary epidemiologist (LS) and a taxonomy developed (LS, SJ, SIB). This was run by a data engineer (SIB) and extracted 133 posts between 30 June 2011 and 08 March 2014. This time period was chosen as prior to 2011 Twitter use in Indonesia was low and there were few signals. As the sample size was very low it was possible to analyze the content of each post individually. Qualitative analysis of content was conducted by an Indonesian-speaking data engineer (SIB). Rabies (community perception about culling dogs) Selection criteria were determined by a veterinary epidemiologist (LS) and a taxonomy developed (LS, SJ, SIB, TT). As known events surrounding public statements for dog killing occurred in July 2014, Tweets from July were initially extracted from the firehose by data engineers (SIB, TT). Multiple iterations of extraction were conducted by a data engineer (TT) to get a sense of ongoing signal patterns and identify the events that had occurred in the months prior to July. These automated visualizations from the ForSight™ social media analytics platform were then cross-checked though manual searches on online media, and qualitative analyses conducted (by TT) to build a narrative of events occurring prior to the public announcement about dog killing.

 

 

 

66  

Findings   Information related to disease control was disseminated through social media and we could analyze it rapidly…

Relevance of the data It was possible to extract information about communicable diseases in Indonesia from social media, and we were able to find it quickly – using commercial social media analytics platforms it is possible to mobilize data collection within a matter of hours. Tweet content therefore presents a potential source of unstructured information that is now available relatively rapidly in unprecedented volumes. High volumes of tweets were found for avian influenza and MERS. Very few tweets were found for exposure to dog bites for rabies (n=133) or dog culling. The vast majority of the Tweets captured for dog bites were either jokes or derogatory comments that had used the words crazy and dog. Only two Tweets referred to actual bites (one human and one puppy bitten). When rabies was explored from the angle of the communities’ perception on dog culling, relevant information was captured. Posts about dog culling were not continuous but seemed to be triggered by releases of information e.g. political statements or graphic videos.

…but tweet volume and relevance varied widely by topic…

Outbreak experts from WHO/USAID/CDC looked briefly at the information collected about the MERS outbreak and got the impression that it was roughly consistent with the informal outbreak reports that they had received from other sources. This impression is only anecdotal - triangulation of the data with existing surveillance system data was not conducted. From a sample of 360 posts about avian influenza, 58% (207/360) related to either an incident or information relevant to Indonesia. 12% (42/360) were about events in a different country (predominantly China), and 15% (53/360) were about corruption/politics. Within the avian influenza topic, the software’s algorithm automatically identified 18 themes (by word association). These themes reflected the broad scope of the taxonomy. When a small sample (Lot Quality Assurance Sampling) of Tweets was taken for manual analysis it was seen that relevance of Tweets varied widely by theme.

…and by themes within topics

Refinement of taxonomies could be conducted to improve relevance…

Refinement of taxonomies could be conducted to improve relevance as classification algorithms were only trained with a small subset of data - it is likely that tweets were missed through misclassification. Keyword selection criteria need to take the trade-off between sensitivity and specificity into account – for MERS we combined words relating to travel with geographic regions and symptoms to improve the ability of the analytics tool to pinpoint relevant tweets. Refinement of all of the taxonomies may be able to improve relevance of extracted Tweets, although it was not conducted for this study as it was considered unlikely that improvements in relevance would be sustained over time.

…but as most Tweets represented a secondary, not a primary source of information, it seems unlikely that improvements would be sustained over time

The majority of the tweets were secondary sources of information, predominantly from online news sources - although some tweets from individuals were also extracted. As an example for MERS, “recently my friend was asking me about MERS symptoms. He just got back from Umrah and is having respiratory symptoms similar to pneumonia”. As content related to news stories was found to dominate the Twitter feed and Tweet content appeared to shift rapidly with new releases of information, selection criteria that capture highly relevant Tweets at one time period are likely to capture irrelevant Tweets in another. As very little demographic information is available about gender, age, or location of Twitter users, it was unclear whom the data represents – or whether it captured information from the people that we aimed to target (those experiencing communicable disease events or interventions). In the case of rabies it was clear that Tweets did not represent the whole population as animal rights groups dominated the social media landscape in relation to content about the killing of dogs.

It was unclear whom exactly the sample captured, but in the case of rabies it was clear that Tweets did not represent the whole population…

 

67  

…although this could be because keyword logic for dog killing was generated based upon prior knowledge about the situation…

It is also possible that this observation about those posting content about dog killing was an artifact of keyword logic. Taxonomy selection criteria were generated based upon prior knowledge about the situation and included the names of animal rights groups and campaigns. This then, biased sample selection criteria towards fitting what was already known or assumed about the communities perception about killing dogs, rather than collecting an objective sample. Another bias with social media content comes from the user. Online perceptions may not represent offline perceptions as people use different outlets of communication for different purposes and communications can shift from one medium to another. This movement of ideas from one source to another was exemplified in this research through the peaks of twitter activity related to the release of information from news sources – many Tweets directly mentioned or linked to the sources themselves.

…or because social media users use social media as one of many communication outlets with different purposes…

As described in Wilcox and Stephen (2012), “people use social networks to fulfill a variety of social needs, including affiliation, self-expression and self-presentation (Back et al. 2010; Gosling, Gaddis and Vazire 2007, Toubia and Stephen 2012)”. “Although “Facebook profiles have been shown to reflect actual characteristics as opposed to idealized characteristics that do not represent one’s actual personality” (Back et al. 2010)”, “people present a positive self-view to others on social networks” (Wilcox and Stephen 2012). Therefore social desirability bias (that people express what they think others want to hear, not what they actually think) should be given careful consideration when interpreting information.

…or want to present themselves in a particular way online As more people come online, social media data is likely to better represent the population but a shifting baseline compromises time-series analyses

In the long-term as more people come ‘online’ in Indonesia social media will be used by a greater proportion of the population. As the population ‘catchment’ of technology increases, it will expand to incorporate groups of people with more diverse characteristics (e.g. demographic or socioeconomic) and may represent a more complete picture. In the short-term however, the uptake of technology leads to a shifting online baseline. This makes any comparative evaluation of one time period to another problematic: any shift in either represent a shift in events or perception – or alternatively, a shift in the attributes of the online population.

Automated analyses were helpful to quickly visualize content at any given point in time but they were not sufficient in isolation…

It was possible to use the ForSight™ platform’s automated analyses to gain quick visualizations of trending topics. Several of these visualizations are shown on the following page (figures E4-E6). Automated volume analyses appeared to be meaningless beyond checking whether Tweet content existed on a topic or not, as they did not respond to any one particular event (the reasons behind Tweet volume were multifactorial), data was noisy (not sensitive/specific). As these analyses were insufficient in isolation, qualitative analysis of content (manual) was necessary to gain an understanding of the meaning behind the visualizations.

  The analysis approach  

Manual analysis of Tweet content enabled more meaning to be gleaned from social media content – and in the case of MERS, specific information – to be extracted from Tweets. Figure E7 illustrates how people who had returned from the Middle East with respiratory symptoms featured on social media between the 5th and 13th of May (shown for Java – the most populated island in Indonesia).

…so manual, qualitative analysis was essential to establish any meaning behind trends

 

 

 

68  

Figure E4. A word cloud enables users to get a very quick idea about which words are trending on a topic within a specified time period

Figure E5. A topic wheel visualizes trending topics in more detail – here illustrating how words are trending within different subtopics

Figure E6. Snapshots of discrete time periods capture topics as they emerge (for MERS during one day – 08.05.2014)

Time period:

 

12.00am06.00am

6.00am12.00pm

12.00pm6.00pm

6.00pm11.30pm

69  

Figure E7. Number of cases linked with MERS on social media (illustrated for deaths and ‘susp.’ = suspected cases)

The value of social media analytic data was not established in this study… …and information gleaned about disease events or perception of interventions is not generalizable to other settings, but some more general insights gained from this approach may help in establishing future direction

The value of social media analytic data was not established in this study because value of information derived from any data source or analytic method is dependent on how it compares relative to existing sources of information (e.g. lower cost or time to gain insights, and metrics related to data quality). It is dependent upon the data that is collected within the existing system. Information from online media is already used as an informal data source in health units in Indonesia. To be able to establish the value of information from social media analytics relative to existing sources and methods, use of social media analytic platforms needs to move out of the lab setting and into practice. Generalization of the findings about each of these diseases to other contexts would be inappropriate: Contextual factors that play a role in the generation the data (the use of social media), occurrence of events (epidemiological), and community perception of interventions are likely to be localized. The generalizability of our findings for tweet content analysis is also limited by the fact that the three diseases assessed were zoonotic, trans-boundary viral diseases that were emerging in the population of interest. However, some of the general insights gained from this approach may help in guiding the direction of social media analytics for communicable disease control in Indonesia in the future.

 

 

70  

Although social media appears to hold potential in theory, it remains unclear as to whether social media holds any value as an information source for communicable disease control in practice

Moving forward Social media appears to hold some potential for obtaining information relevant to communicable disease control. Information related to disease control was disseminated through social media and we could analyze it rapidly. However, we were not yet been able to establish the full value of the data through this study. This is because the value of information gained form social media analytics can only be established when it is compared with data collected within the existing system. We did not triangulate the data with data from other sources, nor are we able to say whether the time taken to create and run the taxonomy was any more rapid than existing methods to obtain information from informal sources e.g. searching for online media through a search engine. It is therefore unclear how much value information gained from social media analytics holds relative to other sources of information. The value social media analytics is likely to vary on a case-by-case basis, as generation of social media is dependent on disease characteristics and the local context.

It is likely that its value will vary on a case-by-case basis but theoretically speaking, information from social media is likely to be timely relative to traditional surveillance systems…

As social media analytics can be rapidly mobilized, and once mobilized, information bypasses hierarchical systems of data flow (e.g. village to sector to district to central) it has the advantage of being a timely informal source of information for disease event monitoring. The value of the tool is dependent upon the timeliness of the existing surveillance process: we learned from key informant interviews with experts that with some diseases it can take several weeks for data in the central database to be complete. In previous instances, outbreaks had been identified through local news sources before they were received on the surveillance system. Where systems function heterogeneously (as oppose to a perfectly functioning system), triangulating existing surveillance system data with information from social media could help identify cases that were not captured by the surveillance system, and identify geographical gaps in the system that require strengthening.

…and may hold value where existing systems are heterogenous...

As was exemplified with MERS, information from social media can be used for informal monitoring of emerging diseases for which no existing surveillance system is in place. However, when there is no existing surveillance system in place, each case reported requires extensive individual follow up for validation. This is a highly resource intensive approach and the sensitivity/specificity of the information contained in social media needs to be relatively high so that resources are not wasted on following false leads.

…or where no surveillance system exists

In the case of trans-boundary diseases, social media analytics also has the advantage that it can be used across geopolitical boundaries.

It can also be used across geopolitical boundaries

As relevance varied by topic, and appeared clustered over time series within the same topic, generation of a generic ‘one size fits all’ tool may not be the most appropriate approach - any approach needs to be flexible so that it can be rapidly adapted to capture emerging issues of interest. Similar findings have been found in other studies: a study on immunization perception by UNICEF in Eastern Europe noted that over time social networks will evolve, and that it is expected that “consumers and providers of information, will constantly change in terms of channels and tactics”. An adaptable tool is also required from the perspective of the intervention - there are no ‘one-size-fits-all’ interventions. In the case of IEC “strategies with tailored messages that use appropriate channels are required to reach specific segment of the population, whether decision-makers or remote, “hard to reach” populations.” (Waisbord & Larson).

Social media analytic approaches need to be adaptable so fixed tools and indicators are not likely to be useful Other social media analytic techniques are possible, although we currently lack the capacity to conduct them at PLJ

With the methods we employed we were able to gain automated information rapidly (such as trending topics), and use a manual qualitative analysis approach to gain deeper insights. We did not conduct automated sentiment analysis as it not currently available to us at PLJ in Bahasa Indonesia, nor did we conduct network analysis. It was not considered necessary to answer the questions we posed, nor was it considered appropriate for gaining rapid, low-cost insights: the method currently available to us is manual, and time-consuming.

 

71  

Moving forward, social media analytics will need to be moved from the lab and into practice to assess its full potential, but not all of the information required to make an informed decision about this is currently available   Social media contains information that is related to communicable disease events and community perception of interventions. As Tweets tend to peak in volume in response to releases of information, primarily from online news sources, we found that the vast majority of Tweets were secondary and relatively uniform in content. The number of Tweets that contained primary information from individuals was very low.

To fully understand the value of social media analytics it will be essential to move social media analytics out of the lab and into practice. Information from social media is likely to add the greatest value if it is incorporated into monitoring units for use alongside existing informal information sources. The value of information derived from any data source or analytic method is dependent on how it compares relative to existing sources of information (e.g. lower cost or time to gain insights, and metrics related to data quality). To be able to establish the value of information from social media analytics relative to existing sources and methods, use of social media analytic platforms needs to move out of the lab setting and into practice. Putting it into practice will also be able to give us a better idea as to whether time-appropriate information can be gleaned on an ongoing basis. Health units that process informal, online sources of information exist in Indonesia. These units present an ideal entry point for this technology. The social media analytics tool that was used for this study was a ‘plug and play’ dashboard, the ease of use of which would facilitate rapid capacity building. From a purely technical perspective, scale-up is possible. Integration however, presents a risk. The cost of implementing a commercially available social media analytics tool by the GoI has not been established and is likely to depend upon public-private partnerships. To the best of our knowledge, these do not currently exist between the GoI and social media analytic software providers. The value of the data (in terms of quality, and time and cost savings) also remains unclear. It is therefore not possible to balance the cost of using the social media analytic platform (initial and ongoing) against its value, which does not make a strong case for investment. More information about the process and costs for integration is required, as is a research plan that would provide more information about the value of social media analytics in this setting – particularly in terms of data quality, cost and time-saving, and opportunity cost.

 

72  

F. Research strategy: some observations from practice at Pulse Lab Jakarta Working to the strengths of big data Observation #1: Many new digital data sources do not and perhaps will never meet validity requirements sufficiently to be considered as robust, ‘high-level’, statistical indicators by statistical agencies. This is due to their secondary nature, asymmetrical use, and anonymisation - the lack of user characteristics prevents adequate disaggregation and it is often unclear as to whom the data represents. Potential strategy: Consider moving more towards mixed-methods and participatory approaches - particularly combining big data analytics with more extensive use of qualitative methods for the interpretation of big data. Reasoning: • Big data is most likely to be better suited to use on a much more local level where information gleaned from analytics can be considered alongside other informal information sources within the local context. A more localized use of big data is also in line with a common use-case of big data in the private sector (product ‘individualization’). • Advocating for the use of big data for ‘high-level’ statistical indicators within the development community is more likely to cause a ‘big data backlash’ than other potential approaches where big data is used as a complementary, informal source of information. Implications: A ‘data revolution’ in this sense would involve increased use of mixed-methods, and moving away from a ‘top-down’ narrative through aiming to use new data sources as a way to encourage dialogue (both laterally and from the bottom up).

Considering which approaches are likely to be most successful given the lab setting and data ecosystem, including innovative approaches outside of the realm of experimentation Observation #2: Pulse Lab Jakarta’s (PLJ’s) work programs face a challenge in determining how to respond to local demands in relation to counterparts (observations 5-7) while contributing to the wider evolving policy debate (observation 4). Potential strategy: Clarify the products of the lab and strengthen the conceptual foundation of work plans by incorporating ‘contribution analysis’ and ‘theory of change’. Reasoning: Better defining the product will help focus internal R&D plans; and enable PLJ to present a more understandable product externally. Big data is currently presented externally as a diverse field and this seems to spark interest in it, but it is evident that it can be difficult for counterparts to envision how the theory translates into a tangible product that adds value to existing programs and policies. Implications: Two main categories of products seem relevant for PLJ – knowledge products that contribute to the wider policy debate, and data products (both discussed below). These different products require different strategies. Observation #3: PLJ’s research strategy has so far focused on experimentation. It seems that innovation through experimentation may prove challenging for labs that are isolated from day-to-day problems on the ground (“necessity is the mother of invention”), especially when data is frequently inaccessible as the data ecosystem is fragmented. Potential strategy: Considering the potential for success of a variety of different strategic approaches to data innovation given the lab setting, its internal capacity for experimentation, and the fragmented ecosystem. Reasoning: Although it is likely that experimental research could be better targeted to meet with a higher chance of success, other approaches to innovation may also be worthy of consideration: it is possible that they may be met with a higher chance of success, or contribute more to the overarching policy debate than experimentation; or that they may be better suited to a relatively low capacity (in terms of research team size) research environment. Implications: See observations 5-7 for ‘focusing on making experimental research programs demand driven and well targeted’. Other innovations options for consideration could include:

 

73  

• Identifying and providing expertise to strengthen shadow systems (i.e. not looking at the system of how M&E ‘should’ be done, but how it is actually done by program managers at the local-level and providing expertise to help strengthen these local approaches) • Validating existing innovations and assessing their scalability through replicating and evaluating them in different local contexts • Encouraging bottom-up innovation (e.g. mini-grants) • Increasing civil society involvement with decision-makers (e.g. hackathons, data challenges, and other events) • Acting as a coordinating body for innovative public-private research partnerships • Advocating for overarching issues such as data accessibility and privacy (in line with the open data agenda)

Contributing to the wider policy debate Observation #4: PLJ’s role remains unclear in respect to its contribution to either domestic or international (e.g. ‘data revolution’) policy. Potential strategy: Clarify the overarching role of PLJ’s research in the wider policy debate – the policy questions that the research aims to generate evidence towards. Reasoning: The role of research in the wider policy debate will influence the probable impact of different types of research; and the political context surrounding research could pose a conflict of interest in terms of balanced presentation of evidence. Researchers need to be aware of these drivers so that research can be targeted towards priority areas, and so that evidence can be critically assessed and reported in a balanced way. Implications: • Be conscious not to blur the line between research and advocacy. • Clearly state the research rationale; improve documentation to increase transparency of the methodology and enable reproducibility of the findings; and present a balanced and critical assessment of the research findings including the limitations as well as the strengths. • Consider how can we best capture systematic results and lessons learned from iterative innovation experiments and feed them back into the activities own results management cycle, and into the overarching performance monitoring frameworks.

Focusing on making experimental research programs demand driven and well targeted Observation #5: Some solutions are likely to be more appropriate than others, particularly in environments where data is fragmented Potential strategy: Consider putting a greater focus on targeting based upon attributes of the data and the context, in addition to the existing, thematic targeting approach. Reasoning: Some types of data and analysis techniques are more appropriate for answering certain types of questions than others; data needs context for appropriate interpretation; and to be able to respond to data it has to be available at the scale of the intervention. Implications: An approximation of the types of data sources, questions, and needs that are likely to be a successful/unsuccessful match, consider the matrices is provided in the report/toolkit. For example: • From the perspective of the question, an approach that looks at how to use big data methods to pull together diverse sources of information for descriptive analysis may be more successful/impactful in a fragmented data ecosystem than a predictive analysis approach (success heavily dependent on measuring the right variables, and therefore data availability); • When considering likely entry points and types of data, Type 2A data at a sub-national level could be an approach worthy of consideration. However, note that this matrix is an approximation: each scenario for application is highly specific and will require consideration on a case-by-case basis by someone trained in a discipline-specific area of data application (for more information on this see the notes in the team section of the report/toolkit).

 

74  

Observation #6: To date, experimentation at Pulse Lab Jakarta has not been bred out of identified problems but by the availability of the solution (e.g. social media analysis) Potential strategy: re-defining the problem statement to contain a dilemma statement (in Italics) e.g. “how can we accelerate innovation and uptake of big data for sustainable development, while still maintaining the capacity to respond flexibly to a problem to deliver the most appropriate solution?” Reasoning: Working from the basis of the solution makes the solution space for any given problem very small. If solutions are generated, and problems identified secondarily it is unlikely that impact will be made on programmatic implementation or policy. Implications: Capacity needs to be increased to enable the following: 1. High quality needs assessment. Requires training and experience in needs assessment (e.g. actor analysis, defining the problem etc.); discipline-specific knowledge and experience (e.g. economics, health, environment, transport etc.); and a systematic approach. Possible options could include: a) Building in house capacity to conduct needs assessments: could be challenging as PLJ works on numerous thematic areas and an in depth domain specific knowledge of existing data systems and decision-making is required. A more focused operational model e.g. a micro-segmentation approach where customers for a specific product are limited to small segments or even individuals and product industrialization is then conducted, could facilitate in house needs assessments: the application of the product would be well defined, and ‘things to look out for’ would become increasingly apparent through practice. b) Needs assessment as the responsibility of the client: PLJ specific Standard Operating Procedures (SOP’s) and checklists would probably need be developed to ensure that all needs assessments conducted by clients are conducted to a minimum standard before further investment is put into developing a solution. 2. Increased ability to ‘screen’ potential projects to identify the ones that are likely to be most successful. Relates to observation #5. 3. Increased flexibility to respond to needs through diverse methodology and solutions. To widen the available solution space, the following activities may be worth considering: • Defining existing ‘core’-staff’s research capacity (e.g. programming languages, experience with different analysis techniques/data sources). If a micro-segmentation approach is taken, perhaps this could be in line with a specific ‘core’ product that Pulse Lab Jakarta would like to increase uptake of in the development sector? • Build capacity to conduct mixed-methods research and project design (staff who have worked with a variety of different quantitative/qualitative techniques) • Defining needed and highly skilled areas of expertise (as an example, spatiotemporal modeling) and form partnerships in these areas • Building a database of skillsets in different organizations so that clients can be appropriately referred to other specialist organizations Observation #7: Partner organization incentives affect research quality Potential strategy: Considering different operational models, or using different approaches with partners who have different funding mechanisms Reasoning: With PLJ’s current operational model, incentives appear to present a challenge. It seems that in a situation where data scientists provide free big data technical expertise and rely on domain experts to identify opportunities within their own programs for the use of big data, that partners often do not fully invest and are more likely to move forward with product development without a concrete application in mind. Lack of ongoing investment could also present a challenge to project sustainability. This is particularly likely to be a problem where the project is a ‘side project’ and it is not integrated into the partner organization’s annual work plans. Implications: A common approach with fee-paying clients is for the client to conduct a needs assessment, obtain funding, and then create a detailed, standardized Terms-of-Reference (TOR) document that contains information about the needs and the desired end product to sub-contract a programmer. Some commercial developers also insist on a ‘maintenance contract’ to ensure that a project is sustainable for a minimum period of time after initial development. Would using a ‘proposal-based’ operational model at PLJ shift the incentive dynamic more towards greater investment?

 

 

75