Big Data: Big Potential

Imagine being able to gain information instantly about whether a development project is reaching its intended recipients, or where the next cholera outbreak might be, or track the fluctuating wealth of every district in Cote d’Ivoire. These are some of the possibilities of Big Data analysis, where data of increasing volume, velocity and variety is mined for the unprecedented patterns and insights it may yield.

More data was created in 2013 than in all the preceding years of human history combined, and every minute the world generates enough data to fill more than 360,000 standard DVDs. This includes tweets, public Facebook posts, geotags that locate where photos were taken and news stories. It can also include de-identified records of mobile phone activity, anonymised internet search data, consumers’ purchasing trends, and soon perhaps, transcripts of talk-back radio. In the near future for those seeking to understand demographic change, official government statistics will be just one component of a much richer picture.

The long-term implications are vast. In the short-term, development practitioners can look to Big Data to provide faster and more detailed feedback on the impact of projects, or allow donors to pre-empt where other initiatives are required.

With such opportunities in mind the United Nations has called for a “data revolution”.

“When we speak of a data revolution, we are calling for a sustained, transformative effort to improve how data is produced, used and disseminated,” UN Deputy Secretary-General Jan Eliasson told a World Bank event in April. “This will generate high-quality, timely and sustainable data in support of the post-2015 development agenda.”

Robert Kirkpatrick, executive director of the UN Global Pulse programme, said the data revolution, mainly through the use of Big Data, has arguably already arrived, but not in the sector where it is needed most.

“The data revolution has already happened in the private sector. We have to figure out how we can harness the innovation that’s already out there, changing many aspects of society, for development and humanitarian action.”

For example, the retailer Tesco combines weather information with its own sales history to predict demand for certain items and prevent waste. Adidas is able to immediately redirect supply trucks to stores running low on a particular style of shoe, based on its real-time monitoring.

However, for all its potential, there have so far been few large-scale, practical adaptations of Big Data in the global development sector.

“Big Data should be embraced and become an integral part of a data revolution, but for it to happen lots of effort is necessary,” said Dorota Panczyk-Piqueray, an economic analyst at EuropeAid. “Big Data is currently rather underused by development practitioners. Except for in the private sector, Big Data has been explored through research rather than in the development community itself.”

The UN’s Global Pulse Initiative, launched in 2009, forges partnerships between UN agencies, governments, academia and the private sector to accelerate discovery, development and adoption of Big Data and real-time analytics for sustainable development and humanitarian action. Pulse Labs in New York, Jakarta and Kampala research and develop projects examining how new sources of digital data can support development goals.

Mr Kirkpatrick recently held an information session at EuropeAid in Brussels to highlight the potential for collaboration between Global Pulse and the European Union. He explained how Big Data could help provide enhanced early warning, and real-time awareness and evaluation in the development and humanitarian sectors.

In one well-known study, researchers followed the position of 1.9 million active mobile phone SIM cards in Haiti around the time of the 2010 earthquake to track the displaced population. 

“If you see that a certain number of people have been displaced to a nearby town from an area where there was already cholera that’s probably where you should look for targeting your interventions next.”

Another application could be the monitoring of development targets. One-third of all countries still have difficulty reporting on up to half of the Millennium Development Goal indicators. Yet as the world considers new objectives for the post-2015 development agenda, the need for up-to-date information on the success of existing priorities is essential. 

“Being able to understand where people don’t show any awareness of how you contract HIV could be very useful for targeting programs,” Mr Kirkpatrick said, of social media, for example. Development work has long relied on surveys, the results of which soon become outdated. Mr Kirkpatrick likened this approach to taking a photograph, whereas Big Data now represents live-video. He believes the two are complementary. “No one makes a long-term plan based on a Twitter post.”

At the same time, Big Data allows novel ways to monitor a population’s characteristics in real time. For example, researchers from Belgium used anonymised data on how much airtime credit mobile phone users in Cote d'Ivoire purchased to estimate the relative income of individuals, and the diversity and inequality of income. This analysis paints a nuanced picture of shifts in the distribution of wealth. 

Big Data may also play a role in deciding which development priorities to set in the first place. People on social media in developing countries are already talking about what matters to them in topics such as education and healthcare. In a project done by Global Pulse, global twitter conversations are being filtered and visualized to reflect Post-2015 development topics of most interest across the globe. According to Mr Kirkpatrick, analysing this “messy, subjective, perceptual, human conversation” can be part of deciding what problems to target next.

Image

Image: UN Global Pulse

Global Pulse is not the only initiative experimenting with applications of Big Data.

Ulrich Mans and his colleagues at the Peace Informatics Lab at the University of Leiden’s Centre for Innovation are combining various data sets from the World Bank, UN Department of Economic and Social Affairs and elsewhere, to measure the effect of UN Security Council resolutions. Huge amounts of data from two years before and two years after the resolutions were signed are fed into a single database; using new search algorithms the team then identifies potential correlations and trends between seemingly unrelated aspects, from terrorism to climate change.

“It’s not about predictions,” Mr Mans said. “It’s about patterns you might not have thought of.”

Speaking recently at the European Peacebuilding Liaison Office in Brussels, Mr Mans said an NGO struggling to justify its funding could use Big Data to achieve the otherwise difficult task of showing how its actions had helped avert conflict.

He said the growing use of smartphones in developing countries, such as Somalia, and “data philanthropy”, where companies share certain data sets for public benefit, meant Big Data would become even more powerful in years to come. Those who invest in developing the appropriate skills will reap the rewards in the near-future as more and more data becomes available. 

Some corporations have entered into “data philanthropy” partnerships with UN Global Pulse to provide access to archival and real-time data for analysis – most recently, UK-based social media company DataSift which provides access to Twitter data.

Finally then, given the many applications of Big Data, what Global Pulse project is most exciting for Robert Kirkpatrick at the moment?

He summarised a project currently in development in which Global Pulse researchers are attempting to develop a technology that can capture and analyse keywords and insights shared on talk-radio by people in remote areas in the developing world. 

“We see an opportunity to build tools that can transcribe public conversations from radio, into a text transcript that can then be analysed so you could see where people are talking about flooding, rising food prices, the symptoms of dengue fever,” Mr Kirkpatrick said. “This could provide whole new insights in areas where you don’t have a lot of internet penetration, where you don’t have a lot of social media usage. We could suddenly get text, and that would be very exciting.”

This collaborative piece was drafted with input from Dorota Panczyk-Piqueray, UN Global Pulse, and support from the capacity4dev.eu Coordination Team. Teaser image courtesy of Tom Tolkien.