2. Tools & Methods
Why evaluators should embrace the use of geospatial data during Covid-19 (Coronavirus) and beyond - IEG - Hiroyuki Yokoi, Jos Vaessen & Joachim Vandercasteelen (Main Author)
Geospatial data encompass all information that is ‘geotagged’ to an exact geographical location on earth. This information can be remotely sensed from space—i.e. satellite imagery—but can also be collected from databases, surveys, project documents, and Monitoring & Evaluation (M&E) systems. The use of geospatial data on project variables has become an attractive solution to fill the void of field missions during the Covid-19 pandemic. Evaluators, however, were using geospatial data in evaluation even before travel was restricted. There is now an incredible opportunity for evaluators to use geospatial data more effectively and efficiently.
In this blog, the authors discuss the advantages as well as the drawbacks of the use of geospatial data in evaluation.
Participatory Evaluation - Theories & Methods for Remote Work
A very practical and inspiring guide written by Evaluation Learning Consulting (ELC) with detailed descriptions of the principles of participatory evaluation, the tools that can be used to work remotely and suggestions for the facilitation of the process.
The unexpected arrival of the Covid-19 virus at the start of this year has forced many sectors to adapt their activities and to plan around restrictions which strictly limit travel and social interaction. When data collection processes are remotely managed, there is often a greater reliance on technology to replace some of the quality checks which could otherwise be done in person. The risks to the quality of the data are therefore greater but well-designed forms which make use of the functionality available through ODK-based tools such as Kobo, ONA and SurveyCTO among others can provide reliable and accurate data and allow remotely monitored projects to proceed while minimising as much as possible the risks associated with travelling between different locations.
Building on CartONG guidance documentation on checklists for mobile data collection and designing forms with data quality in mind, this document provides details of the main approaches which can be taken, through metadata and multimedia capture, to ensure that data collected in a remote context is as accurate and reliable as possible.
Comparative analysis of remote data collection tools for call-based surveys - Cart-ONG
This benchmarking document presents an overview of the types of computerized support solutions available for call-based surveys for humanitarian and development NGOs having to adapt their field practices to the physical distancing constraints of the current Covid-19 pandemic.
It also includes guidance to help them select the type of tool, with different scenarios that organisations can relate to, and a detailed comparison of some of the categories of tools less known by the sector.
The tool comparison was done in August and September 2020 by staff members of CartONG, based on documentation available online and through the testing of the solutions.
MERL TechState of the Field1The Evolution of MERL Tech - Linda Raftre
If there ever was a compelling time to take a hard look at the lessons from five years of convenings of thousands of monitoring, evaluation, research, and learning technology (MERL Tech) innovators and entrepreneurs, it is now, in the midst of the historic global COVID-19 pandemic. The urgent need to know what is happening, where, and to what extent has never been more important. It can literally mean the difference between life and death.
This paper explores the peer-reviewed evidence base of “traditional” technology-enabled monitoring, evaluation, research, and learning (MERL Tech) in international development assistance from 2015 to 2019. The authors conducted a scoping review that searched seven databases, screened 3,054 reference titles and abstracts, coded 886 abstracts, and extracted and analyzed conclusions and recommendations from the full texts of 256 studies. The findings reveal the most frequently reported technologies, MERL activities, and the sub-sectors, and the geographies where those tech-enabled activities occur. Gaps in the evidence for specific technologies, MERL activities, and sectors are mapped. The data reveals which technologies are trusted more than others and reported barriers to effective MERL Tech implementation and areas that researchers suggest for further investigation. The results suggest that the evidence from peer-reviewed studies is not proportional to estimated MERL Tech activity, significant publication bias exists, and further knowledge synthesis of unindexed grey literature is needed to provide a more comprehensive and possibly accurate description of MERL Tech practice.
This paper probes trends in the use of big data by a community of early adopters working in monitoring, evaluation, research, and learning (MERL) in the development and humanitarian sectors. Qualitative analysis was conducted on data from MERL Tech conference records and key informant interviews. Findings indicate that MERL practitioners are in a fragmented, experimental phase, with use and application of big data varying widely, accompanied by shifting terminologies. We take an in-depth look at barriers to and enablers of use of big data within MERL, as well as benefits and drawbacks. Concerns about bias, privacy, and the potential for big data to magnify existing inequalities arose frequently. The research surfaced a need for more systematic and broader sharing of big data use cases and case studies in the development sector.
Emerging Technologies and Approaches in Monitoring, Evaluation, Research, and Learning for International Development ProgramsEmerging technology is making monitoring, evaluation, research and learning (MERL) more precise and enriching data. However, this evolution is so rapid that it can be difficult to stay informed about the field overall. This paper presents examples of emerging technology that are most often used in MERL for development programs, describes the pros and cons of their use, and discusses technology and ethics concerns that practitioners should keep in mind. The paper covers new types of data sources (application data, sensor data, and drones), new ways to store data (distributed ledger technology and the cloud), and new ways to analyze data (text analytics and supervised and unsupervised learning).
The world today is more connected, interdependent, and data-rich than at any time in human history. Yet we increasingly see populations divided into those who benefit from the policies, products, and services driven by advances in data science, and those who are left behind or actively harmed. At its best, the global development sector creates tangible improvements in people’s lives, with evaluation among the sector’s most critical tools for knowing what is and is not working. By taking advantage of new thinking emerging from the field of data science, development sector practitioners and professionals can make their toolkit even more powerful.
Accelerating Use of Remote Surveys During the Covid-19 Pandemic
As the world reels from the effects of Covid-19 and widespread lockdowns, we collectively lack some of the basic information needed to navigate this crisis. Just as we are struggling with limited testing that obscures our understanding of the scale of Covid-19, we also are unable to answer basic questions about the human impacts – health, financial, and social – of this new reality.
The Rockefeller Foundation is continually experimenting with ways to measure impact that are inclusive, responsive, and rigorous. One of the methods in their impact measurement toolkit is 60 Decibels’ Lean Data approach, which is optimized for nimbleness, flexibility, and remote deployment through the use of voice surveys over mobile phones.
Guide for Adopting Remote Monitoring Approaches During COVID-19 - USAID
In the current operating environment, USAID and implementing partners face new challenges in implementing activities, monitoring progress, collecting data, and tracking indicators. As they adapt their approaches, they work with implementing partners to find innovative, responsible, and safe ways to monitor and evaluate programming. Digital tools can support novel approaches to remote monitoring. Responsible use of digital tools also supports Operating Unit (OU) alignment with the Digital Data Collection mandate in the Agency’s new Digital Strategy. This guide aims to provide information for Agency staff and implementing partners on remote monitoring techniques and when they can be employed.
This guidance brief aims to provide support to national, local and civil society organizations on the frontlines of gender-based violence (GBV) prevention and response on how to set up remote monitoring and management of their interventions. As the COVID-19 pandemic reaches new corners of the globe, its impact is compounded on already overstretched humanitarian operations in crisis-affected communities. In some settings, humanitarian assistance is being blocked as borders close and entire communities are quarantined, leaving populations without access to basic services, including water, food, healthcare, and sanitation.
As pandemic-driven health, social, economic, and hunger crises deepen across the globe, it is increasingly clear that COVID-19 is widening systemic inequalities that have long affected women, girls, and other people who face discrimination because of race and migration status. These dynamics threaten decades of progress in realizing the rights and equalities that all people should enjoy, and that women have fought hard to claim. CARE has warned from the beginning that the pandemic would have a disproportionate impact on women and girls. But foresight is only as good as the action it enables. The efficacy of CARE's and others' COVID-19 responses depends on understanding how marginalized people are affected, in all their diversity, across contexts, and over time. Women's needs are routinely overlooked without deliberate efforts to fill persistent gender data gaps. So CARE sought the advice of experts: women themselves.
The findings reinforce the understanding that men and women prioritize, experience, and report on issues differently. The gaps these findings reveal illustrate the vital importance of listening to many voices, and giving diverse groups of women equal opportunity to influence people who make decisions about COVID-19 support. Only by examining these differences can we ensure that responses are designed to work effectively and reach people with the assistance they need most.
How to assess a situation when the area concerned is not only affected by COVID-19, but also heavily flooded? In April-May 2020, Concern Worldwide, Mercy Corps, and Practical Action, members of the Zurich Flood Resilience Alliance in Bangladesh, published a report on the impact of COVID-19 in flood vulnerable communities based on structured questionnaire surveys to understand the compound risks of the pandemic and the monsoon season. Since the initial analysis, Bangladesh has been devastated with one of the worst and longest-lasting flooding in recent years with at least a quarter of the country under water. Officials reported more than four million people affected and a million homes inundated. COVID-19 cases are still rising, leading to health and socio-economic challenges. Mercy Corps conducted three remote online discussions with the 15 UDMCs and the findings are presented in this report.
Remote Humanitarian Monitoring - Guidance Note - Humanitarian Advisory Group & Glow Consultants
The Work in the Age of COVID-19 Guidance Note series falls under the Partnerships and Practice Stream of the Humanitarian Horizons research program. Intended as short and practical documents for operational actors, the series includes humanitarian guidance notes on remote management, remote monitoring, remote facilitation and remote working.
This guidance note can be used to plan remote monitoring, raise awareness on best practices (it’s colourful and engaging!), and advocate that donors support remote monitoring requirements.
Much qualitative research typically relies on face-to-face interaction for data collection through interviews, focus groups and fieldwork. But there are myriad ways researchers and students can collect qualitative data online or gather textual data that already exists.
Participatory scenario planning in times of uncertainty: five key lessons - Tiina Pasanen
Trying to get to grips with uncertainty feels more important than ever. We can’t know or predict for certain all the possible mid- and long-term consequences Covid-19 will have globally and in our societies, organisations or (global development) programmes and projects. However, we can explore and prepare for alternative futures. For programmes, this means both reacting to what is happening at this moment but also preparing for different future scenarios.
Talking Circles are safe spaces where relationships are built, nurtured, reinforced, and sometimes healed; where norms and values are established; and where people connect intellectually, spiritually, and emotionally with other members of the Circle. The Circle can also be an evaluation method that increases voice, decreases invisibility, and does not privilege one worldview or version of reality over another. The purpose of this article is to describe how the Circle can be a culturally responsive evaluation practice for those evaluators wishing to build relationships, share power, elicit stakeholder voice, solve problems, and increase participants’ capacity for program design, implementation, and evaluation. Circles can be used by both Indigenous and non-Indigenous evaluators. By offering the global evaluation community this concrete, practical, and culturally responsive approach, we open the door so that others can build on this work and offer additional insights as this practice is used, refined, and documented.
The restrictions on personal movement ordered globally to fight virus spread have required rethinking of certain methods, as the travel crucial to development and evaluation work cannot be undertaken. In response to the current extraordinary circumstances, the IOE has been exploring ways to fully leverage the content of existing documentation and the power of new technologies, to resume travel, field visits and shorter missions only when safety conditions allow. These alternative options are compared, to facilitate tailoring the approach taken to a range of different situations.
Using big data for evaluating development outcomes: lessons for evaluation during COVID
Significant data gaps remain in monitoring and evaluating development outcomes. Big data—that is digitally generated, passively produced and automatically collected—offer great potential for answering some of these data needs. The use of big data in evaluation has become ever more relevant after the COVID-19 pandemic outbreak, which has severely limited the researchers’ opportunity to collect data in the field with traditional methods. The authors present a systematic map highlighting how big data are being innovatively used in measuring and evaluating development outcomes. They also discuss the risks, biases and ethical challenges in using big data. The presentation offers an opportunity to discuss what tools and technologies are available to conduct evaluations in the time of COVID.
In times of a global crisis -such as the COVID-19 pandemics- everything moves faster, and evaluation tools require adaptation to this new situation. This tool has been developed for use in humanitarian crises and “is relevant in both rapid onset crises and protracted crises.” We publish it as a further contribution to the reflection on how the traditional evaluation instruments used in development cooperation can be adapted to face global or large-scale crises.
There has been a huge constraint in collecting data due to Covid-19 pandemic: depending on countries situation, data on food production, prices, trade, market access, nutrition and data on population without access/constrained access to food items are all affected. This page shows an online discussion on questions, such as: how can we improve the quality of the available data? How can we enhance the analysis and interpretation of data? How can we help coordinate the efforts by different players on data and analysis during the Covid-19 pandemic to ensure meaningful analysis?
Covid-19 crisis: how to adapt your data collection for monitoring and accountability? - CARTONG
While humanitarian and development programs need to adapt to the epidemic by integrating the constraints of social distancing (limiting the frequency, proximity and quantity of “face-to-face” activities), the same applies to Monitoring and Evaluation (M&E) and accountability mechanisms.
This note is intended for headquarters and field teams in charge of M&E and aims to help them adapt by summarizing the possible technological alternatives concerning data collection mechanism, ways information is exchanged with communities, and related data protection issue.
Adapting evaluation in the time of COVID-19 — Part 2: DEFINE - Better Evaluations, Patricia Rogers
The second of the Adapting Evaluation in the time of COVID-19 blog series focuses on defining what is to be evaluated. To do this, the authors have brought together some advice and resources on how to more effectively develop, represent and use theories of change in the current context – whether you are reviewing the theory of change for an existing initiative or developing a theory of change for a new initiative.
A guide containing examples of supportive evaluation activities for organisations and leaders managing COVID-19 response efforts.
Lupton, D. (editor) (2020) Doing fieldwork in a pandemic (crowd-sourced document)
Social research has been conducted online for many years, of course. There are many examples of using online survey tools or doing content analyses or ethnographies using existing online interactions as research materials. Interviews have been conducted by phone or Skype for a long time. This document was initially directed at ways for how to turn fieldwork that was initially planned as using face-to-face methods into a more ‘hands-off’ mode. However, people have added useful material about ‘born digital’ research (content already generated on the internet by online interactions), which provides an alternative source of social research materials if researchers decide to go down that path.
Remote Survey Toolkit Prepared in Response to COVID-19 - Ashley Speyer
For the past six years, the authors have been investing heavily in phone-based surveys to gather high-quality feedback and social performance data. Their belief has been that, when done correctly, phone-based phone surveys can get high-quality customer data at a fraction of the cost of traditional in-person surveys. With nearly all face-to-face research being temporarily suspended, and many of the organizations conducting research shifting some or all of their work to phone-based surveys, the authors want to help make this transition as seamless as possible and have put together this free guide to help.
Pocket tool Gender Responsive Evaluation and Research during COVID-19
UN Women issues a "pocket tool", which includes practical guidelines for gender-responsive evaluation management and data collection, structured around the four main evaluation phases: planning, preparation, conduct and reporting and follow-up. It contains tips and good practices to aid evaluation teams in determining the best course of action for the gender-responsive evaluation or research exercise in the context of COVID-19, while keeping stakeholders engaged and upholding relevant ethical and data protection protocols. It also provides numerous links to remote data collection methods and tools.
Resources for #eval in a time of crisis – Tom G. Archibald
A listing of various useful resources, from blog posts and articles to Facebook groups.
Using technologies for monitoring and evaluation in insecure settings – Better Evaluation
Operating in insecure environments is one of the more critical tests for humanitarian, development, and peacebuilding organizations alike. Access constraints or even direct attacks make monitoring and evaluation extremely challenging. Technologies like mobile phones, radios, Internet platforms and GPS trackers promise new solutions for collecting vital data or tracking implementation of projects.
Digital data collection during #covid-19; Tips for conducting remote M&E – IEG Worldbank
A blog post by the Independent Evaluation Group of the World Bank on digital data collection.
A quick primer on running online events and meetings – Emma Smith @BetterEvaluation
Practical guidance on how to run online events and meetings.
Best practices for conducting phone surveys
This is a living document that aggregates crowd-sourced tips and factors to consider when conducting remote surveys (in particular phone surveys) while practising social distancing. We will be updating this page regularly.
Lessons for COVID-19 from GEF IEO Evaluations - Independent Evaluation Office
Series of blogs on the use of remote techniques making use of geospatial to measure outcomes and long-term sustainability of environmental interventions by the Global Environment Facility.
Evaluating the Impact of Environmental Interventions during a Global Quarantine - Geeta Batra
In the era of COVID-19 where fieldwork is halted and working from home is the new normal, methods like Quasi-experimental Geospatial Interpolation (QGI) can help evaluators leverage open source and readily available data to determine the impact of projects. Working with geospatial data can help project evaluators maintain flexibility, and encourages them to move away from designing new and often expensive data tools for every unique impact evaluation. The method provides a novel, replicable approach to the estimation of environmental and socioeconomic outcomes.
Bowling in the Dark: Monitoring and evaluation during COVID-19 (Coronavirus) - World Bank Group
Mindful of the need to conduct quality evaluations, especially in times of crisis, the World Bank Group shares lessons on remote technology and the importance of communication.
Evaluation During a Crisis: Covid19 - UNDP
UNDP shares an infographic with suggested responses in evaluation to the crisis. How to rethink plans and prioritize? How to make evaluations more useful?
Disasterready.org features the largest online learning library for humanitarian and development professionals. It proposes free training content developed by a great variety of organisations, including on Monitoring & Evaluation in humanitarian crises and COVID-19. The site requires registration to consult its resources.
Related documents
Participatory Evaluation - Theories & Methods for Remote Work
English (915.71 KB - PDF)