#EvalCrisis Blog - 10 - Growing as a field

#EVALCRISIS BLOG - A DEVCO/ESS INITIATIVE

Growing as a field

Using data to its full potential to evaluate in times of COVID-19

NINO CHOKHELI, MARIA MONTENEGRO, SUSAN RACHEL JOSE, UMA MAHESWARA RAO, SARAH OSMANE & NICOLE ALMEIDA

Foreword by ESS

We are particularly happy to host in our EvalCrisis blog series a post prepared by the team FIX IT that scored second at the 2020 IPDET / EvalYouth Hackaton.  You can read more about the Hackaton in our previous blogpost 7 here and listen in one of our previous podcasts the interview to the main organisers of the event, Stefanie Krapp and Antonina Rishko-Porcescu. We have asked the team FIX IT to describe the challenges they wanted to contribute to address with their work and to share a short presentation of EvalConnector, the result of their Hackaton efforts.

The members of the team FIX IT were Nino Chokheli (Georgia), Maria Montenegro (Ecuador/Canada), Susan Rachel Jose (Philippines), Uma Maheswara Rao (India), Sarah Osmane (Lebanon) and Nicole Almeida (India).

We wish you good reading!

The ESS EvalCrisis team

Introduction

Covid-19 changed our lives from everyday social interaction to economic instability. The lockdowns reduced mobility and hindered many public services by state and non-state actors. In some countries, disruptions to the timely provision of goods and services have threatened communities’ survival. As we continue to face challenges, the utmost responsibility of every individual and organisation is to take safety measures to reduce the spread of the virus.

These challenges are not foreign to the evaluation field. Our organizations have to ensure the safety of their workforce, partners, stakeholders, and beneficiaries. Resources are scarcer, research activities and traditional data collection methods have been disrupted, and programs have been interrupted. Yet the need for evaluating policies and programs is more critical than ever. We face several fieldwork challenges, including accurate assessment of program impacts in light of COVID-19, collecting data while respecting measures to reduce the spread of the virus, and maintaining quality assurance practices. As our field adapts to face these challenges, the coordination between NGOs, community-based grassroots organisations, and state actors has never been so important.

Adapting to the new normal

International organisations have been key players in guiding the way in the crisis. Organizations like the World Food Program published guidelines to support evaluators as they adapt, modify, or cancel their evaluation projects amid the pandemic. Knowledge sharing was crucial in the pandemic’s early days as evaluators adapted their work to the lockdowns and social distancing guidelines. The UNODC and many other organizations published a guidance report describing the limitations, requirements, and risk mitigation strategies of remote data collection methods. Innovation also played a key role in addressing challenges faced by traditional data collection methods. Evaluators drew from other disciplines, such as Geospatial science.

Continued challenges

Despite these efforts, evaluation projects across the world continue to face challenges. While some field activities, stakeholder meetings, and capacity building events have shifted to virtual platforms, not all evaluation activities can be remote. This is especially the case for evaluation activities involving remote communities with limited access to the internet or those involving vulnerable populations. For instance, the anonymity of participation may be crucial to ensure the safety of domestic abuse survivors receiving support services. In addition, appropriate response rates and participant engagement when conducting data collection remotely are challenging as individuals face ‘zoom fatigue’, have caring responsibilities, health concerns, and economic issues. The pandemic’s impact on programs has also forced some organizations to reallocate funding away from evaluation activities. As a result, many evaluation activities have been canceled or postponed, leaving decision-makers without much needed evidence-based information to support their work. 

Growing Importance of Data-sharing and Open Source

While the pandemic has posed several challenges, it has also led to opportunities for information sharing. Governments are investing in their digital infrastructure to improve data sharing with researchers, health officers, and the public. Dashboards have been widely adopted to share real-time statistics related to the pandemic (See US Centre for Disease Control and Prevention as an example). Non-governmental and research institutions, such as the John Hopkins University, have also developed popular dashboards that have been crucial in informing the world about the pandemic. There has also been noticeable interest in crowdsourcing and other types of knowledge-sharing as researchers and institutions continue to try to understand the virus and its impacts. Crowdsourcing has been used to solve various COVID-19 related problems including understanding the impact of the virus on cancer patients^[1], building low-cost ventilators^[2] and developing knowledge on how to build your own equipment to treat COVID-19.

What can we as evaluators learn from these innovations?  First, we can recognize that collaboration and data sharing are more important than ever in light of the challenges imposed by COVID-19 in our traditional data collection activities. Second, we can acknowledge how the digital transformation we are experiencing has facilitated knowledge-sharing across institutions and international boundaries.

There are already open data sources that can support the work evaluators do in the pandemic-era. A notable example is the Common Operational Dataset (COD) for Emergencies, a tool developed by Australian Aid, UNFPA and UNOCHA aid organizations in the Philippines who need baseline data in preparation for response and other humanitarian contexts. The tool is available offline; there is no need for an internet connection nor a data specialist to extract, clean, edit and prepare demographic data at national and sub-national levels. The demographic data includes estimates for select indicators in fertility, maternal and newborn health, contraceptive use and gender-based violence. Evaluators working in health projects in the area can turn to this source of data for their baseline measures. By doing so, evaluators can reallocate their efforts for planning and conducting remote data collection to better understand the project and how it has been impacted by the pandemic, ensuring the evaluation findings are relevant to the crisis we are facing. 

EvalConnector

EvalConnector was born during the IPDET Evaluation Hackathon 2020 as a solution to the challenges evaluators are facing during COVID-19. Drawing from the examples of innovative knowledge-sharing initiatives across the world, we developed an idea that would allow evaluators to access existing data to support their projects, reducing the need for conducting data collection in remote areas where access to the internet is limited.

EvalConnector is a coordination platform and information management tool that reduces overlaps in data collection among different stakeholders and organizations. EvalConnector provides the means to evaluators to find local partners, to get baseline data of similar interventions, coordinate data collection, and avoid duplication of efforts. The collaboration and coordination using this platform would reduce the need for in-person data collection, thus reducing the spread of the virus. It is a platform that would serve all stakeholders involved in evaluations, such as the beneficiaries, donors, the implementing organisation, universities, etc.

Figure 1: Objectives of the EvalConnector

Moreover, EvalConnector is intended to build trust among state and non-state actors, communities, stakeholders, beneficiaries, and to create a safe environment based on professionalism, transparency, and respect. Open source and data sharing will maximize the efficiency in the COVID-19-era by supporting evaluation and research projects and ensuring evidence-based recommendations are timely and relevant.

What makes our solution different?

EvalConnector would not just be an online platform that connects evaluators and local organisations for data purposes. It would also ensure that these partnerships would be fair by ensuring that all users adhere to a code of conduct. Fair partnerships are a cornerstone of the project. Resources are scarce during COVID19 but will be more scarce while rebuilding the world. Thus, having easily accessible data through platforms like EvalConnector would be integral to make sure that resources are not wasted on duplicated data collection efforts. EvalConnector would systematize the data based on region and sector of interest, such as education, food, health, etc. The goal is to promote fair and smart collaborations. To learn more about our project, click here.

Figure 2: EvalConnector screen

What’s next?

While our idea is in its initial stage, IPDET has kindly offered us an opportunity to discuss it with an incubation expert from the World Bank Group to move our idea forward. We are hoping to develop our idea further in the upcoming months. In the meantime, we want to encourage all evaluators to think boldly about other ways in which we can collaborate, share knowledge, and become more efficient to meet the growing demand for our work.

Disclaimer: Views expressed here are those of the authors and do not necessarily reflect the opinion or position of DG INTPA.

^[1] https://www.nature.com/articles/s43018-020-0065-z

^[2] https://helpfulengineering.org/

Nino Chokheli - from Georgia, Tbilisi. MA in Sociology, currently working as a Monitoring, Evaluation, Accountability and Learning Manager for People in Need Myanmar.  

Email

Nicole Almeida - from India, recently graduated with a MA in Public Policy. Currently working as a Researcher at Mission Measurement, Chicago.

Maria Montenegro - from Ecuador, living in Canada. MSc. in Agricultural and Resource Economics, currently working as an Evaluation Specialist for Fraser Health Authority.

Email

Susan Rachel Jose -from Manila, Philippines, engineer  and economist  working as independent development specialist consultant with the Asian Development Bank and The Asia Foundation

Email

Sarah Osmane - PhD. Evaluation Analysis and Learning Coordinator at the Norwegian Refugee Council, from Lebanon