Analysis of the sources of information of ROM monitoring questions: Main findings

The Results Oriented Monitoring (ROM) system is the primary external monitoring tool for Operational Managers in EU Delegations and Headquarters to assess intervention performance. Following the standardised methodology described in the ROM Handbook, ROM reviews provide an assessment based on eight monitoring criteria: four of the OECD-DAC criteria, i.e. relevance, efficiency, effectiveness, and sustainability, as well as four additional criteria which are coordination, complementarity and EU added value; intervention logic, monitoring and learning; cross-cutting issues; and communication and visibility. For each ROM review, external monitoring experts comprehensively answer a total of 30 questions against the eight criteria.

To determine the main sources of information for each question, DG INTPA’s Unit D4 commissioned an analysis within the framework of the “Support to programming, monitoring, evaluation and knowledge sharing in the EU external action” (meldea) contract. The analysis: 

• looks at the key challenges faced by experts when researching and reporting the monitoring questions.
• provides information about the questions that are more difficult to analyse and potential improvement in the data collection or the triangulation of information. 
• is based on data available in EU systems and complemented by a survey collecting information from ROM contractors.

1. Most monitoring questions can be answered in a detailed and comprehensive manner within the framework of a ROM mission.

As a main conclusion of this analysis, the sources of information exist and are available within the ROM mission for a large majority of questions. The wording of these questions is clear, and the requested information can be accessed. ROM experts have the capacity to collect and triangulate data in a robust way to report the questions comprehensively.

2. Some of the monitoring questions require:
   • a type and quantity of data that can be difficult to collect within the frame of a ROM Review;
   • more clarity in the wording of the questions;
   • additional support to collect robust information.

Three other cases are highlighted in the analysis. For some of the questions the sources of information exist, but the ROM Experts do not have the capacity to collect, develop or triangulate them adequately within the timeframe of a ROM review. The scope of these questions requires a deeper analysis than the one allowed by the time and resources available. Six questions are included in this category, they are mainly related to the effectiveness and sustainability of the interventions. 

The second case includes questions where the wording gives room for interpretation. This could lead to more generic or vague answers depending on the available information.

Thirdly, for some questions, the ROM expert needs the support of other key stakeholders since the information is not usually found in intervention documents/interviews. Answering them requires access to decision-makers or institutional representatives with wider political and strategic vision. The involvement of Delegation staff is very useful in this regard. 

Conclusion

The analysis and reporting of the 30 monitoring questions in the framework of the ROM mission is a process that depends not only on the existence of reliable and detailed sources of information but also on the scope and strategic focus of the questions themselves. In addition, it relies heavily on the coordination and cooperation of relevant stakeholders to facilitate access to these sources. 

Share your views!

Did you participate in a ROM review and found the process difficult due to lack of data or information? Your opinion would enrich the picture. What is your experience? Share it in the comments below or start a discussion in the ROM (Results Oriented Monitoring) Public Group on Capacity4dev.

______

Photo copyright: Harandane Dicko, UNICEF 2019 | Source: EC – Audiovisual Service

DISCLAIMER: The information provided is based on the content of the document “ANALYSIS OF THE SOURCES OF INFORMATION OF ROM MONITORING QUESTIONS” and should not be interpreted as the official view of the European Commission. This article was produced in collaboration with the expert Carlos Madridejos, author of the analysis.

These findings are drawn from a document produced for the Directorate-General for International Partnerships (DG-INTPA)– with support from the INTPA.D4-managed “Support to programming, monitoring, evaluation and knowledge sharing in the EU external action” (meldea). It is available to staff of DG INTPA, DG NEAR and FPI.