Purposes of Monitoring and Evaluation for EU-funded interventions: Main findings

Results Oriented Monitoring (ROM) reviews and evaluations are the two external assessment tools for Operational Managers in EU Delegations and Headquarters to assess intervention’s performance. Following the standardised methodology described in the ROM Handbook, ROM reviews provide an assessment based on eight monitoring criteria: four of the OECD-DAC criteria, i.e. relevance, efficiency, effectiveness, and sustainability, plus four additional criteria of coordination, complementarity and EU added value; intervention logic, monitoring and learning; cross-cutting issues; and communication and visibility. 

Evaluation is defined in the EU Better Regulation guidelines as the assessment of the effectiveness, efficiency, coherence, relevance and EU-added value of one single EU intervention: “An evidence-based judgement of the extent to which an existing intervention is: (i) effective in fulfilling expectations and meeting its objectives; (ii) efficient in terms of cost-effectiveness and proportionality of actual costs to benefits; (iii) relevant to current and emerging needs; (iv) coherent both internally and externally with other EU interventions or international agreements, and; (v) it has EU added value i.e. produces results beyond what would have been achieved by Member States acting alone.” The Evaluation Handbook is aimed at guiding the preparation, launch, and management of evaluations.

To compare the current use of both external assessment systems, DG INTPA’s Unit D4 commissioned an analysis in the framework of the meldea contract “Support to programming, monitoring, evaluation and knowledge sharing in the EU external action.” The analysis compares the purposes of ROM reviews and of evaluations, the coherence of the respective methodological approaches and, ultimately, the overlaps and complementarities between these different types of assessments.

1. In ROM reviews, the purpose of the exercise is usually declared explicitly, resulting in strong consistency between the exercise's purpose and the type of recommendations issued. In evaluations, however, the broader variety of purposes makes this consistency harder to assess.

The purpose of evaluations is generally broader than in ROM reviews: they should serve accountability, decision-making, learning, and management purposes for a wider group of actors. As a result, evaluation recommendations are often more general, whereas, in the case of ROM reviews, recommendations are concrete, frequently focused on implementation issues and addressed to the stakeholder that should implement them. 

2. ROM reviews follow a well-established approach with a formalised process. In contrast, evaluations are less standardised, requiring greater involvement from evaluation managers and continuous learning and adaptation by all actors involved.

In ROM reviews, the product and purpose are well-identified. Evaluations gather, analyse and use more extensive data, making the process less clearly defined from the outset. This implies clarifying the process and evaluation approach during the inception phase and increases the need for both Evaluation Managers and evaluators to learn and adapt.

3. Evaluation provides a more extensive and deeper study of all criteria except for the cross-cutting, and communication & visibility areas which appear to be more comprehensively addressed in ROM reviews.

Areas of context and change dynamics are deeply analysed in evaluations. Other areas, such as efficiency, effectiveness, impact and sustainability of the interventions, are assessed in both systems (i.e. ROM and evaluations) but at greater length in the case of evaluations. Conversely, cross-cutting aspects, communication & visibility are often treated more comprehensively in ROM review reports.

Conclusion

The highly standardised system of ROM reviews provides recommendations that are fully aligned with the purposes while requiring limited time investment from the Operations Managers, whereas evaluations provide more and deeper information but require a higher level of investment and knowledge throughout the process from all the actors. ROM reviews and evaluations are complementary tools. There is no evidence that recent ROMs and interim evaluations for the same interventions are occurring too closely together to suggest duplication. On the other hand, many final evaluations are preceded by a ROM, which enhances coherence by allowing evaluations to build on ROM findings.

Share your views:

Have you benefitted from both systems? What is your experience? Your opinion will enrich the picture. Share it in the comments below or start a discussion in the ROM (Results Oriented Monitoring) Public Group on Capacity4dev.

______

Photo copyright: Bertha Wangari, European Union 2020 | Source: EC – Audiovisual Service

DISCLAIMER: The information provided is based on the content of the document “IDENTIFICATION OF THE PURPOSES OF MONITORING AND EVALUATION AS WELL AS COHERENCE OF THEIR METHODOLOGICAL APPROACHES AND OBJECTIVES” and should not be interpreted as the official view of the European Commission. This article was produced in collaboration with the expert Colm McClements, author of the analysis.

These findings are drawn from a document produced for the Directorate-General for International Partnerships (DG-INTPA)– with support from the INTPA.D4-managed “Support to programming, monitoring, evaluation and knowledge sharing in the EU external action” (meldea). It is available to staff of DG INTPA, DG NEAR and FPI.