Evaluation methodological approach

One of the team members collects the set of basic official documents such as:

• Programming documents (e.g. project fiche), and subsequent modifications if there are any.
• Ex ante evaluation.
• EC documents setting the policy framework in which the project/programme takes place (EC development and external relations policy, EU foreign policy, country strategy paper.
• Government strategy (PRSP).

The evaluation team reviews the logical framework as set up at the beginning of the project/programme cycle. In the absence of such a document, the project/programme manager has to construct one retrospectively. As far as necessary, the evaluation team identifies the points which need clarification and/or updating. Any clarification, updating or reconstruction is reported in a transparent way. 

The analysis of the project/programme logic covers:

• Context in which the project/programme has been launched, opportunities and constraints.
• Needs to be met, problems to be solved and challenges to be addressed.
• Justification of the fact that the needs, problems or challenges could not be addressed more effectively within another framework.
• Objectives.
• Nature of inputs and activities.

Of particular importance are the various levels of objectives and their translation into various levels of intended effects:

• Operational objectives expressed in terms of short-term results for direct beneficiaries and/or outputs (tangible products or services).
• Specific objective (project purpose) expressed in terms of sustainable benefit for the target group.
• Overall objectives expressed in terms of wider effects.

Once the analysis has been performed on the basis of official documents, the evaluation team starts interacting with key informants in the project/programme management and EC services. Comments on the project/programme logic are collected.

The scope of the evaluation includes all resources mobilised and activities implemented in the framework of the project/programme (central scope). 

In addition, the evaluation team delineates a larger perimeter (extended scope) including the main related actions like:

• Other EC policies, programmes or projects, plus EU policies.
• Partner country's strategy (PRSP), or sector policy or programme.
• Other donors' interventions.

An action is included in the perimeter as far as it reaches the same groups as the evaluated project/programme does.

The evaluation team consults all relevant management and monitoring documents/data bases so as to acquire a comprehensive knowledge of the project/programme covering:

• Full identification.
• Resources planned, committed, used.
• Progress of outputs.
• Names and addresses of potential informants.
• Ratings attributed through the "result-oriented monitoring" system.
• Availability of progress reports and evaluation reports.

The evaluation team establishes the list of questions on the following bases:

• Themes to be studied, as stated in the ToR.
• Logical framework.
• Reasoned coverage of the seven evaluation criteria

The following evaluation criteria correspond to the traditional practice of evaluating development aid, formalised by the OECD-DAC (the first five criteria), and to the specific EC requirements (the last two criteria).

Each question is commented on in line with the following points:

• Origin of the question and potential utility of the answer
• Clarification of the terms used
• Indicative methodological design (updated), foreseeable difficulties and feasibility problems if any.

This site presents a list of typical questions associated with evaluation criteria. 

Within a few weeks after the start of the work, and after a review of basic documents complemented by a few interviews, the evaluation team defines its overall approach.
This approach is presented in a meeting with the evaluation manager and the reference group members. Subjects to be discussed include:

• Logical framework.
• Evaluation questions, either from the ToR or proposed by the evaluation team.
• Provisional methodological design.
• Access to informants and to documents, and foreseeable difficulties.

The presentation is supported by a series of slides and by a commented list of evaluation questions. Where relevant, the meeting may be completed by an email consultation.

Specific guidance in the case of:

• Multi-country programme

In the case of a multi-country programme, the evaluation builds upon a number of country case studies which should be selected as soon as possible and preferably before the end of the inception stage. 

• Participatory evaluation

The evaluation team extends its initial interviews in order to understand the expectations of beneficiaries and other outside stakeholders.
A stakeholder analysis is performed and discussed in the inception meeting.

The evaluation team prepares an inception report which recalls and formalises all the steps already taken, including an updated list of questions in line with the comments received. 

Each question is further developed into:

• Indicators to be used for answering the question, and corresponding sources of information
• Strategy of analysis
• Sub-questions.

Indicators, sources of information and sub-questions remain provisional at this stage of the process. However, the inception report includes a detailed work plan for the next step. The report needs to be formally approved in order to move to the next step.