Evaluation methodological approach

• The evaluation mandate is defined. It specifies the intervention, the geographical area, and the sector or theme to be evaluated.
• An evaluation manager is appointed within the concerned service, together with a deputy manager. The manager conducts the process from beginning to end on the Commission's behalf and under the responsibility of the hierarchy. He/she plays a key role in preparing the evaluation, engaging and supervising the external evaluation team, as well as in the dissemination process and the follow-up of recommendations. He/she is the chairperson of the reference group.

Specific situations(s)

Country level evaluation

Country strategies follow multi-annual cycles. The decision to evaluate determines the time of the evaluation in the cycle. The evaluation covers the ongoing cycle and possibly the previous cycles. Depending on the position of the evaluation in the cycle, feedback possibilities will vary. For instance, a mid-term evaluation can be used for adjusting the implementation, an evaluation at the end of the cycle may help preparing the next cycle, and an ex-post evaluation allows for drawing transferable lessons or for accountability purposes.

Evaluation conducted by the evaluation unit

The Unit's working programme is adopted by the hierarchy and published on the Commission's site.

The evaluations conducted by the Unit fall within the evaluation services multi-annual framework contract. The concerned contracting party is notified in advance of the approximate launch date for the works. 

Evaluation conducted by another service

If the service does not have staff members with enough evaluation experience, support for quality assurance of the evaluation is sought.

• The evaluation manager reads the basic documents on the actions to be evaluated, hereafter referred to as "the intervention".
• He/she identifies the global objectives as well as the sectors, themes and cross-cutting issues of particular importance in the context of the evaluation.
• He/she consults the Commission's data bases for an overview of the support and its financing.
• He/she identifies a few key informants within the Commission services and has informal talks with them to better understand the stakes of the evaluation.
• He/she specifies the central scope of the evaluation as well as the themes and cross-cutting issues to be examined as a priority.

Specific situations(s)

Country level evaluation

The documents consulted at this point are the following:

• Currently applicable country strategy document and strategy documents applying to previous periods.
• Documents pursuant to the Commission's strategy in the region.
• Documents relating to the Commission's strategy in the focal sectors.
• Communications, regulations and delegation's decisions
• Relevant documents on cross-cutting issues (e.g.: gender, environment, good governance, etc.)

• The manager identifies the services to be invited to join the reference group, with a view to meeting three goals: provision of both expertise and information, expression of the variety of relevant views within the Commission, and legitimacy of the evaluation process.
• A note is sent to the services invited to join in, explaining the role played by the reference group.
• Every service that has been invited to join in appoints a person who will be a group member.

Specific situations(s)

Country level evaluation

In the case of a country level evaluation, the delegation is part of the reference group and is completely involved in the exercise. 

At this point, the manager agrees on the working modalities insofar as the delegation is associated at a distance. 

The manager invites a person from the country's embassy in Brussels to the reference group. 

The head of the service managing the evaluation sends a note to the head of the concerned delegation asking him / her to appoint one or several representatives as reference group members. 

• On the basis of the basic information gathered, the manager specifies both the expertise required and the schedule.
• He/she draws up a first version of the terms of reference.
• The manager sends the first version of the terms of reference to the reference group members for their comments.
• After these consultations, the manager drafts a new version of the terms of reference taking into account the opinions received.
• He/she launches the process of engaging the external evaluation team as per the terms of reference.

Specific situations(s)

Country level evaluation

Country level evaluations often deal with recurring evaluation questions, such as:

• To what extent does the Commission's health strategy meet the basic needs of the population, in particular those of the poorest?
• To what extent do European interventions contribute to the strengthening of regional and global cooperation processes and economic integration?
• To what extent is the good governance cross-cutting objective built into all of the interventions?

The manager can refer to such typical questions (see other examples). It is also possible to be inspired by thematic or sector evaluations published on this site. 

The manager points out the themes to be covered and leaves it to the evaluation team to propose questions. 

Global sector or thematic evaluation

When preparing the terms of reference, the manager may be unable to point out which countries will be subject to a mission. In such case, the manager may ask for a budget proposal that is limited to the desk phase. 

Evaluation conducted by the evaluation unit

The manager sends the terms of reference to the contracting party responsible for the multi-annual service contract under which the evaluation takes place. He/she requests a technical and financial proposal (launch note)

On the basis of the terms of reference and his/her own expertise, the author of the proposal formulates basic assumptions on:

• Areas requiring specific expertise.
• Possibility to mobilise consultants with the right profile in the country or countries involved.
• Number, nature and probable difficulty of the questions.
• Existence, quality and accessibility of data on aid implementation.
• Existence of previous evaluations which may be reused.

The method is broadly defined keeping the constraints set by the terms of references into account. The author of the proposal defines the main tasks and draws up a first outline of the field data collection tools to be implemented. 

The tasks are temporarily divided among:

• consultants from partner country or countries and international consultants
• senior, medium, junior consultants
• experts in the sector(s) of the intervention and experts in evaluation methods

The core evaluation team members are identified and absence of conflict of interest is verified. 

Both the budget and the schedule are specified within the framework of constraints set by the terms of reference.

The contents of the proposal are as follows:

• Understanding of the context in which the evaluation is being launched, of its purpose (intended users and intended use) and of its scope
• Understanding of the main areas to be covered by the evaluation questions and process to elaborate and select the questions (unless the questions are specified in the terms of reference).
• Main methodological options.
• Understanding of the evaluation process and of the respective roles played by the various actors.
• Time schedule.
• Core evaluation team members, their field of expertise and their role.
• Management modalities.
• Budget in the standard format EN
• Team members' CVs in the standard format EN and declaration EN of absence of conflict of interest (annex).
• CV of an expert from outside the evaluation team who will be in charge of quality control.

• The evaluation manager receives the technical and financial proposal(s) that must include the following elements:
  • Understanding of terms of reference.
  • Core team members' names and CVs.
  • Planned schedule and justification of any differences from the schedule in the terms of reference.
  • Indicative budget (including final report discussion seminar or dissemination seminar) according to the applicable template.
• For each proposal, the manager makes sure that the team has adequate knowledge and experience in the following fields: evaluation methods, concerned region or country, concerned sectors and cross-cutting issues, the functioning of European foreign aid.
• He/she verifies that both the human and the financial resources offered are compatible with the allocated budget and are suitable for the particular difficulties that might have been identified while preparing the terms of reference.
• The manager then engages the external evaluation team in line with the applicable procedure and after consultation with the reference group, members, in particular on the risks of conflict of interest if a candidate for the evaluation has had any responsibility in the intervention under evaluation.
• The team's engagement is then confirmed by an official letter signed by the head of unit, the date of which marks the formal beginning of the evaluation team's work.

Specific situations(s)

Evaluation conducted by the evaluation unit

In case the works are conducted by the Evaluation Unit, the engagement of an external evaluation team falls within a multiannual service framework contract. The process is as follows:

• The evaluation manager receives the first draft of the technical and financial proposal from the contracting party (launch note) and makes sure that it is both complete and of good quality.
• He/she sends the proposal to the reference group members including a request for comments, in particular on the risks of conflicts of interest.
• He/she collects the opinions received and specifies his or her requests for modification.
• He/she receives the proposal as modified by the contracting party and approves it.
• Approval is then confirmed by a letter signed by the head of Unit.

Country level evaluation

The delegation and other reference group members are consulted on the technical and financial proposal(s) (launch note). 

The manager makes sure that the proposal(s) provide(s) for the required resources for a pilot visit and a seminar in the partner country. 

Global sector or thematic evaluation

In the case of a sector or thematic evaluation, particular attention must be drawn to the delineation of the evaluation's core scope as well as to making a distinction between sector and theme.