Evaluation methodological approach

.

What does this mean?

The three key players are the evaluation manager, the reference group and the external evaluation team. The distribution of roles is specific to each phase and each stage of the evaluation.

All the actors in the evaluation process have to know what their role and responsibilities are, for example: Who is responsible for the recommendations? Who has to assess the quality of the report?

What is the purpose?

• To mobilise the different actors, depending on their specific competencies and resources.
• To avoid deadlocks due to possible confusion regarding roles.
• To preserve the impartiality of the evaluation and avoid biased judgements.

Who does what?

The three key players in an evaluation are the evaluation manager, the reference group and the external evaluation team. Their roles differ, depending on the following phases and steps:

- Decision to evaluate

The decision is taken by the authorities responsible for the intervention and is reported in a document such as:

• the annual evaluation programme
• the document scheduling the intervention.

The decision refers to an evaluation framework that defines the rules of the game. If necessary, it completes this framework.

- Preparation (phase 0)

Constitution of the reference group:

• by the evaluation manager.

Drafting of the Terms of Reference:

• by the evaluation manager.
• after consulting the reference group.

Selection or approval of the external evaluation team:

• by the evaluation manager,
• after consulting the reference group.

- Desk (1)

Clarification of the intervention logic:

• by the external evaluation team,
• after interaction with the evaluation manager and the reference group.

Choice and drafting of evaluation questions:

• by the evaluation team,
• validation by the reference group,
• arbitration by the evaluation manager if necessary.

Definition of the judgement criteria (also called reasoned assessment):

• by the external evaluation team,
• approval by the reference group,
• arbitration by the evaluation manager if necessary.

Choice of indicators:

• by the external evaluation team,
• after consulting the reference group.

Design of the method (data sources, tools and work programme):

• by the external evaluation team,
• after consulting the reference group.

Collecting available data at the head office of the Commission and/or the Delegation (s):

• by the external evaluation team,
• if necessary, with the aid of members of the reference group for access to informants and in interaction with the evaluation manager.

- Field (2)

Data collection:

• by the external evaluation team,
• if necessary, with the aid of members of the reference group for access to informants, and in interaction with the evaluation manager in cases of serious problems.

- Synthesis (3)

Report, including the conclusions and recommendations:

• by the evaluation team,
• after interaction with the reference group.

Verification of the quality of the report:

• by the evaluation manager, possibly with an external support,
• after consulting members of the reference group.

- Dissemination and follow up of recommendations (4)

Internal and external dissemination:

• by the evaluation manager

Use of the conclusions and recommendations:

• by the Commission's services concerned
• with follow up by the evaluation manager.

.

.

Who is this?

The evaluation manager is a member of the service in charge of the evaluation. He/she manages the entire evaluation process, from A to Z on behalf of the commissioning body.

Why appoint an evaluation manager?

• To ensure consistency throughout the evaluation process, from the terms of reference to the dissemination of the report and the follow-up of recommendations.
• To be the contact person for administrative issues and to coordinate the activity of the different actors (reference group and evaluation team).
• To organise, supervise and assess the quality of the different phases of the work.
• To ensure the smooth running of the evaluation.

What does he / she do?

• He / she forms the reference group after consulting the heads of the services concerned.
• He / she drafts the terms of reference after consulting the reference group members.
• He / she engages the external evaluation team after consulting the reference group.
• He / she helps the evaluation team to get access to information and informants.
• He / she organises the discussion and approval of conclusions and recommendations in interaction with the reference group and the evaluation team.
• He / she performs a methodological quality assessment of the reports, double checked by a second person.
• He / she disseminates the evaluation to the different actors concerned.
• He / she monitors whether the recommendations are taken into account in interaction with the authorities concerned.

Recommendations

• Appoint a deputy to help the evaluation manager on a daily basis and to stand in when he/she is absent.
• Appoint an evaluation manager who knows both his/her institution and the evaluation methods.
• Ensure that the evaluation manager is not too close to the managers of the evaluated intervention.

.

.

What is this?

A small group gathering the services concerned by the intervention and possibly other stakeholders. One or more experts may also be invited to participate.

The reference group is an interface between the evaluation manager and the external evaluation team. Its members help the external evaluation team to identify data sources and to access them. They validate the evaluation questions, and discuss the conclusions and recommendations.

The reference group allows the variety of points of view on the evaluated intervention to be expressed.

What is its role?

Reference group members:

• Discuss and comment on the terms of reference drawn up by the evaluation manager.
• Aggregate and summarise the views of the Commission services and act as an interface between the evaluation team and the services, thereby supplementing bilateral contacts.
• Ensure that the evaluation team has access to and consults all information sources and documentation on activities undertaken.
• Validate the evaluation questions.
• Discuss and comment on notes and reports produced by the evaluation team. Comments by individual members of the reference group are compiled by the evaluation manager and subsequently transmitted to the evaluation team.
• Assist in feedback of the findings, conclusions and recommendations from the evaluation.

Whom to invite and how?

Different types of actor can be invited to participate in the reference group. If the evaluation is managed at headquarters, membership includes:

• Services of the Commission concerned by the intervention, including the Delegation concerned in case of a country/region level evaluation.
• Specialists on a sector or a cross-cutting issue within the Commission.
• Embassy of the partner country in the case of a country level evaluation.

If the evaluation is managed in the partner country, membership may extend to:

• Government services.
• Selected development partners.
• Selected civil society organisations.
• Experts

Experience has shown that it is preferable for the group to be limited in size (10 to 12 members) if it is to function effectively.

After identifying the services, institutions or organisations to invite to the reference group, the head of the service managing the evaluation sends them an invitation to:

• Officially announce the launching of the evaluation
• Request that they appoint one person as a member of the reference group.

When should it be asked to meet and how should it be involved?

The reference group is generally involved in the following way:

Opening the evaluation process

In addition to inputs made by the reference group, the evaluation may benefit from a wider variety of views through other channels like:

• Consultation of key stakeholders by the evaluation team in the first stage of the process
• Gathering of participatory workshops, focus groups or expert panel(s) during the evaluation process
• Discussion of the draft evaluation report in a widely attended seminar in the partner country (when relevant).

What does this mean?

One or more members of the reference group may be invited as a resource person, for their expertise.

Why invite them?

• To orientate the evaluation work towards the acquisition of new knowledge and to learn from others' experience.
• To describe the point of view of certain stakeholders, where relevant.
• To contribute towards quality assessment of work in their field of expertise.
• To facilitate knowledge transfer.

Who should be invited?

• The experts invited may or may not belong to the services of the Commission.
• It is preferable for them to be familiar with the entire evaluation field.
• As part of the global network they can mobilise knowledge about the entire development aid community.

What to do?

• Prepare a precise agenda and ensure that you adhere to it: (see: inception meeting, first phase meeting, debriefing meeting, final discussion meeting)
• Ensure that the discussions are balanced, impartial and constructive.
• Draft clear and synthetic minutes that highlight the points debated and report on points of agreement (this task is preferably accomplished by the deputy manager).

What to do in case of disagreement?

In case of lasting disagreement on the evaluation method, the service in charge of the evaluation arbitrates. If necessary, the decision is taken by the head of the Unit.

.

.

What does this mean?

The external evaluation team is responsible for data collection and analyses and for the evaluation report, including the formulation of judgements (also called reasoned assessment) and the drafting of conclusions and recommendations. It interacts with the reference group and the evaluation manager. It provides evaluation services to the commissioning body under contract.

The members of the external evaluation team are independent of the organisations that participated in the design and implementation of the evaluated intervention. They belong to national or international research organisations or consultancy firms, or are experts working independently. Where possible, some members of the team are from the country or countries targeted by the evaluated intervention.

Why contract an external evaluation team?

• To ensure the independence, impartiality and credibility of the evaluation process.
• To provide expertise on evaluation methods and techniques.
• To provide expertise in the sector(s) concerned.

How to select the team?

The external evaluation team is selected on the basis of proposals via different procedures, e.g. pluri-annual evaluation service contract or specific call for tenders.

The evaluation manager engages the team after consulting members of the reference group.

The following criteria are used as guidelines:

• Relevant experience in evaluation.
• Knowledge of the evaluated development aid modalities.
• Knowledge of the evaluation scope (e.g. sector, country).
• Presence of local experts and consultants.
• Team leader's managerial skills (organisation, coordination, dialogue with the client, etc.).
• Team's independence as regards the evaluated intervention and absence of conflicting interests.
• Evaluation skills and particularly the ability to formulate judgements, to draw up conclusions and recommendations and to draft syntheses.
• Mastery of data collection and analysis tools.
• Diversity of team members' profiles and expertise.
• Price.

The evaluation manager sets the relative weight of criteria and specifies it in the terms of reference.

Recommendations

• Involve local consultants, as far as possible, to promote the development of local capacity (development of national evaluation expertise) and benefit from their close knowledge of the field.

What does this mean?

Having evaluation capacity implies sufficient knowledge and experience to:

• Guarantee impartiality in the choice of questions and the formulation of conclusions
• Clarify the intervention logic
• Design and apply a data collection and analysis method suited to the questions and context
• Guarantee the reliability of data and the validity of analyses
• Treat value judgements in a structured, transparent and impartial way
• Distinguish the findings, conclusions, lessons and recommendations
• Present evaluation work synthetically
• Optimise the usefulness of the evaluation for different audiences.

Evaluation capacity is required both:

• in the service managing the evaluation
• and in the external evaluation team.

How to assess professional experience?

An evaluation professional meets most of the following criteria:

• Varied evaluation experience (regions, themes, sectors, tools)
• Training and experience in evaluation methods and tools
• Active participation in one or more professional evaluation networks
• Practical experience.

.

.

What does this mean?

The stakeholders are individuals, groups or organisations that have responsibilities and/or direct or indirect interests in the evaluated intervention. They may be affected by the intervention or not.

Why identify them?

• To take their points of view into consideration
• To benefit from their expertise and knowledge
• To encourage them to use the evaluation.

How to take them into account?

• Identify the different stakeholders from the outset, on the basis of available documentation and interviews.
• Identify their expectations and take them into account when drafting the evaluation questions.
• Analyse each one's possible contributions (information, expertise, contribution to the interpretation of data).
• Choose an option to benefit from their contribution, for example: individual interviews, participatory workshops, focus groups, expert panel(s), integration in the reference group, etc.
• Choose the modalities of dissemination of the evaluation results to ensure that they reach the stakeholders.

Recommendations

• Talk to the stakeholders to know what their needs and priorities are.
• Involve the stakeholders in the interpretation of information to avoid mistakes and reach sounder conclusions.