Evaluation methodological approach

What is this?

The rationale of an intervention is to address the needs, problems or issues that are considered to be priorities in a given context, and that cannot be addressed more effectively in another way.

It is in the programming stage that the rationale of an intervention must be justified. At the evaluation stage it is enough to note the main points or to redefine them if the programming documents lack precision.

The evaluation identifies the reasons for which the priorities have been chosen, for example: priorities of the policy in which the intervention takes place, urgency of the needs to be satisfied, and comparative advantages compared to alternative options.

What is the purpose?

• To help in defining the criteria of the relevance family.
• To present the intervention in the evaluation report.

How to clarify it?

.

- Reviewing the problems and responses

The evaluation report succinctly sets out the following:

• Context of the intervention when it was initiated
• Main problems diagnosed (needs, issues)
• Why the institution responsible for the intervention and its partners (where relevant) are in the best position to solve the problems diagnosed.
• Why certain strategic options have been chosen rather than others.

The evaluation team searches for this information in the official documents that instituted the intervention and in the preparatory design work. It completes its understanding with interviews with the key informants. 

Certain evaluation questions may concern problems addressed by the intervention, for example:

• Are there alternative options for solving the problems identified?
• Has the nature of the problems that justified the intervention changed?
• What is the precise extent of the needs justifying the intervention?

- Reviewing the rationale

Where relevant, the evaluation report highlights the following:

• Justification of the fact that the needs, problems and issues cannot be solved ad hoc by private initiatives.
• Justification of the fact that they cannot be solved more effectively by other public initiatives.

The evaluation team looks for this information in the official documents that instituted the intervention. It completes its understanding of the rationale by interviews with key informants. 

Certain evaluation questions may concern the rationale of the intervention, for example:

• Could the institution that initiated the intervention have chosen other partners to address these problems?
• Would the intervention have been more effective had it been run by other public institutions?

Recommendations

• The official documents often focus on the strategic options that were finally selected. If we want to know what the alternatives were, it may be useful to hold interviews with key informants, whilst remaining wary of possible biases.

What is this?

All the activities and expected effects (outputs, results and impacts) of an intervention, as well as the assumptions that explain how the activities will lead to the effects in the context of the intervention.

The intervention logic may be "faithful" to the programming documents and to the documents establishing the policy to which the intervention is related. In this case, the expected effects are inferred from the stated objectives in the official documents.

When the intervention logic is reconstructed during the evaluation, implicitly expected effects that were not mentioned in the initial documents may be taken into account. The fact that this is no longer a "faithful logic" must then be mentioned. The "faithful" approach is relevant when the objectives are expressed precisely and in a verifiable way. The other option is preferable if objectives are too vague or ambiguous.

The intervention logic often evolves over time. In such cases, it is justified to reconstruct it several times, for successive periods.

The intervention logic is a useful simplification of reality, but one has to bear in mind the complexity of the real world. In addition to the reconstruction of the intervention logic, it is useful to identify the main external factors that condition or limit the implementation and effects. One also has to remember that real causal explanations are often more complex than initial assumptions.

What is the purpose?

• To help to clarify the objectives and to translate them into a hierarchy of expected effects so that they can be evaluated.
• To suggest evaluation questions about these effects.
• To help to assess the internal coherence of the intervention.

How can it be reconstructed?

- Collect and analyse reference documents:

• Collect the official documents establishing the intervention and allocating resources.
• Identify the main activities.
• Identify the objectives.
• Translate the objectives into expected results and impacts.
• Connect the activities to the expected impacts by reconstructing the logical cause-and-effect relations (if… then…).
• Check that cause-and-effect relation are logical, i.e. considered as plausible in the light of available knowledge.
• Discuss the reconstructed logic with a few key informants (designers and managers) and with the experts of the policy domain/country concerned.
• Present and discuss the intervention logic in a reference group meeting.

The most common presentations

- Logical framework

This technique consists in producing a presentation in the form of a matrix that specifies the objectives (first column), how they should be verified (second column: indicators, and third column: sources), and the main assumptions on the external factors (fourth column).
The lines of the logical framework have a simplified presentation of the intervention logic, on only four rows: activities, results, purpose (i.e. direct benefit for the targeted group) and goal.
This representation is adequate for a simple intervention, like a project or well-targeted programme. However, it cannot fully grasp the complexity of a heterogeneous intervention such as an integrated rural development programme, a country strategy or support for a sector policy.
Due to its simplified nature, the logical framework allows for specifying the indicators that should be used at each row, as well as most external factors.

- Objectives diagram

This technique consists in the identification of officially stated objectives and a graphical presentation of the logical relations between objectives, from the most operational to the most global. The intervention logic is represented in the form of boxes and arrows.
A particular form of representation is the objectives tree. It is applied in the case where each objective on a lower rank contributes towards a single objective on a higher rank.
Unlike the logical framework, the diagram can have as long chains of objectives as necessary. It can be used to highlight synergies and complex relations between objectives (except in the case of the objectives tree).
This representation is appropriate for complex interventions such as integrated programmes, country strategies or sector policies.

- Diagram of expected effects

This technique is similar to the diagram of objectives since it also builds upon officially stated objectives. However, the objectives are translated into expected effects before being presented as a diagram. By translating objectives into expected effects, more concrete and easily verifiable concepts can be worked on.

Recommendations

• Always re-examine the intervention logic. In the ideal world the intervention logic should be part of the intervention design and should be available at the time of the evaluation in the form of a framework or diagram. In the real world these points should be verified.
• In the case of complex interventions (and especially country level evaluations), it is useful to produce several detailed diagrams and a synthesis diagram. A highly complex diagram cannot be used.
• Beware of the risk of ex post rationalisation. To reduce this risk, clearly explain the reconstruction process and the options chosen.

What are these?

Objectives are the initial and formal statements on what is to be achieved by the intervention. They may relate to outputs, direct results or impacts. 

Objectives may be expressed in a verifiable way, which specifies clearly the expected achievements. They may be expressed in more political or rhetoric way, making difficult to connect them to a precise achievement. 

The evaluation process may uncover implicit objectives, i.e. cause-and-effect assumptions that necessarily explain how an objective may be achieved.

What is the point?

Identifying and clarifying objectives is a necessary step towards:

• Reconstructing the intervention logic.
• Asking questions and defining evaluation criteria from the families of relevance and effectiveness.
• Judging internal coherence.

How to clarify them?

- Examining the clarity of objectives

The evaluation identifies the objectives and, for each one, sets out the following:

• statement of the objective
• possible ambiguities in some of the terms and clarifications proposed
• means proposed for checking whether the objective has been achieved
• date at which the objective has to be achieved.

The evaluation team searches for the objectives in the official documents that instituted the intervention and in the documents framing it at a higher political or strategic level. 

When the statement of an objective contains ambiguities, the evaluation team proposes clarifications based on interviews with key informants.

- Examining the hierarchy of objectives

The evaluation ranks the objectives of the intervention:

• global objectives correspond to the rationale of the intervention,
• operational objectives correspond to the products and services delivered by the intervention (outputs),
• one or more levels of intermediate objectives can be identified.

The reconstruction of the intervention logic helps to differentiate the levels of objectives as well as the effects, direct finding s and impacts corresponding to them. 

Some evaluation questions may concern the articulation of objectives, e.g.:

• Is there a high level of internal coherence between the operational objectives and the more global objectives?

Recommendations

If the intervention is framed by numerous documents on a political or strategic level, or if these documents also contain many objectives, the evaluation team is faced with an over-abundance of objectives. Rather than presenting them all, the evaluation team underlines this fact and concentrates on the most important ones in the context of the evaluated intervention. 

There is a continuum between more or less explicit objectives:

• an objective clearly stated in the official documents
• an ambiguous objective clarified with the stakeholders' help
• an intermediate effect that has to be obtained if an objective is to be met (implicit objective)
• an effect expected by the stakeholders but not initially anticipated, even implicitly

All these elements warrant explanation in the report, but the status of each one must be specified.

What does this mean?

The idea is to identify the main related interventions and to situate the evaluated intervention in relation to them.

What is the purpose?

• To help ask questions belonging to the coherence/complementarity and relevance families
• If necessary, to examine the quality of the intervention design and especially the fact that the objectives are complementary to those of the other related policies.

How to proceed?

The evaluation report briefly sets out the following:

• Main national and international institutions active in the same region and sector and/or targeted at the same group.
• Main policies implemented by these institutions in the same region, in the same sector and/or targeted at the same group.

This analysis offers a brief insight of the main related policies, if possible with their starting and finishing dates, the tools used, the groups and objectives aimed for, and the resources allocated. 

Certain evaluation questions may concern related policies for a more in-depth examination, for example:

• Is the intervention consistent with and complementary to the policies and priorities of the partner country and/or other Community policies?
• Has the implementation been coordinated with the actions of the other sponsors and has the complementarity been improved?
• To what extent is there value added when the intervention is conducted at EC rather than member-State level?