Evaluation methodological approach

Why are these tools used in evaluation?

These diagrams are usually used as organisation tools. They provide a framework for the information collection and the undertaking of in-depth interviews during the assessment of the programme or strategy. They relate direct outcomes and field effects with the expected objectives and effects of the programme. The construction of the diagrams should therefore constitute one of the first tasks of the evaluation.

The objectives diagram illustrates the objectives classification, from the global objective to detailed operational objectives.

The effect diagram displays the classification of the results, outcomes and effects of what is intended from the implementation of the objectives system.

Figure 1: Rationale of the objectives diagram and the effect diagram

What are the possible uses of these diagrams?

In the evaluation context, diagrams are used to reconstruct the intervention logic of the European Commission's support to the country. This reconstructed logic will be shaped into one or more logical diagrams of effect. Prior to the preparation of effect diagram(s), the team will have to prioritise the stated cooperation objectives and translate these into intended effects. These intended effects will form the "boxes" of the diagram(s). Possible "holes" in the intervention logic will be indicated and filled on the basis of assumptions to be validated by the reference group. The effect diagram(s) will help to identify the main evaluation questions.

Figure 2: example of an objectives diagram

The faithfully reconstructed objectives diagram displays the objectives system and provides the evaluator with a first approach to the strategy and policies inner quality. Indeed, an unclear, incomplete or incoherent diagram means a lack of relevance in the resulting planning or a lack of faithfulness to the initial objectives system.

During the implementation of the strategy or programme, external events, such as the evolution of the world commodities market price, elections, political changes, or the conclusion of international agreements can influence the achievement of the objectives and expected outcomes. Comments dealing with the discrepancies between the expected and the observed outcomes should take these events into account. A would-be coherence
Diagrams establish logical links between objectives or effects. Each objective or effect is presented as logically dependent on a higher row objective or effect. The outputs of the activities implemented or scheduled by the programme appear as a contributions to the overall objective/effect and support the coherence of the objectives and effect system.

A strategy or a programme seldom address the full scope of the overall objective, which is limited to choices for each row of objectives or effects. Unless the evaluation can find an explicit explanation of the choices made in strategic documentation, it must provide an answer to series of questions:

• Do these choices represent the only possibilities?
• Are the objectives the most relevant ones for the row 'x'?
• What external factors have led to this selection?
• What reasons have led to the elimination of certain objectives?

How are they constructed?

After determining the evaluation scope, the evaluators should construct a diagram illustrating the objectives presented in the strategy and planning documents. The objectives diagram is drawn from this diagram. When the evaluation scope covers one or more strategy papers (geographical) or strategic policies (sector-based, thematic), it is recommended that one diagram per document is created (unless there is a logical continuity in the strategy or the policy). When a logical framework has supported the drafting of a programme, it clearly states the various levels of objectives targeted by the programme. The logical framework is consequently a reference point for the evaluation, as a consequence of the presentation of the objectives diagram's basic constituents. Although the establishment of result-based logical frameworks has not yet been generalised, the effect diagram can usually be deduced from the logical framework of the intervention's objectives. The objectives diagram and effect diagram can also be used in evaluations for projects and programmes whose rationale is not explicit in the logical framework. In practice, the objectives and intended effects of complex policies and strategies often lack explicit presentation and logical structure, whereas the justification for an evaluation is to be able to answer the following question:

• Have the objectives or intended effects of the policy or the strategy under assessment been achieved?

What are the preparation stages for the construction of the diagrams?

Stage 1: delimit the evaluation scope

The terms of reference include information about the timeframe and the financial tools to be assessed. Note that projects and programmes originating from previous documentation can also be implemented during this timeframe. 

Stage 2: identify the objectives and effects

Collect the documentation required for the establishment of the diagrams. It should comprise:

• Policies, strategies and programmes baseline documentation
• Complementary official documentation

An overall analysis of the break points in the global strategy should be undertaken, without including the objectives of the projects and programmes in progress and scheduled for a different timeframe from the one assessed.

Establish a list of the objectives recorded in this documentation. 

Stage 3: construct a faithfully reconstructed objectives diagram

A provisional classification could be carried out on the basis of the distinction between three levels of objectives:

• Higher-order objectives (goals)
• Short-term and medium-term intermediate objectives
• Operational objectives

La définition des relations logiques occupe une place centrale dans le classement. Elle est avant tout affaire d'expérience. 

Stage 4: convert the faithfully reconstructed objectives diagram into a logically reconstructed objectives diagram

Some faithfully reconstructed objectives diagrams may reveal logical defects in strategy or political papers, such as:

• Strategy or political papers involving a range of objectives without sufficient details to classify them
• No definition of the higher-order objective (goals)
• Weak relevance of the causal links between objectives
• The objectives of a given row do not result in any objectives

In order to prepare a comprehensive and coherent objectives diagram, the evaluation team will need all available documentation, its own expertise and, if required, that of experts.

Each of these rationalisation operations should be explained in a technical note. 

Stage 5: construct the effect diagram

Objectives diagrams and intended effect diagrams share the same rules of construction. The effect diagram is constructed from the conversion of each of the objectives into the corresponding intended effect:

• The higher-order objective corresponds to the higher-order effect
• Intermediate objectives correspond to intermediate effects or outcomes
• Operational objectives correspond to results

Computer devices

Most of the graphics can be addressed with software such as MS PowerPoint.

How are the findings presented?

Objectives diagrams are established during the organisation stage, where reports and notes should be provided. At this stage, the diagram's construction must be precisely described.

For the faithfully reconstructed objectives diagram, the sources of the objectives/effects (quotations, references to the original documentation) must be provided. References to documentation, interviews and expertise must support the objectives' location in the diagram, and the assumptions developed during the construction of the diagram must be explained.

Figure 3 : example of the EC strategy in Tanzania

The process through which the logically reconstructed objectives diagram has been extracted from the faithfully reconstructed objectives diagram must also be clearly explained. 

Verbal presentations

The evaluation team will need to present its work (methodology and findings) to different types of people (the evaluation reference group, participants in debriefing sessions). The objectives diagram and/or the effect diagram are very efficient tools for this purpose, providing that they are readable without being over-simplistic.

To do so, a main diagram, and several sub-diagrams developing fundamental sections of the main diagram, should be presented, each of them not exceeding 20 items (boxes).

What are the preconditions for its use?

What are the advantages and limitations of the tool?