Evaluation methodological approach

Why is this tool used in evaluation?

Strategies result from a procedure, taking into account global objectives, contextual elements as well as the position of the stakeholders. This procedure is usually empirical. The decision diagram sketches the strategy drafting process, particularly for the selection of key information, the participation of stakeholders in the process, and the management of the implementation arrangements. It highlights the choices that have been made when the strategy was elaborated as well as the consequences of the selected objectives and their forecast impact. The decision diagram is a useful complement to the objectives diagram

The documents which present co-operation strategies usually display a range of objectives among which one or occasionally more global objectives, operational objectives relative to development assistance projects, and a range of intermediary objectives at various levels can be identified. The selection carried out by the authors of the strategic and programming documents depends on various sources:

• Recommendations of the European institutions (Council, Parliament, Commission)
• Analyses of the external events (major events, country situation)
• Intervention of non-European Commission actors (partner governments, Member States, other donors)
• Lessons learned in previous programmes or projects

The aim of the decision diagram is to describe the impacts of such orientations, contextual data and analyses. Indeed, each box of the diagram's central column represents a choice (selected and rejected objectives), while the boxes on each side illustrate the flow of inputs which represents the external justification for these choices. 

What are the possible uses of these diagrams?

The decision diagram highlights:

• The range of options for the establishment of the objectives system (overall objective, selected and rejected intermediary objectives, etc.)
• The external events influencing the decision-making

The decision diagram facilitates the analysis of the strategy in terms of internal coherence (logical succession of the choices) and external relevance (contextual elements and position of the stakeholders).

When the terms of reference of an evaluation require an analysis of the partnership, the diagram is used to highlight the intervention of the main partners (governments, Member States and other donors) in the strategy design, the establishment of the programmes and the selection of the projects.

The diagram can perform the same role for the analysis of the 3 Cs (Coherence, Co-ordination, Complementarity).

How is the decision diagram constructed?

The drafting process of the decision diagram continues in two steps:

• Determination of the points at which the decision-making took place (selection of the objectives)
• Identifying, collecting and analysing of the relevant information

What are the preparation stages for the construction of the diagrams?

Stage 1: determination of the points when the decisions were made

Usually, the establishment of the decision diagram follows the construction of the objectives diagram. The objectives diagram is the basic tool for identifying the points at which the decision-making takes place. Apart from the definition of the overall objective, each intersection of the diagram represents a decision-making point. It stimulates the following questions: Why have these objectives been selected? Why have others been rejected?

Stage 2: drafting of questions

The questions focus on the justification of the selection of objectives. The identification of the rejected objectives helps the accuracy of the wording of questions. How are these objectives identified? At each decision-making point, the evaluator may encounter four situations:

• Strategy and planning papers explain why certain objectives have been rejected and identify reasons to support the decision.
• Working papers (interim versions, notes and various correspondences) provide elements to pinpoint rejected objectives and may justify the choices made (in principle, they always include the positions of Member States and the response of the European Commission).
• The documentation available does not explain the choices made, but interviews with the decision-makers provide relevant information.
• The evaluators have not collected any written or verbal information relevant to the explanation of the choices made.

In the first three situations, the evaluators should investigate which of the objectives were planned but eventually rejected, and the reasons for rejection. In the last situation, would-be objectives should be identified.

Stage 3: collection of useful information

Before the construction of the diagram, the quality of the sources of information should be checked. If this verification reveals insufficient sources, the construction of the diagram should be abandoned.

Usually, written information can be found in four types of documents:

• Strategy papers and programmes can provide information about the lessons learned from previous implementations, the political, economic and social context, and the interventions of Member States and the main donors.
• The preparatory documentation (meeting reports, notes at the end of preparatory missions, correspondence, internal notes or notes between services) may provide explanations about the priorities chosen.
• Evaluations can be valuable in terms of lessons learned.
• The European Council and Commission's more general documents (conclusions, recommendations, reports), as well as the international treaties and agreements often display the contextual elements, lessons learned and priorities which are known to the designers of strategies and programmes.

The evaluation team should formulate its conclusions about the quality of the sources in terms of quantity, relevance, reliability and accessibility. This judgement is presented to the managers, who take the final decision.

Stage 4: constructing the temporary diagram

Preliminary selection of the relevant information is carried out on the basis of the objectives diagram. It requires:

• The establishment of one or more timelines, describing the successive events and/or information included in the drafting process of the strategy and the programme
• The selection of texts from the documentation collected, relative to the choice of the objectives and the scheduled assistance process, or relative to the factors influencing such choices
• The classification and the construction of an index for the texts

Provisional answers to the evaluation questions are formulated on the basis of the information collected. Some of the questions may not be answered at this stage.

The objectives diagram supports the decisions chain. It identifies four (sometimes five) decision-making points dealing with:

• The global objective
• The first row intermediary objectives
• The secondary row intermediary objectives
• Operational objectives

The drafting of strategies and programmes is not strictly and exclusively driven by such a rationale. Implicitly or explicitly, the designers of the strategy and programme begin with an overall objective. Thereafter, they examine which means at the disposal of the European Commission are able to achieve this objective. The successive choices can be synthesised in two points:

• The choice of the global objective
• The choice of the fields and the intervention processes

At the left of the sketch, a magnifier explains the decision's outcome:

• Selected objectives
• Rejected objectives.

A global fishbone shaped diagram is thus completed through detailed diagrams corresponding to each decision-making point.

Stage 5: testing of the temporary diagram

The decisions and their explanation must be confirmed by the main actors responsible for the drafting of the strategy and the programming, including the European Commission's services (head office and delegations), the representatives of the other stakeholders (Member States, NGOs, etc.), the usual interlocutors in beneficiary countries and/or their government.

Stage 6: constructing the final diagram

The process of testing of the temporary diagram may question some of its parts when the justifications do not illustrate the real strategic and programming drafting process.

In this case, the information should be reviewed and augmented by another consultation round. The final and temporary diagrams have the same shape (a main diagram and the sketch of the point of decision-making). The final diagram includes an explanatory table about the analysis of the information collected.

What are the preconditions for the diagrams use?

What are the advantages and limitations of the tool?