Evaluation methodological approach

.

What is this?

The scope of the evaluation is everything that is judged. It is defined in terms of various dimensions such as the territory concerned, the period under consideration and the regulatory framework, for example:

• All the funds allocated by the European Union to Latin America since 1998.
• Local development actions planned in Albania over the past four years.

The central scope of the evaluation matches the very contour of the intervention that is under evaluation. A second broader perimeter specifies the related actions and the elements from the context to be taken into account, especially for the analysis of external coherence/complementarity. This is the extended scope, as opposed to the central scope. 

The central scope is specified in the terms of reference and the extended scope in the inception report.

Why specify the evaluation scope?

• To identify the commissioning body's expectations better and define clearer priorities for the evaluation
• For the evaluation team to focus on priorities and not to waste its resources on areas of secondary interest.

• the world (global evaluation)
• a region
• a country
• one or more areas within a country.

- in terms of the period under consideration

• the entire period during which the evaluated intervention was implemented (case of an ex post evaluation)
• or only that which was implemented during the years [n-4 to n-2] so that the evaluation team can observe real effects in the field
• or by including the preceding and/or next intervention.

- in terms of the regulatory framework

• a project
• a support programme to sector policy
• a country strategy
• a regional co-operation agreement
• an instrument.

- in terms of the sector

Only where relevant:

• an entire sector or field of intervention (education, agriculture, etc.)
• or only a sub-sector
• or a set of sectors.

• The interventions of national authorities or other donors covering the same territories
• … or targeting the same groups
• … or addressing the same problems.

- in terms of co-funding

• interventions co-funded with national authorities
• interventions co-funded with other donors.

• Limit the scope to allow more in-depth analysis of effects and a better understanding of the phenomena.
• In the case of co-funded, multisectoral interventions, select a portion of the institutional and sectoral scope for a high-quality evaluation.

A warning

Be careful not to confuse a sample (of individuals, firms, etc.) selected for survey purposes, with the evaluation scope. The sample is used to collect data and to infer results extrapolated for the entire survey group. In this case, the evaluation scope is the surveyed group. 

Don't confuse scope and question, Examples:

• scope = the entire country strategy; question = the effect on gender equality (question)
• scope = multi-donor programme to support the national education policy; question = sustainability of the impact on the management of the education system.

A sector (e.g. education) can constitute a delineation of the scope, but the same does not apply to the cross-cutting issues (e.g. equal opportunities) that resemble questions. For example, a programme to support the national education policy (scope) will be evaluated from the point of view of gender equality (question).

.

.

What does this mean?

The evaluation may concern a simple intervention such as a project that produces effects directly in the field, or a more complex intervention that produces its effects through other interventions at a lower level. The questions will obviously differ, depending on what is evaluated.

Why make the distinction?

• To adjust the questions and the process to the nature of the evaluated intervention.
• To clearly understand that a complex intervention is not evaluated with the same methods as a simple intervention.

A project is an indivisible operation, delimited in terms of schedule and budget, and usually placed under the responsibility of a single operator. The simplest projects to evaluate are homogeneous, that is, characterised by:

• One main activity
• Implemented in a single context
• Intended for a single group
• A single direct result expected for that group
• A single expected impact at a more global level.

The evaluation of a simple project is facilitated by the fact of focusing on a single cause (the main activity) and a single effect (expected result for the targeted group). Owing to this simplicity, techniques such as questionnaires or comparison groups can be used. It is also possible to ask many different questions and to answer them satisfactorily. 

A project evaluation often focuses on questions of interest to managers, operators and the targeted group.

- Evaluation of a programme

A programme is a set of simple, homogeneous interventions grouped together to attain global objectives. It is delimited in terms of schedule and budget and usually placed under the responsibility of a monitoring committee. 

A homogeneous programme is made up of parallel and similar projects. Evaluation of a homogeneous programme can consist of syntheses of project evaluations or extrapolation from a sample of projects. 

So-called "integrated" programmes encompass heterogeneous but complementary interventions with a common objective. Such programmes are complex objects in so far as they are designed with:

• Several activities (subsidies, direct investments, technical assistance, etc.)
• Implemented in several contexts (different geographical areas, different sectors, etc.)
• Targeted at several groups
• Expected to produce multiple results for the different groups
• But expected to generate one common impact at a more global level.

The evaluation of a complex, heterogeneous programme is complicated by the fact that multiple causes (activities and instruments) and effects need to be studied, not to mention the synergy effects between the different components of the programme, which in fact justifies the programme's very existence. 
Due to this complexity, some traditional evaluation techniques such as questionnaires or comparison groups cannot be used, although approaches based on case studies remain applicable. 
To avoid adding to the complexity and to preserve the quality of answers, it is necessary to limit the number of questions processed. 
Evaluation of a complex programme is neither the sum nor the synthesis of evaluations of components of the programme. It focuses on questions that are relevant to the programme managers but are seldom if ever addressed at lower levels (effects of synergy, relevance of the sharing of resources between components, contribution to meeting the goal).

- Evaluation of a strategy

A strategy covers a wide set of simple or complex, weakly inter-related interventions. It is defined by priorities rather than as a set of clearly delimited interventions. 

Unlike a programme, a strategy generally has several objectives with differing degrees of priority. 

Evaluating a strategy involves the same difficulties as those encountered in the evaluation of a complex programme. The same approaches are used to deal with these difficulties:

• Focus the evaluation on a few key questions
• Ask questions that specifically concern the strategy and not only components of the strategy.
• Answer these questions with more than simply a combination or synthesis of evaluations of the components.

.

.

What does this mean?

Public interventions are usually classified by sector or theme. A particular intervention or set of interventions can be evaluated from the point of view of a cross-cutting issue. This is also referred to as a thematic evaluation. These terms need to be clarified.

Examples

Interventions are classified in a particular sector according to the nature of the activities and outputs. For instance:

• Support to training primary school teachers will be classified in the Education sector
• Advice provided to groups of farmers will be classified in the Agriculture sector

- Cross-cutting issues

A cross-cutting (or horizontal) issue is defined in relation to impacts and not outputs, as shown by the two examples in the table below:

- Themes

The term "theme" is sometimes used in the sense of a sector and sometimes in the sense of a cross-cutting issue. It is therefore advisable to redefine it every time it is used.

• Take cross-cutting issues into consideration in the preparation of evaluation questions. Questions on cross-cutting issue s are particularly relevant in the evaluation of a complex intervention such as a strategy or programme.

Warning:

The same term can denote a sector or a cross-cutting issue , as shown in the table below: