Evaluation methodological approach

The questions serve to concentrate the work on a limited number of points in order to ensure that the conclusions are useful and of a high quality. They therefore have to be carefully prepared and worded with precision.

Ensure that the answer to the question will be useful

As far as possible, the evaluation questions are proposed together with a comments on the following points:

• Which users will be interested in the answer to the question?
• How will they use it?
• Considering the time needed to finalise the evaluation, will the answer to the question arrive in time to meet users' expectations?

If there is uncertainty on the usefulness of the question, it is better to exclude it and to concentrate the evaluation on other more useful questions.

A question for knowing, understanding and/or estimating the effects of the intervention, for example:

• Has the intervention contributed towards generating effect X and, if so, in what way?
• To what extent has the intervention helped to generate effect X and in what way?
• Which groups have benefited the most from the intervention and in what way?
• What types of know-how have the former trainees acquired?

Questions of this nature reveal new aspects of the intervention. They help to understand the effects and impact mechanisms, and raise the level of knowledge.

A question for formulating or helping to formulate a judgement on the evaluated intervention, for example:

• Has the intervention contributed towards generating effect X satisfactorily compared to the objectives? In this example the question implies an 'effectiveness' judgement.
• To what extent has the intervention contributed to generate effect X at a reasonable cost?
• To what extent have the actions funded by the Commission improved gender equality?
• Were the causes of shortcomings in municipal management sufficiently analysed and clarified ?

Questions of this nature allow users to judge the merits of the intervention and to recognise good and unsatisfactory practices. They use the evaluation to communicate on the intervention, positively or negatively.

A question for showing how the intervention can be improved, for example:

• Has a particular way of implementing the intervention contributed better towards generating effect X as expected?
• Has the use of a particular implementation modality contributed to generate effect X more sustainably ?
• To what extent did the distribution and coordination of work among the partners improve the sustainability of progress regarding food safety?
• Was the choice of a strategy based on occupational training rather than schooling relevant to obtain the creation of new tourist activities?
• To what extent did the choice of infrastructure projects contribute towards improving the accessibility of the most disadvantaged areas?

Questions of this nature lead to recommendations based on lessons from experience. The answers serve to prepare reforms or adjustments.

The three types of question are not exclusive. On the contrary, there is a progression in the nature of the questions:

• in order to judge one first has to know
• in order to decide one first has to know and then to judge.

If all the questions of the same evaluation have no purpose other than furthering knowledge and understanding, the exercise is more a study or a piece of research than an evaluation.

The nature of use has different levels: to decide, one has to have judged, and to judge one has to have understood. It is therefore enough to draft the question in relation to the highest level of use (question for decision-making or judging). Uses on a lower level can be considered as sub-questions. For example the question might be:

• Has the intervention contributed towards generating effect X at a satisfactory level compared to the objectives?

and a sub-question would be:

• Has the intervention contributed towards generating effect X and, if so, in what way?

• Prepare several questions of a different nature within the same evaluation.
• Avoid having a list of questions that concern knowledge only, for this would constitute research or a study rather than an evaluation.

• If the question concerns only compliance with rules and procedures, it is an audit rather than an evaluation question.
• If the question covers only the progress of outputs, it is a monitoring question.

If a question concerns audit or monitoring, there are two options:

• Exclude the question and ask for it to be included in another exercise.
• Amend the question so that it has an evaluative dimension.

Evaluation, auditing and monitoring do not serve the same purpose. Consequently, the questions asked are not the same in each exercise. It is important to check whether a question is relevant to evaluation and, where necessary, to amend it so that it does not primarily concern auditing nor monitoring.

Amending an auditing question

• How were the regulations applied during the implementation of the intervention?

This question is limited to the verification of the legality and regularity of the implementation of a project, which is a matter of auditing. In an evaluation it would be relevant rather to ask whether the application of regulations was a particular factor of effectiveness or ineffectiveness.

• To what extent did the application of regulations favour or hinder Effect X?

Amending a monitoring question

• Did the pace of outputs correspond to the original schedule?

This question concerns only the programme outputs, which is a matter of monitoring. In an evaluation it would be relevant to ask whether the quality of outputs is a particular factor of efficiency of inefficiency.

• Was the pace of outputs sufficient in underprivileged areas for the poorest groups to be reached first?

Generally, an evaluation question concerns both the effects of the intervention and the intervention itself:

• either the entire evaluated intervention, from its design down to its implementation (the scope of the question is the same as the evaluation scope)
• or an aspect of the intervention, for example, its design, its implementation, the use of an instrument, the application of a principle of implementation (the scope of the question is more limited than the scope of the evaluation).

• to focus the question on aspects of particular interest to the designers and managers
• to produce recommendations concerning a particular aspect of the intervention.

These questions may refer to the design of the intervention as a whole or to a particular step in the design process, such as:

• The diagnosis of problems to solve and needs to satisfy
• The analysis of other policies that overlap with the intervention
• The choice of a relevant level for designing the intervention (country, sector, project)
• The stakeholder analysis
• The choice of relevant partners (local authorities, other donors)
• The risk analysis
• The impact assessment
• The definition of objectives
• The choice of instruments
• The choice of field operators
  
  or to a particular modality such as:
• Stakeholders' involvement of the design
• Quality assessment of the design
• Taking into consideration of lessons from past experience
• Etc.

Have the procedures of dialogue with the actors favoured ownership of the strategy and increased the chances of sustainable impacts?

These questions concern the entire implementation of the intervention or a particular aspect of the implementation process such as:

• Allocation of resources to the field operators
• Modalities of devolution of decision-making in the field
• Tendering procedures
• Criteria for selecting beneficiaries
• Monitoring system
• Quality management
• Operational coordination with the authorities of the partner country
• Operational coordination with the other donors
• Etc.

To what extent do administrative funding and project management procedures facilitate or hinder the adaptation of aid to beneficiaries' needs?

To what extent have phasing out procedures favoured the sustainability of impacts as regards food safety?

These questions concern modalities such as:

• Instruments of regional co-operation such as the Cotonou Agreement or the MEDA programme
• Instruments of country-wide co-operation such as partnership or co-operation agreements
• Modalities of funding such as general or sector budget support, projects, etc.

Has the funding modality opted for made it possible to obtain better effects in terms of food safety?

To what extent have interventions in the field of transport been conducted in the form of sector-specific programmes, and what difference does this make from the point of view of effects generated?

To focus the question on the effects that are considered to be the most important or the least known.

To find the right balance between:

• the evaluation of long-term or global effects which are of particular interest to policy-makers but are difficult to evaluate, and
• short-term or direct effects that are easier to evaluate but are of interest mainly to managers.

Where possible and relevant, the question specifies the effect concerned. This is easy for effectiveness questions such as:

• Has effect X been obtained? Is it likely to be obtained?

It is also preferable to specify the effect concerned in more complex cases, for example:

• Will the effect be sustainable? Will it be achieved at a reasonable cost compared to …? (effect)
• Was the objective of achieving effect X consistent with the needs of the population? (effect / need)
• Was the objective of achieving effect X justified by the resolution of the main development problems in the sector? (effect / problem)

As shown above, many questions can be related to the intervention logic either directly (effectiveness) or indirectly (sustainability, efficiency, relevance). The only real exception is the question on unexpected effects.

Effect

• To what extent has the intervention contributed towards improving the most underprivileged groups' access to basic services such as education, health and other social services?

This question concerns an expected effect: better access to basic services. It is inferred directly from the intervention logic.

Need

• Do those actions that favour the contribution of fishing to food security give priority to the nutritional needs of the local communities dependent on fishing?

This question concerns a need that must be satisfied: better food for local communities dependent on fishing. It may be indirectly related to the intervention logic in so far as satisfaction of the need corresponds to an objective.

Problem

• Was the objective of improving the local authorities' managerial capacities relevant in the context of decentralisation reforms implemented in the country?

This question concerns a problem. Is it realistic to want to improve the managerial capacity of local authorities in the current context of the country? The question is indirectly related to the intervention logic in that it concerns an objective.

The question specifies whether it concerns:

• a precise effect, for example: Have the former trainees acquired the required skills?
• a set of logically related effects, for example: What have the effects been in terms of strengthening the institutions?
• all the effects, for example: Has the sector-specific budget support made it possible to obtain effects that are at least as good as previously, with lower transaction costs?

Questions on sets of effects are of interest to policy-makers and strategic decision-makers but are generally more difficult to answer. 

Operators and field level managers are more interested in questions on precise effects. These questions are also easier to answer. 

If the question concerns all effects, it is focused on a specific aspect of intervention design or implementation. A question that covered the entire intervention and all its effects would probably not be evaluable.

All the effects

• Did the procedures of dialogue with the actors favour actual ownership of the strategy and increase the chances of sustainable impacts?

The scope of this question is very broad: all the impacts of European Aid in a country.

Group of effects

• To what extent have the actions funded by the Commission reinforced democratic processes and civil society?

This question concerns a group of logically related effects.

A particular effect

• Can an improvement of know-how be observed among those professionals of tourism who were trained?

This question concerns a precise effect, for a clearly defined public.

The wording of the question indicates whether one is interested in:

• A direct and immediate effect, for example: Have the former trainees acquired the required skills?
• A more indirect or distant link in the chain of effects, for example: Have the former trainees spread their know-how throughout their community?
• A very indirect or distant effect, for example: How has the training contributed towards local economic development?

Questions on the most distant impacts are of interest to policy-makers and strategic decision-makers but are generally more difficult to answer. 

Operators and field level managers are more interested in questions on direct results, on the uptake by the targeted group, on their needs, or on the closest impacts. These questions are also easier to answer.

Reaching beneficiaries

• Did eligible firms request the available aid in sufficient proportion?

This question concerns an initial result.

Result

• Did the aid have a leverage effect on the investments of farmers who received it?

This question concerns a short-term result for direct beneficiaries.

Intermediate impact

• To what extent did the actions co-financed with local authorities improve access to basic services in disadvantaged areas?

This question concerns an intermediate effect (improvement of access to services).

A more direct effect would have been, for example: greater priority given to the territories neglected by local authorities.

A more distant effect would have been, for example: reduction of regional disparities in access to basic services.

Global impact

• To what extent have the modalities of implementation of EC interventions contributed towards furthering democracy and the rule of law and strengthening civil society?

This question concerns a set of distant effects.

A closer effect would have been, for example, local authorities' adoption of practices that involve actors in project design.

.

The questions are classified in different families that correspond to different "viewpoints" on what is being evaluated. Seven of these viewpoints, also called evaluation criteria, are to be considered: relevance, effectiveness, efficiency, sustainability, impact, coherence/complementarity, and Community value added.

A question is drafted in relation to an evaluation criterion unless it is intended only for knowledge and understanding. 

Questions of efficiency and sustainability are asked more rarely, partly because they are difficult to answer. Yet they are often more useful.

• To ensure that essential viewpoints are not forgotten since certain evaluation criteria tend to be neglected (e.g. efficiency or sustainability) although they can lead to very useful conclusions.
• To guide the drafting of evaluation questions.

This page proposes a typology of seven families of criteria. The first five correspond to the traditional practice of evaluation of development aid formalised by the OECD (DAC). The following two apply to all EC policies.

Relevance

The extent to which the objectives of the development intervention are consistent with beneficiaries' requirements, country needs, global priorities and partners' and donor's policies.

Note: Retrospectively, the question of relevance often becomes a question as to whether the objectives of an intervention or its design are still appropriate given changed circumstances.

Examples

• To what extent does the concentration of aid on basic education correspond to the needs of the partner country?
• To what extent are interventions in trade and regional integration in line with the needs and priorities of the partner country?
• Do actions favouring the contribution of fishing to food safety prioritise the nutritional needs of local communities dependent on fishing?
• Was the objective of improving the managerial capacities of the local authorities relevant in the context of decentralisation reforms implemented in the country?

Effectiveness

The extent to which the development intervention's objectives were achieved, or are expected to be achieved, taking into account their relative importance.

Note: Also used as an aggregate measure of (or judgment about) the merit or worth of an activity, i.e. the extent to which an intervention has attained, or is expected to attain, its major relevant objectives efficiently in a sustainable fashion and with a positive institutional development impact. Related term: efficacy.

Examples:

• To what extent has the aid contributed to equal access to high-quality basic education?
• To what extent did the support contribute towards fair access to basic education of a high standard?
• Were the outputs achieved fast enough in disadvantaged areas to be able to reach the poorest groups first?
• To what extent did the funding procedures and efforts at dialogue and co-ordination encourage/hinder the programme to reform the road sector?

Efficiency

A measure of how economically resources/inputs (funds, expertise, time, etc.) are converted to results.

Examples:

• Has implementation in the form of sector-specific financial aid made it possible to obtain the same effects with lower transaction costs for the EC and the partner country?
• Did implementation in the form of sector budget support produce the same effects as previously with lower transaction costs for the EC and the partner country?
• Did the establishment of regional co-ordinators make it possible to limit costs per person trained?
• Have the components of the programme that consumes the most resources produced the best effects in terms of accessibility of disadvantaged areas?

Sustainability

The continuation of benefits from a development intervention after major development assistance has been completed. The probability of continued long-term benefits. The resilience to risk of the net benefit flows over time.

Examples:

• To what extent has the aid contributed towards durably remedying the backlog in road network maintenance?
• To what extent has the support helped sustainability to remedy the area's backwardness as regards maintenance of the road network?

Impact

Positive and negative, primary and secondary long-term effects produced by a development intervention, directly or indirectly, intended or unintended.

Example:

• From the point of view of the groups concerned, are environmental nuisances acceptable compared to the positive effects of the intervention?

Coherence/complementarity

This criterion may have several dimensions:

1) Coherence within the Commission's development programme

Example: Can it be said that the activities and outputs logically allow the objectives to be achieved? Are there contradictions between the different levels of objective? Are there duplications between the activities?

2) Coherence/complementarity with the partner country's policies and with other donors' interventions

Example: Can it be said that there is no overlap between the intervention considered and other interventions in the partner country and/or other donors' interventions, particularly Member States?

3) Coherence/complementarity with the other Community policies

Example: Is there convergence between the objectives of the intervention and those of the other Community policies (trade, agriculture, fishing, etc.)?

Community value added

The extent to which the development intervention adds benefits to what would have resulted from Member States' interventions only in the partner country.

Examples

• To what extent has the sharing of roles between the EC and Member States contributed to optimise the impact of the support?

Use of the term criterion: A warning!

This document concerns evaluation criteria, that is, the main ways of judging the intervention.

In order to formulate fully transparent value judgements, the approach needs to be refined into evaluation questions, and then into judgement criteria.

The word "criterion" is also used with a third meaning in the framework of quality assurance. In that case it concerns the quality assessment criteria of the evaluation.

Generally questions requiring a 'yes or no' answer are avoided for two reasons:

• the answers are often multi-dimensional and have to be qualified
• decision-makers are averse to evaluations that dictate their choices.

Examples:

• To what extent has effect X been obtained more satisfactorily and at a lower cost since reform Z?
• What was the beneficiaries' opinion on the proposed aid?
• To what extent did the coordination enable the Commission to concentrate its strategy on those areas with the highest value added?
• To what extent was result-oriented management successfully applied?

This type of wording is more appropriate if the question is intended to acquire knowledge or understanding, or to aid decision-making. In such cases the users expect qualified answers.

Examples:

• Did reform Z produce effect X with a better quality/cost ratio?
• Were the beneficiaries sufficiently satisfied with the proposed aid?
• Did the coordination result in the EC strategy being concentrated on those areas with the highest value added?
• Did application of result-oriented management produce significant value added to achieve the expected effects?

This type of question is more appropriate for accountability purposes, in a context where the objectives were set with precision. A closed question can also be useful if the intention is to validate a procedure or innovative instrument or to confirm a good practice.