| EVALUATION AT REGIONAL LEVEL |
|
For an evaluation at regional level the questions concern, for example:
- The strategy (and possibly the preceding strategies), its design and implementation, and the relevance of its objectives, taking into consideration the evolution of its context
- The instruments of cooperation set up at regional level
- Support for regional integration.
The questions below are examples. Many are drawn from recent country evaluations undertaken by the Evaluation Unit.
Questions on relevance
- To what extent is the Commission's strategy suited to the priorities of the region's integration process?
- To what extent has the Commission identified the needs/problems that justify support at regional rather than national level?
- To what extent is the Commission's strategy concentrated on the institutions/partners in the best position to deal with regional integration problems?
- To what extent does the Commission's regional strategy take into account the specific needs of countries in the region?
- To what extent is the Commission's strategy suited to the changing needs of the region and the countries comprising it?
- To what extent does the strategy incorporate the experience gained in comparable regions?
|
|
Questions on effectiveness
This site proposes a method for choosing questions peculiar to a particular aid sector, based on the analysis of the intervention logic. The method generates the following standard type of question:
- To what extent does European aid in the sector contribute towards [generating effect X (peculiar to the sector)]?
These are questions on effectiveness (achieving a specific objective or obtaining an expected effect). Each standard question can be formulated in many other ways:
...by specifying the scope for example:
- To what extent has European aid in the sector [planned in the framework of the regional strategy underway] contributed towards generating effect X?
or by specifying the effect concerned, for example:
- To what extent has European aid in the sector contributed towards generating effect X [in the least developed countries]?
...or by changing the evaluation criterion, for example:
- Has European aid in the sector contributed towards generating effect X [sustainably]? (sustainability)
- Has European aid in the sector contributed towards generating effect X [with limited costs compared to …]? (efficiency or cost-effectiveness)
It is also useful to ask questions on generic effectiveness, that is, cutting across an entire intervention logic, for example:
- To what extent have the Commission's interventions contributed towards reinforcing the process of regional integration?
- To what extent have the Commission's interventions contributed towards improving the capacity of regional aid institutions?
Questions on efficiency
Questions on efficiency (also called cost-effectiveness) may be specific to an aid sector and may be in the following form:
- Has European aid in the sector contributed towards generating [effect X (specific to the sector) with limited costs compared to …]?
It is also useful to ask generic questions on efficiency, for example:
- To what extent have Commission interventions helped to produce the expected effects with the lowest possible transaction costs?
- Is there evidence of greater value added by Community aid in the case of actions that have mobilised the most resources?
- In what proportions have EC resources been used to achieve the various objectives of its regional strategy?
|
|
Questions on sustainability
Questions on sustainability are generally specific to an aid sector and are formulated as follows:
- To what extent has the Commission intervention helped to generate [effect X] in such a way that it will continue after the end of the aid?
|
|
Questions on coherence/complementarity
- To what extent does aid in sector X (e.g. roads) contribute directly or indirectly towards attaining the objectives in sector Y (e.g. trade)?
- To what extent does the strategy design take into account potential interactions and conflicts with the other EC policies?
- To what extent is the Commission's strategy at regional level complementary to the strategies defined in each of the countries of the region?
Questions on cross-cutting issues
The cross-cutting issues most frequently mentioned in recent evaluations are:
- Gender
- Environment
- Good governance (decentralisation, efficient administration that is accountable and oriented towards serving the people)
- Human rights, the rule of law, developing the role of civil society
- HIV/AIDS
Each of the following questions may concern one or more cross-cutting issues:
- To what extent has the EC integrated [cross-cutting issue X] into the design and implementation of its interventions?
- To what extent has the EC contributed directly or indirectly towards the development [of cross-cutting issue X]?
Questions on instruments and implementation
- To what extent has the choice of different instruments and their combination been appropriate for generating the expected effects?
- To what extent have the monitoring mechanisms been focused on the results and helped to improve them?
- To what extent have administrative procedures of allocating funds and managing actions facilitated or impeded progress towards the strategic objectives?
- To what extent have the design and implementation of interventions allowed good take-up by regional aid institutions?
|
|
Example of a recent evaluation
The following example presents the nine questions chosen in the case of a recent evaluation in the MERCOSUR region (with several important simplifications):
- How consistent is the Commission's strategy or support with the priorities of MERCOSUR's integration process?
- To what extent does the EC strategy take into account the specific needs of the member States of MERCOSUR and their evolution during the evaluated period?
- To what extent has the Commission's support helped to enhance MERCOSUR's ability to negotiate an inter-regional partnership agreement effectively?
- How have the economic and commercial projects helped to reinforce MERCOSUR's integration process?
- To what extent have the Commission's cooperation procedures (including mechanisms for implementing and granting funding) affected potential results?
- To what extent has the EC coordinated its strategy and ensured its coherence with its other policies that have an international dimension, especially the environment?
- To what extent has the European strategy contributed towards the continuation of MERCOSUR?
|
| SECTOR OR THEMATIC GLOBAL EVALUATION |
|
For an sector or thematic global evaluation the questions concern, for example:
- The effects peculiar to the sector, and their sustainability
- The integration of cross-cutting issues
- The effectiveness or efficiency of instruments peculiar to the sector.
The following questions are examples, many of which are drawn from recent country evaluations undertaken by the Evaluation Unit.
Questions on relevance
- To what extent are the Commission's interventions adjusted to the specific needs of countries (or regions) [category X]?
- To what extent does the design of the Commission's interventions make it possible to obtain an appropriate combination of activities and instruments for meeting the objective [X] ?
- To what extent does the Commission consider the evaluated sector/theme as a priority, and to what extent are these choices justified?
- To what extent has the Commission's strategy concerning the sector/theme incorporated the lessons learned from other sponsors' experiences?
|
|
Questions on effectiveness
This site proposes a method for choosing questions from the analysis of the intervention logic. The method is generally based on the analysis of one or more strategic documents that define a coherent set of objectives. Such documents may not exist in the case of thematic or sector evaluation on a global scale, but that does not preclude the more informal reconstruction of the intervention logic. It is then possible to formulate questions with the following standard form:
- To what extent did European aid help to [generate effect X (specific to the sector/theme)]?
These are questions on effectiveness (attaining a specific objective or one of the expected effects). Each standard question can be reformulated in many ways:
...by specifying the scope for example:
- To what extent has European aid [specifically devoted to the sector/theme under evaluation] contributed towards generating [effect X]?
- To what extent has European aid [implemented according to a particular procedure / using a particular tool] contributed towards generating [effect X]?
...or by specifying the effect concerned, for example:
- To what extent has European aid contributed towards generating [effect X for the poorest categories of the population]?
or by changing the evaluation criterion, for example:
- To what extent has European aid contributed towards generating effect X [sustainably]? (sustainability)
- To what extent has European aid contributed towards generating effect X [with limited costs, compared to …]? (efficiency or cost-effectiveness)
It is also interesting to ask generic effectiveness questions, that is, questions cutting across the entire intervention logic, for example:
- In which cases has European aid contributed most to developing institutional capacities and a governance suited to needs in the sector?
- To what extent have the Commission's interventions made countries able to identify their needs and to plan appropriate actions?
- To what extent have the Commission's interventions allowed it to respond as a priority to the people's needs and especially to those of the most underprivileged groups?
Questions on efficiency
Questions on efficiency (or cost-effectiveness) can be specific to the sector/theme. They are formulated as follows:
- Has European aid contributed towards generating [effect X] with a cost comparable to that of good practice observed in similar countries?
- In which countries (and circumstances) has European aid contributed towards generating [effect X] with a particularly favourable cost-effectiveness ratio? What are the reasons for this success?
- In what proportions have EC resources benefited the various groups/regions concerned?
|
|
Questions on sustainability
Questions on sustainability are generally formulated in one of the following two ways:
- To what extent has the Commission's aid contributed towards generating [effect X] in such a way that it will last after the end of the aid?
- To what extent has the Commission's intervention contributed towards generating [effect X] in such a way that it will last even if the aid is limited [to investment only]?
|
|
Questions on coherence/complementarity
- To what extent do interventions in the sector contribute directly or indirectly to the objectives of another sector?
- To what extent do interventions in the sector have mutually reinforcing effects (or contradictory effects) with other EC policies?
- To what extent are the various EC policies mutually reinforcing (or contradictory) as regards generating [effect X]?
- To what extent does European aid in the sector take into account potential interactions or conflicts with other European policies concerning the sector?
- To what extent has coordination with bilateral and multilateral interests allowed more productive dialogue as regards generating [effect X]?
- To what extent has the Commission contributed towards coordination between the different development partners, and to what extent has this allowed progress on the various paths towards achieving [effect X]?
Questions on cross-cutting issues
The cross-cutting issues mentioned most frequently in recent evaluations are:
- Gender
- Environment
- Good governance (decentralisation, efficient administration that is accountable and oriented towards serving the people)
- Human rights, the rule of law, democracy, development of the role of civil society
- HIV/Aids
Each of the following questions can concern one or more cross-cutting issues:
- To what extent has the EC integrated [cross-cutting issue X] into the design and implementation of its interventions?
- To what extent has the EC contributed directly or indirectly towards the development [of cross-cutting issue X]?
- In which countries/circumstances have EC interventions contributed most successfully towards the development [of cross-cutting issue X]?
Questions on instruments and implementation
- To what extent have interventions in the sector implemented [tool X] and with what effect?
- To what extent do administrative procedures of allocating funds and managing actions facilitate or impede the achievement of [effect X]?
|
|
Example of a recent evaluation
The following example presents the nine questions chosen in the global evaluation of the 'Commerce' sector:
- To what extent are the Commission's programmes affected by the specific needs of the country/region for the negotiation of future agreements?
- To what extent are the Commission's programmes affected by the specific needs of the country/region for the benefit of international trade?
- To what extent do diagnostic and analytical practices in the identification and programming stages of the project cycle make a combination of activities, contributing to a balanced achievement of the objectives, more likely?
- To what extent do the modalities of implementation (including available instruments and resources) determine the achievement of objectives?
- To what extent have Commission interventions enabled the countries to strengthen their institutional framework for regional commercial agreements?
- To what extent have the Commission's programmes made the countries able to identify and articulate their needs and to improve their position in international trade negotiations?
- In those countries and regions in which the Commission has financed the largest number of interventions for trade, to what extent have agents in the productive sector effectively used the outputs of those interventions?
- In those countries and regions in which the Commission has financed the largest number of programmes for trade, to what extent have those interventions affected the volume, value and direction of trade?
- To what extent have EC initiatives helped to counter-balance possible negative effects of trade on human rights, basic social rights and environmental issues?
|
