Evaluation methodological approach

Various uses in evaluations

Indicators are commonly used in the evaluation process, as four sets of indicators correspond to four specific evaluation stages:

1. Context indicators, which produce information about the changes in the country under evaluation, the location and the assistance provided
2. Programme indicators, which detail the resources, implementation, results, and, if possible, the impacts of an ongoing activity
3. Evaluation indicators, which, in addition to the evaluation criteria, enable the evaluator to formulate a judgement on the programme's relevance, coherence, efficiency and effectiveness, and support answers to evaluation questions
4. Monitoring indicators, which are included in the monitoring system of programmes and sometimes used by evaluators to assess the implementation of programmes

An indicator can belong to several types of indicators (for example, a socio-economic indicator may also be appropriate as a programme impact indicator) and be used in a variety of situations.

Types of indicators

There are many types of indicators.

The second volume of the Means Collection ("assessment of socio-economic programmes", European Structural Funds, Means Collection, 1999. The new edition is available here and is entirely dedicated to indicators.

Definitions

The main types of indicators include:

• Elementary indicators: basic information from which more complex indicators can be derived
• Derived indicators: ratio or rate, from the relationship between two elementary indicators
• Compound indicators: combination of several indicators (elementary or derived)
• Specific indicators: appropriate for an intervention, but not for comparisons
• Generic indicators: appropriate when comparing several activities
• Key indicators: appropriate for internal comparisons between different activities of a programme and external comparisons with other programmes
• Context indicators: appropriate for a country, population or a category of the population

Definition

EuropeAid's general definition of the indicator is:

"A datum which produces simple and reliable information
describing a change, an outcome, an activity or an input"

The definition of the context indicator could be:

"A datum which produces simple and reliable information
describing a variable relative to the context"

How are context used?

Evolution

Context indicators provide foundations for the development of other indicators. Statistical services in developed countries design socio-economic indicators in order to measure the impact of policies on society. Today, they are in widespread use, and the media exploit them to describe the evolution of the situation (for example, trends in GDP per capita, unemployment rate, inflation rate).

Construction of indicators

In the countries within the scope of EuropeAid assistance, indicators are designed by national statistical services and in the context of specific programmes. National statistical services, many donors and international organisations have cooperated to establish standard indicators, to ease comparisons over time and between geographical areas. Methodological reports covering the production of reliable and comparable data include:

• The OECD Development Assistance Committee (DAC)
• The United Nations
• The European Commission
• The World Bank Group

The European Commission's use of context indicators in country evaluations

Types of indicators

The European Commission's country evaluations usually include indicators in the chapter dealing with the context, such as:

• Economic indicators: GDP, growth, balance of payments
• Social indicators: population, unemployment, educational level, health
• Indicators of services provided to the population: education, health, drinking water, electrification
• Indicators for the analysis of poverty in ACP countries

Limitation

These indicators are often designed to highlight the specificities of a local context without, however, enabling the evaluator to:

• Make a comparison between countries (for example, information from a comparison of the level of the indicator in another country)
• Get a global and normative view of the country's situation

10 key indicators

With respect to the use of monitoring indicators in Country Strategy Papers (CSP), the European Commission has suggested 10 key indicators to support the monitoring of ACP countries.

Possible improvements

The European Commission should require the use of databases of indicators during the implementation of country evaluations.

Why are context indicators used?

This methodology, which deals with the evaluation of co-operation strategies, focuses on context indicators and therefore excludes the study of sector-based indicators (addressed in the relevant section of this website).

It is mainly designed to:

• Quickly describe the various types and models of indicators available, and the characteristics they should display
• Provide documentary sources for the range of norms used by donors in country assistance, where many indicators are already available to evaluators

To present the country

Context indicators are usually displayed in the introductory chapter of the evaluation. They deal with:

• Economic and financial fields (GDP, trade flows, debt)
• Social fields (demography, occupation, gender)
• Specific important sectors (education, health, environment)

In certain cases and countries, the emphasis may be put on particular sectors or issues (for example, poverty analysis, conflict analysis).

To portray the country's level of assistance

Context indicators also describe the nature of the assistance provided to the country. They should indicate:

• The type of assistance
• The amounts disbursed
• The sector-based allocation of assistance
• The European Commission's activity, compared with other donors'

To answer evaluation questions

Context indicators can also be used to answer evaluation questions which need a preliminary presentation. They facilitate the understanding of the country's situation for the readers.

In country evaluations

In country evaluations, the presentation of the context of the assistance is a crucial element of the study, and at the basis of all the subsequent analysis. It often requires the support of global statistical data, comparable in time and location. Some donors have already worked on general and thematic indicators which briefly and clearly present the country situation.

The indicators' value varies with the context: they are more or less available, depending on the country, but a series of indicators is not automatically relevant for every area under study.

What are their advantages and limitations?

What is a good indicator?

Although "perfect" indicators are impossible to find, their advantages and drawbacks can be depicted. Institutions have drafted grids describing the indicators' qualities, such as the following which highlight two different but close viewpoints on the qualities deemed necessary for an indicator.

First grid

According to the European Commission's MEANS Collection, a good indicator should provide information described as:

• Simple
• Communicable
• Easily understood by both the provider and the user of the information

In addition, the indicator should:

• Be available
• Be measured regularly
• Be sensitive, i.e. the quantity of measurement must reflect, to a sufficient degree, a change that has occurred in the variable to be measured
• Be sufficiently reliable to yield information which will be trusted by stakeholders
• Support internal comparisons within the programme (between activities) and external comparisons (between countries)
• Refer to a policy goal, objective, and/or target

Second grid

According to the USAID, indicators must be coherent and comparable over time and in different settings. They should also be:

• Direct, i.e. they should precisely reflect the outcome to be measured
• Objective, i.e. unambiguous about what is measured and which data are collected
• Practical, i.e. the data should be collected quickly and at a reasonable cost
• Adequate, i.e. the progress which has been accomplished should be sufficiently highlighted

Advantages

Context indicators represent:

• A way to quantify information, preferably in a standardised form, in order to make comparisons in time and space
• A way to simplify situations in order to better understand them
• Elements that can be used as evidence for presentation

Limitations

• A simplistic explanation of the situation, which, in turn, becomes exaggerated. This bias is corrected when the indicator is only viewed as a measuring tool and to be interpreted by the user.
• Errors of measurement due to the conditions under which the indicators are constructed (no systematic cross-checking and the risk of getting an incorrect indicator).
• Difficulty in developing indicators which are sensitive to slight changes in the context at the macro-economic level.
• Problems with the availability of reliable and standardised data over a long period of time.
• Differences in the understanding of the meaning of an indicator between various users, and particularly between the donor and the beneficiary country. Words may be understood but given different interpretations.

What are the preconditions for their use?

The time span

The time span is the time dedicated to finding available indicators. The evaluator can quickly complete this task using the Internet.

Difficulties may be encountered when:

• Choosing the appropriate indicator among a series of available indicators, in various websites and systems
• Checking the indicator's calculation process (not always available)

Human resources

The human resources required for the collection and selection of indicators differ largely with the themes under study.

The selection of an appropriate indicator, such as maternal health progress, may require the advice of a health expert. This type of situation will occur whenever a generalist in charge of the evaluation is confronted with technical themes.

Financial resources

The financial resources required for data collection are very limited because indicators are available on Internet, where access is free most of the time.

.

.

EUROPEAN COMMISSION INDICATORS

DG Dev indicators for the appraisal of country assistance performance

Country evaluation indicators

- Typology

The indicators used for the appraisal of country assistance performance by the European Commission's DG Dev have been constructed with the assistance of various donors, including the European Member States, the World Bank, the UNDP and the OECD/DAC.

Their construction is mostly based on the following typology, and particularly on impact indicators:

Typology of indicators

Image

- Two objectives

The construction and implementation of such indicators target two goals and require the monitoring of two distinct series of indicators, in order to:

• Measure the results of the country's policies in terms of poverty alleviation and improvement of living standards
• Detail the outcomes of sector-based policies which are targeted by the Commission's assistance

Country evaluators will probably be more interested in the first series which will help them to identify context indicators.

- Collection process

For such indicators, the monitoring system of the Poverty Reduction Strategy Papers (PRSP) is recommended. If data are not available in the country, they can be found in the website of the World Bank or the United Nations. These websites are a straightforward source of data at the disposal of evaluation teams. 

Types of indicators

A European Commission's approach proposes classifying indicators into the following categories:

• Gender
• Public or private sector
• Geographical location (with an emphasis on the poorest areas)
• Rural and urban areas
• Level of income
• Main source of income (scarce information on this)

The European Commission also suggests restricting the list of indicators to essential topics. Particular attention should be given to the reliability of these indicators. 

List of indicators

In developing countries, evaluators are advised to use a selection of 10 key indicators measuring the outcomes of national policies in terms of poverty alleviation and improvement of living standards. This list is a sample of the Millenium Development Goals Indicators .

In addition to these indicators, the evaluator may wish to monitor the percentage of the budget dedicated to education and health, and study macro-economic indicators (provided by national sources or by the IMF).

Some country-specific indicators may be added to this list, which will have been incorporated within the Country Strategy Paper (CSP) during the annual reviews.

Likewise, the list of indicators selected by the European Commission's DG Dev may be incorporated into all CSPs.

List of country outcomes indicators used by the European Commission's DG Dev

Image

Such typology of indicators does not explicitly refer to context indicators; yet, a significant part of these global indicators stand as good context indicators, and could be used for all country evaluations of developing countries.

Tables footnotes

The following points must be clarified, in order to properly understand certain items of the previous table:

• "Suggested periodicity" means the period in which it is possible to identify a significant shift in the indicator
• Indicator n°1: in the monitoring of country poverty trends, where available, indicators based on national poverty lines should be used,
• Indicator n°7: data on "maternal mortality ratio" are infrequently available, model-based and unreliable; a good proxy is provided by "births attended by skilled health personnel"

EUROSTAT data

EUROSTAT provides the Union Member States with numerous data on several topics which are partly expressed by indicators and ranked into short-term, long-term and structural factors.

Short-term indicators

• Balance of payments
• Business and consumer surveys
• Consumer prices
• External trade
• Industry, commerce and services
• Labour market
• Monetary and financial indicators
• National accounts

Long-term indicators

• European population
• Economy and ecology
• The business structures

Structural indicators

• General economic background
• Employment
• Innovation and research
• Economic reform
• Social cohesion
• Environment

Such indicators are particularly relevant in country evaluations, where they allow a comparison between members of the European Union countries and their neighbours (for example, with candidate countries). Moreover, EUROSTAT holds data on trade flows between European Union members and the rest of the world.

UNITED NATIONS INDICATORS

Indicators related to the Millennium Development Goals 

Millennium Goals

In September 2000, the members of the United Nations unanimously voted for the Millennium Declaration. Following a series of interviews with international agencies, the General Assembly defined a road map for the implementation of the Millennium Declaration called the Millennium Development Goals.

These 8 goals are designed to:

• Eradicate extreme poverty and hunger
• Achieve universal primary education
• Promote gender equality and empower women
• Reduce child mortality
• Improve maternal health
• Combat HIV and other diseases
• Ensure environmental sustainability
• Develop a global partnership for development

Millennium Development Goals Indicators

- 48 indicators

For the Millennium goals, the levels to be targeted and 48 Indicators Development Goals have been identified. They are available on the websites of the United Nations and the World Bank. They deal with more than 200 countries, and their methodologies and their precise definitions are presented to ease their understanding and use.

Likewise, DG Dev's indicators focus more on outcomes than inputs. Evaluators will benefit from the description of the context provided by such indicators.

They are available in national reports and may be obtained by country, area, theme and period. In the country-specific statistical reports of the World Bank and the United Nations, such indicators can be encountered in maps.

The Human Development Report also uses these indicators, which enables comparisons between geographical areas and over time, if required.

Common Country Assessment indicators

Common Country Assessment

In 1997, the United Nations decided to create a system of Common Country Assessment (CCA) linked with the United Nations Development Assistance Framework (UNDAF).

In this system, the CCA is used as a tool for analysing the country's development situation and identifying the main development challenges.

Many countries have carried out this appraisal task, and some still do. The task's evolution and the reports are available on the United Nations' website.

Definition of a common indicator framework

Among the tools used to carry out such a country appraisal, a common indicator framework has been drafted. Although closely linked to the Poverty Reduction Strategy Papers, this framework is independent.

Therefore, whenever CCAs are available or are in the course of preparation, they will provide the evaluators with a useful source of context indicators.

As this system originates from the United Nations, an important part of these indicators are common to the Indicators Development Goals, although differences can be found.

Development index

The United Nations database

The United Nations database is one of the most developed database in the world. Approximately 200 indicators have been developed.

This database is less developing country-oriented than other databases; yet, it presents indicators and development indices, which are designed by the United Nations and benefit from the participation of many international agencies. Based on internationally standardised data, it eases comparisons between countries and over time.

The United Nations' website also displays databases using pre-formatted tables (for example, data per country, per indicator), and with criteria more adapted to the individual needs.

List of indicators

The list of available indicators can be found in the United Nations' website, where a direct access to results per indicator can be found.

Human Development Report

Each year, the UNDP publishes its Human Development Report which includes a large proportion of the United Nations indicators.

Some reports use the whole or part of the Millennium Goals Indicators for specific countries (such as the Assessment of Development Results).

Integrated Environmental and Economic accounting

This Integrated Environmental and Economic Accounting focuses on the measurement of sustainable development, and can yield interesting data for specific countries.

WORLD BANK INDICATORS

International Development Association indicators

Context indicators

The International Development Association has recently presented in "IDA Results Measurements System: Progress and Proposal, April 2003" a series of indicators aiming at improving the monitoring of the countries' development outcomes, and particularly for countries benefiting from a PRSP programme.

- List of indicators per field

15 indicators are proposed to monitor country outcomes, covering areas that are priorities in the PRSP and reflecting IDA's activities in borrowing countries.

These indicators cover the following themes:

• Income and poverty
• Malnutrition
• Maternal and child health
• HIV
• Gender
• Primary education
• Drinking water
• Infrastructure
• Private sector development
• Public sector management
• Economic growth

- Specificities of this list

IDA's document warns that "no such list can be considered definitive or comprehensive. It should be regarded as:

• Reasonably representative of the priorities that have emerged in PRSPs to date
• Shaped by data availability
• Subject to improvement over time as our knowledge base improves"

Some of the indicators are common to those of the United Nations', and others are specific to IDA.

List of country outcomes indicators used by the European Commission's DG Dev

Image

This list displays almost the same indicators as the DG Dev's country outcomes indicators table. Many of these indicators can be used to describe a context, or a support answers for evaluation questions.

- Sources

The data come from the World Development Indicators database.

When strategy papers relating to poverty alleviation (whose name varies but are always connected to PRSPs) and periodic reports specialised in this topic are available, they usually present series of indicators which are close or complementary to the World Development Indicators database. Sometimes, they can convey a more accurate description of local specificities.

Indicators of contribution to development results

One of IDA's approaches, which assists the evaluators when completing their study of the context, is the monitoring of the following 4 indicators, which consists of measuring the indicators' contribution to development outcomes.

- Country Assistance Strategy (CAS)-level indicators

The indicators related to this field deal with:

• The adoption of results-based CASs in IDA-eligible countries, as validated by the Vice-Presidency of Operations Policy and Country Services
• CAS's final outcome ratings, as validated by the Operations Evaluation Department (OED) during the study of CAS ex post evaluation reports

- Portfolio-level indicators

• Project outcome ratings, as validated by OED through the Implementation Completion Report review
• Quality-at-entry indicators for IDA projects, as assessed by the Quality Assessment Group

The donor's decision to implement, develop and collect these indicators is a precondition for their use by the evaluator. This process should be adapted to the European Commission's Country Strategy Papers.

Although these indicators are qualitative, evaluators can use them to assess the impact of the assistance and its implementation.

International Comparison Program

Founded in 1968, the International Comparison Program is a statistical system used to produce data by country. These data facilitate international comparisons based on:

• Prices
• Expenses value
• Purchasing power parities

Because of the information about the purchasing power, this statistical system provides the evaluators with comparable data which are valuable for economic and social topics.

The production of these indicators is based on surveys focusing on the price of products commonly bought and service samples. Such surveys are carried out every 3 to 5 years.

This system will be incorporated within the database of EUROSTAT and OECD, in order to enlarge the country coverage of the World Bank. As a consequence, data dealing with more than 160 countries will be available in the databases.

OECD indicators

OECD statistic portal

OECD produces a variety of statistics and indicators, as well as definitions and methodologies to design indicators.

In its statistic portal, OECD offers a range of precise and updated information about its member countries in domains such as:

• National accounts
• Finance
• Economic projections
• Purchasing power parities
• Productivity
• International trade
• Public management
• Demography and population
• Health
• Education and training
• Labour market
• Agriculture and fisheries
• Industry and services
• Information and communication technology
• Environment
• Energy

In addition, OECD displays a link page to other websites presenting relevant data.

Available data about the development of non-member economies

Precise data (geographical, such as country-, regional-, or sector-based) dealing with non-member economies and their development are available on the OECD website.

Evaluators can use a large amount of these data as indicators. Indeed, these development data generally include precise information, such as:

• Assistance activities
• Assistance provided by members of the Development Assistance Committee (DAC)
• External debt
• Statistics on external debt from BIS-IMF-OECD-World Bank
• Development statistics

Numerous data from the OECD website describe the context in which assistance to a country is carried out.

Non-member economies data give information about topics, such as:

• General information on non-member economies
• Economic statistics on non-member economies
• Agriculture and fisheries
• National accounts

Thematic indicators

OECD thematic indicators are usually presented with precise definitions and may be usefully consulted whenever specific themes must be developed during a country evaluation.

Environmental indicators

The OECD series of key environmental indicators may be useful when describing a context.

Bibliography about sustainable development indicators

OECD also publishes important documentation about the measurement of sustainable development. Among available references, three books are particularly relevant:

• Frameworks to Measure Sustainable Development: An OECD Expert Workshop, February 2000.
• Workshop on Accounting Frameworks to Measure Sustainable Development, Paris, May 2003.
• Measuring sustainable development/ Integrated economic, environmental and social frameworks. August 2004.

Development Assistance Committee

The OECD-DAC website is equipped with a browsing system dedicated to evaluation reports of numerous donors. This documentation includes a series of indicators implemented in the context of evaluations.

Other sources

Sources presenting a series of macro-economic indicators are abundant. Among them are the following institutional sources which may be useful to the evaluators.

The International Monetary Fund

The International Monetary Fund website displays a list of indicators mainly focused on economy and finance.

Websites of international institutions

Search engines linked to websites specialised in statistical data can easily be found. This type of service is offered in the majority of the multilateral donors websites in the section "statistics", such as in the OECD website.

Evaluation reports

Evaluation reports on a specific country and its neighbours can be useful to identify the series of indicators which have been implemented. Part of most donors' websites is dedicated to the display of evaluations or evaluation's abstracts.

PARC website

The Performance Assessment Resource Centre (PARC) website presents a series of addresses enabling the user to find available reports (or their abstract) on many international donors' websites.

CAW website

The Country Analytic Work (CAW) website offers a useful database including documentation and links to country evaluations which are provided by numerous bilateral and multilateral institutional partners.

• Examples