Evaluation methodological approach

What is a cost-effectiveness analysis?

A comparative methodology for economic evaluations

Purpose

Cost-effectiveness analysis identifies the economically most efficient way to fulfil an objective. Focused on the targeted major result of the activity(for example, the number of jobs created), the tool associates the effectiveness of a programme with its cost. 

Definition

The costs are the expenses planned for the implementation of a programme. The effectiveness of an evaluation lies in the relationship results and objectives. The objective of cost-effectiveness analysis is to estimate the cost of the result's implementation. If we take the above example, the analysis estimates the cost of each job generated by a specific measure. 

A comparative tool

The tool compares policies, programmes, or projects (for example, the costs of different programmes with similar impacts - which requires the identification of situations sufficiently similar to allow relevant comparisons). Conversely, in activities with similar costs, results and outcomes are compared. The tool presents alternatives, in order to identify the most appropriate one to achieve an result at least cost.

Fields of application

Decision-making assistance

Cost-effectiveness analysis is especially used in the context of the assistance to decision-making in public policies and programmes, and in public and private investments. In fields such as health, education, environment, employment, and road safety, cost-effectiveness analysis is frequently used to compare complex policies, although the tool is appropriate for other fields.

Its implementation may be challenging, particularly when an activity yields different types of outcomes at the same time.

Analysis of activities

The tool is generally more appropriate for the analysis of clearly defined activities (in which costs and impacts are easily identifiable), than for the collection of information supporting the analysis of activities, whose sources are varied. The tool is not suitable for the analysis of multiple impact activities, whose effects are varied and difficult to rank.

What are the specificities of cost-effectiveness alalysis?

Its place among economic evaluation tools

Cost-effectiveness analysis belongs to the economic analytical tools which assess per se or comparatively the effectiveness or efficiency of a policy or a programme.

Various economic analysis tools available for comparing intervention programmes 

In addition to these tools, some authors include cost-feasibility analysis which assesses the feasibility of a policy with the estimation of the costs it generates.

Effectiveness measured with a single outcome

In order to measure the effectiveness of a policy or a programme, the tool focuses on the main expected impact of the intervention. The analysis does not consider other impacts of the policy or programme, whether they are targeted impacts or not.

Because the tool focuses on a single dimension, the effectiveness of the intervention's results can easily be measured. This is appropriate for analyses dealing with an intervention whose primary objective and results are clearly identified, or obvious. Conversely, the tool may be inappropriate or too partial for broad scope interventions, whose implementation is characterised by various impacts (causing various and concurrent outcomes).

The determination of the single variable is one of the difficulties with the analysis.

Determination of a relevant outcome

A measure based on a physical basis, not a monetary one

The other specificity of cost-effectiveness analysis is to measure the effectiveness with indicators highlighting results and outcomes. Contrary to cost-advantage analysis, cost-effectiveness analysis does not evaluate the monetary value of the outcomes. Thus, it is an economic (and not a financial) analysis methodology.

The analysis yields the unit costs of a result. For example, a heath programme focusing on the decreasing number of ill people:

• How much does the eradication of the illness cost per individual?
• How much does the eradication of the illness cost in an awareness programme dedicated to increase literacy?
• At what cost does the programme increase the proportion of the educated population by 1%?

Which questions can a cost-effectiveness analysis answer?

Cost-effectiveness analysis may contribute to answers to the following questions:

• How much does a programme or a measure cost, compared with the cost of a particular component of its objective (for example, the cost per child attending school)?
• Is it preferable (or has it been preferable) to invest resources in an intervention, to the detriment of another, for the achievement of the target?
• What type of intervention or group of interventions yields (or would have yielded) the best outcomes in respect of the final objective and the available resources?
• How can the use of the resources be optimised (or should have been optimised), given competing needs between programmes?
• At what level of additional investment can the chosen intervention be (or could have been) a factor of improvement for the outcome?

.

.

What are the possible uses of cost-effectiveness analysis in evaluations?

In the context of a co-operation assistance policy evaluation, cost-effectiveness analysis can be used as an evaluation tool dedicated to projects, programmes or sectors. It is an economic tool which analyses operational objectives at various levels. Conversely, it is not adapted to country strategy analyses, except when the strategy focuses on a single objective, whose implementation targets a clearly identifiable result.

Thus, this tool can only contribute to the analysis of the coherence of the assistance.

Possible uses of cost-effectiveness analysis depending on the type of evaluation

At the start of the evaluation

It can be useful to choose the programmes or projects which will be the subject of a cost-effectiveness analysis at the start of the evaluation. Indeed, selecting the programmes with prioritised co-operation objectives, and impacts which are measured in homogeneous units, avoids making distracting analyses of efficiency and impact during the evaluation.

This selection, when conforming with the preconditions of cost-effectiveness analysis implementation, eases the identification of the priority result, to which a simple and quantifiable indicator corresponds. Thus, it orients the evaluation on the major questions and priority goals of the co-operation strategy.

An analysis centred on cost-effectiveness considerations should determine the priority goals of co-operation strategies and the programmes whose quantitative outcomes can be measured precisely.

To assess the choices in the allocation of resources

The process of drafting and negotiation of national co-operation strategies require the distribution of limited resources between programmes. Various factors influence this distribution, such as questions of sensitivity, acceptability, equity and effectiveness.

In this context, cost-effectiveness analysis is useful for the comparison of resources allocation strategies by comparing the costs and outcomes of the main interventions.

To determine strategy planning priorities

Cost-effectiveness analysis is a valuable tool for defining the strategy planning priorities. It enables decision-makers to determine priorities in relation to the costs and consequences (in terms of simple quantitative results) of the various possible initiatives needed for the achievement of a pre-established objective.

To conduct a complex debate

Determining the priorities among the results, quantifying and homogenising indicators of outcome constitute the crucial stages of a country evaluation.

When the priority results (quantifiable and measurable) are highlighted by cost-effectiveness analysis, the evaluators can debate with decision-makers and other stakeholders on the priorities of the intervention strategies and the expected or achieved key impacts.

In country evaluations, such debates with decision-makers can be as useful for the evaluation as the analysis conclusions.

Within which type of evaluation can it be conducted? 

Undertaking cost-effectiveness analysis in the context of an ex ante, intermediary or ex post evaluation generates various methodological challenges and a complex information collection operation during the implementation stage.

In ex ante evaluations

Cost-effectiveness analysis is primarily used to support decision-making. It determines whether a decision to intervene is economically well grounded, or which intervention among others is the most efficient. Thus, this type of analysis is particularly designed for ex ante evaluations, to judge the economic relevance of a measure to be taken.

In ex post evaluations

In ex post evaluations, the analysis can contribute to the measurement of the economic efficiency of an intervention already carried out. For the same intervention, ex post and ex ante analyses can yield different outcomes, for example, because unexpected costs have become necessary for the achievement of the objectives, or because the outcomes are different from the expected results.

In intermediate evaluations

The analysis can be very useful for intermediate evaluations as a means to update the ex ante outcomes and choose which options should be selected for the continuation of the intervention.

What are its advantages and limitations?

Advantages

An ex ante evaluation tool

Cost-effectiveness analysis is a simple and effective tool which compares different measures or programmes with identical objectives. It is a valuable tool for ex ante evaluations, because it facilitates the selection of various interventions on the basis of their cost. 

An educational and communicational tool

Cost-effectiveness analysis summarises the outcomes using a single quantifiable indicator. It expresses in a simple and factual way a single priority objective in an intervention, which has informative and communications value.

Visibility of the intervention's effectiveness

Cost-effectiveness analysis presents the costs of the results of the intervention to beneficiaries and decision-makers, who often appreciate such a simple and factual illustration of the intervention's objectives.

Limitations

Cost-effectiveness analysis focuses on the main direct outcome of the intervention. Focusing on the main impact means neglecting indirect impacts. Indeed, the analysis compares costs to immediate outcomes, and neglects long-term impacts.

Thus, this analysis is limited to the comparison or the evaluation of "straightforward" outcomes programmes, i.e. programmes which are relatively homogeneous and characterised by a clear and quantifiable priority. Yet, in practice, there are few programmes and interventions centred on a direct and straightforward priority impact.

When an intervention generates secondary and/or indirect results, the use of cost-effectiveness analysis may be irrelevant or counter-productive. Because these impacts are neglected, the costs generated by these impacts may be under-estimated, which can lead to a misinterpretation of the situation.

Analysis of the effectiveness, not the relevance

Cost-effectiveness analysis can determine whether or not an intervention is economically efficient and, more specifically, if it is comparatively more efficient. Conversely, the analysis does not determine whether the intervention's implementation is relevant or appropriate. 

Difficulties of the ex post evaluation

In the context of an ex post evaluation, cost-effectiveness analysis requires reliable data on the real costs of the intervention and on its outcomes.

Depending on the diversity of the resources spent during an intervention, collecting the information about the costs required for the implementation of the intervention can be difficult or even impossible. In addition, if the observation of the results was not scheduled and regularly carried out during the monitoring of the programme's implementation, the collection of the information can be complex and approximate.

Whatever the case, ex post situations are more challenging than ex ante situations because the implementation of the intervention generates unexpected costs and impacts. The data collection for these costs and impacts is difficult.

With which tools can it be combined? 

With a cost-advantage analysis

The outcomes analysis is not limited to the study of direct positive outcomes. The evaluator should also quantify negative impacts, as well as indirect and multiplier impacts, which are difficult to measure precisely. Yet, an estimation may still be valuable, to give a more accurate picture of the effective outcomes. Cost-effectiveness analysis can produce these estimations.

With a multicriteria analysis

To assess programmes or national (or regional) co-operation strategies characterised by numerous and complex impacts (direct and indirect, positive and negative), the evaluator should use weighting systems on the results. These compound methodologies (such as multicriteria analysis) can be combined with cost-effectiveness analysis or used on its own.

In this context, cost-effectiveness analysis may constitute a useful introduction to multicriteria analysis.

.

.

STAGE 1: Define the conditions for its use

Conditions for the implementation of cost-effectiveness analysis

Check the relevance of the analysis to the objectives of the programme

This stage checks whether cost-effectiveness analysis constitutes a relevant analytical tool for the programme under study.

If the outcome of a programme cannot be defined as a priority outcome, or if homogeneous and quantifiable units cannot be determined, the use of cost-effectiveness analysis should be avoided.

Identify the availability and reliability of the data

Before undertaking cost-effectiveness analysis, the availability and reliability of the data relating to the costs and the quantitative indicator on the results and outcomes should be checked. The analysis requires reliable data, i.e.:

• In ex post evaluations, a quantification of the outcomes
• In ex ante evaluations, a rigorous modelling of the most probable results

Determine the effectiveness criteria and develop the relevant indicator

Selection of the effectiveness criteria

Once the feasibility of cost-effectiveness analysis is confirmed, the choice of the effectiveness criteria depends on the main objective of the intervention.

In an intervention where the objective is clearly determined, the identification of the effectiveness criteria is straightforward. Conversely, when the intervention's objective is broad (for example, an intervention improving the efficiency of basic education), the identification of the intervention's main objective should be discussed (for example, choosing between the reduction of illiteracy, the increase in the average level of the basic knowledge provided in primary school, or an increase in the number of primary school pupils).

The quality of the answers produced by cost-effectiveness analysis depends on the selection of the criteria. This selection could be done, when required, through the participation of stakeholders in a focus group. In a national programme, the selection should be the responsibility of the specialists of the area under study, gathered together in an expert panel.

Example of the selection of effectiveness criteria

Selection of a relevant criteria

The development of criteria depends on several factors. In many countries, the scarce and limited reliability of statistical sources limits the range of choices. Thus, finding means of observation required for the use of an indicator is crucial.

For example, when an intervention aims at improving the effectiveness of the basic education, the effectiveness criteria could be the increase in the average level of the primary school's basic knowledge. This increase can be measured through the evolution of the grades obtained in all the courses followed by the students, or in the two courses deemed as the most important (for example, reading/writing and mathematics), or through the organisation of a single examination for all primary school students.

In most of the interventions, the selection of the indicator can strongly influence the conclusions of the analysis. Thus, it is recommended that this selection should be conducted during a focus group or an expert panel session. The selection of the effectiveness criteria should be conducted likewise.

Although a single focus group is enough to determine the criteria and the indicator, complex cases require more time to determine the criteria and organise a second working session for the indicator selection.

STAGE 2: Evaluate the total cost of the programme

Add direct costs

In the context of the analysis of the development assistance intervention, the simplest and more commonly used methodology consists of adding all the public resources used in the programme. In this type of calculation, only the direct costs invested in the intervention are considered. These costs are often financial: subventions, financial transfers, decreases in taxes, financing of projects and activities, etc.

Direct costs calculation

Examine indirect costs

Indirect costs, such as the costs of human resources required for the management of programmes, indicate the value of civil servants' work in charge of monitoring the programme or intervention. They should be taken into account whenever they constitute an important part of the intervention's costs.

However, estimating indirect costs can be complex and it is recommended that the relative importance of these costs in relation to the intervention's direct costs is assessed first, in order to decide whether the indirect costs are likely to be important for the analysis.

Indirect costs calculation

Examine other types of costs

Cost calculations can include the loss of earnings and benefits due to the fact that public financing have been attributed to a specific objective (this is called the lost opportunity cost).

This methodology for cost estimation is justified when alternatives for the use of resources are clearly identified, which is seldom the case.

Calculation of other costs

STAGE 3: Measure the impact of the programme

This is the most difficult stage of the process.

Preconditions: nature of the information

The study of the programme's impact depends on the nature of the information at the disposal of the evaluator. 

Ex ante evaluations

In ex ante evaluations, the evaluator must forecast the quantitative results of the programme. Depending on the complexity of the intervention, the use of simulation techniques may be required.

Ex post evaluations

In ex post evaluations, the evaluator can use empirical techniques if the primary data at his/her disposal are sufficiently numerous and reliable. If not, the evaluator needs to estimate the actual quantitative results from secondary data.

The analysis may be undertaken, as follows:

• Allocate weightings in order to examine the various impacts or secondary variables.
• Define the criteria and methodologies used to quantify qualitative elements. Theoretically, cost-effectiveness analysis only uses quantitative elements; yet, quantifying variables which are a priori qualitative (for example, quantification of the results of environmental policies) may also be required and helpful.
• Transform quantitative data of various types and sources into a single and homogeneous quantitative unit.

Example

Impact measure

STAGE 4: Establish a cost-to-effectiveness ratio

Comparison

The analysis requires stable elements to support the comparison between:

• Interventions of different nature, but similar objectives, and occurring in the same country
• Similar interventions occurring in similar contexts
• The results and what would happen without the intervention, etc.

Comparison of programmes with identical outcomes

When the analysis compares different programmes with identical outcomes, the study is straightforward because the chosen parameter is the cost comparison criteria.

Comparison of programmes with identical costs

When, for the same objective, the analysis compares different types of interventions with identical costs, the study becomes challenging.

The comparison between indicators from different quantitative outcomes will be partly supported by qualitative elements. This is the case, for example, when the study compares the number of people cured, and the number of infections averted, with a view to assessing the relative cost-effectiveness of curative and preventive measures in the health field.

Decision-making

Decision-making is usually political. The evaluator uses cost-effectiveness analysis as a decision-making assistance tool designed to facilitate the decision.

Develop and lead the process of thinking

The tool can be used:

• To structure the thinking-process prior to decision-making
• To lead the groups representing various categories of beneficiaries and stakeholders involved in the sectors in which the intervention is planned toward the definition of preferences

Combine cost-effectiveness analysis with multicriteria analysis

Cost-effectiveness analysis is not a sufficient tool to conduct a debate between the representatives of various interests and help them define priorities together. It must be supported by a multicriteria analysis. In this context, cost-effectiveness analysis constitutes the first stage of the multicriteria analysis.

Example

Cost-effectiveness ratio

• Examples

• 'Analyse coût-efficacité et VIH/SIDA', ONUSIDA, Meilleures Pratiques Collection, August 1998.
• Levin H. M., McEwan P.J., 'Cost-effectiveness analysis', 2nd edition, Methods and Applications, Sage Publications, London, New Delhi, 2000.
• Macmillan D. et Morrison R., 'Analyse coût-efficacité de la création d'un écosystème de terres boisées', 11ème Congrès forestier mondial, Turkey, October 1997.
• " Méthodologie de l'évaluation économique des programmes de santé ", MEDCOST website.