Evaluation methodological approach

Why is this tool used in evaluation?

Objectives

Multi-criteria analysis is undertaken to make a comparative assessment between projects or heterogeneous measures.



In the evaluation field, multi-criteria analysis is usually an ex ante evaluation tool, and is particularly used for the examination of the intervention's strategic choices.



In ex post evaluations, multi-criteria analysis can contribute to the evaluation of a programme or a policy through the appraisal of its impacts with regards to several criteria.

What use can be made of multi-criteria analysis?

In ex ante or intermediary evaluations

Multi-criteria analysis can be useful:

• To evaluate the ability of various activities of a programme to fulfil a given objective. This assessment can take place to collect the opinions of decision-makers and beneficiaries about the effectiveness of the activities
• To structure the views of project or programme managers about on-going activities
• To discuss the content of the programmes, and the funding of various activities during the drafting of strategies and programmes

In ex post evaluations

In beneficiary countries, interventions in fields such as poverty alleviation, maintaining security, immigration control, or trade development can benefit from this type of analysis which formulates judgements on these complex strategies.

HOW IS A MULTI-CRITERIA ANALYSIS CARRIED OUT?

What are the prerequisites for the tool's usage?

The time span and the cost of such a high level of analysis may be unsuitable to the timescales and budgets usually agreed for an evaluation.



Thus, in country evaluations where situations are often challenging, multi-criteria analyses should use simple methodologies. The analyses should be limited to the comparison of straightforward activities, and conducted with a limited number of criteria.

Stage 1: Select the field of application and determine the intervention rationale

In evaluations, multi-criteria analysis is seldom used for the whole range of the topics under study.

Once the evaluation team has defined the field of application, the logical framework of the intervention should be identified or reconstructed if missing

Stage 2: Choose the negotiation/judgement group

Multi-criteria analysis is based on the rating and preference of members of the judgement group. The evaluation team appoints the judgement group, whose members are chosen among the programme's stakeholders or their representatives, with a preference for the latter. This should limit any incompetency risks and ease the identification process.

Stage 3: choose the technical team

The technical team is responsible for supporting the judgement team group. It comprises:

• The mediator.
• The technical assistant. He/She must have a full knowledge of the software required for the undertaking of specific multi-criteria analyses.
• The experts are responsible for providing additional data for the negotiation group, in order to complete the information progressively.

Stage 4: Establish the list of competing activities

Depending of the objectives, multi-criteria analysis helps comparison of:

• Scenarios or potential solutions in a planning or ex ante evaluation
• Choices for land-use planning
• Activities implemented in a programme

At the end of stage 4, a list of activities, scenarios and choices relevant to the analysis should be produced.

Stage 5: Determine judgement criteria

This is the core stage of multi-criteria analysis. Basic rules apply to the definition of criteria:

• Criteria should be defined by rules recognised and accepted by all, prior to the undertaking of the analysis.
• They should integrate all the points of view expressed by the members of the group.
• They should be unique.
• They should constitute a coherent whole, resulting in plausible and non-disputable findings.

When an enterprise needs to recruit a employee of a given skill level, the head office publishes an advertisement and uses the following criteria:

• Minimum grades in key subjects in examinations.
• Motivation and experience to be evaluated during a professional interview
• Level of salary expected by the candidate.
• Criteria should be unique, although the criteria for the expected salary and the level of experience are likely to be related. They should be coherent: if two candidates obtain the same rating in two criteria out of three, the third criterion should distinguish them without giving rise to complaint.

Stage 6: Determine each criterion's relative weight
 

Methodology for the weighing of criteria

One of the rules in multi-criteria analysis is to weight these criteria, in order to measure their relative importance for the members.



In secondary school, the coefficient allocated to each subject during the evaluation of the students' work is an example of weighting of criteria.



Various methods have been developed to improve the organisation of the weighing (such as weighing coefficients method or "playing cards" method).

Establishment of veto, indifference and preference thresholds

Some criteria may have such importance that they have to be singled out. This is the case for criteria determined by a veto threshold (some of them can be imposed by the regulation).



Preference and indifference thresholds also need to be defined, especially for long and complex analyses. Indeed, two members with very similar opinions may rank two activities differently: one may put them at the same level, and the other at different levels, because preference and indifference thresholds had not been sufficiently defined.

Sensitivity analysis

This test examines the impact of modifications to the parameters selected by the group on the findings of the analysis.

Stage 7: Formulate a judgement per criterion
 

Study of the impacts of the activities based on criteria

At this stage, values based on criteria are given to each activity's impact. This evaluation can be quantitative, as well as qualitative.

Activities' rating and judgement per criterion

The group has responsibility for the judgement, whereas the technical assistants are in charge of the study of the impacts, prior to the group's task.



This stage aims at providing each activity with a rating for each criteria. Comparisons between activities and between the opinions of stakeholders for the same activity can be made using this rating.

Stage 8: Aggregate judgements

This crucial component of the analysis is also the most challenging. Evaluators should first ensure that all the data are understood the same way in terms of preference by the members of the group (for example, the surface area occupied by a building is preferable when it is small). However, the risk of getting unsatisfactory findings is still great. At this stage, it is important to check whether several ways to carry out the operation yield similar or inconsistent findings (such as the difference on the scoring scale of an activity ranking first in a grid and ranking last in another because a parameter has been changed).



Several methods for the aggregation of judgements can be developed: the weighed sum method, the weighted sum product, the outranking method, etc. Whatever the methods selected to undertake the calculations and the aggregations, multi-criteria analysis should yield one (or more) performance table(s) summarising the findings per activity in each criterion (and possibly for each stakeholder).



If the study to be undertaken happens in a consensus group working with criteria of identical weight (such as for the professor grading his/her students), the performance table represents the findings of the multi-criteria analysis.

What are the preconditions for its use?

What are the advantages and limitations of the tool?