|
Selection of the field of application
In evaluations, multi-criteria analysis is seldom used for the whole range of the topics under study. Thus, the fields where the analysis will be undertaken should first be determined.
Identification of the intervention rationale
Once the evaluation team has defined the field of application, the logical framework of the intervention should be identified or reconstructed. Indeed, as various members will judge competing activities, the intervention rationale must be fully understood by everyone in the team.
STAGE 2: Choose the negotiation/judgement group (s)
Selection of group members
Multi-criteria analysis is based on the rating and preference of members of the negotiation group (in a planning process) or judgement group (in evaluations).
The group must be selected by the evaluation team. This choice is crucial and the members can be selected from two categories:
"Simple" stakeholders
The "simple" stakeholders are the people concerned with the impacts of the programme (for example, beneficiaries or victims). They can also be project technicians, public administration managers, etc.
Representatives of the "simple" stakeholders
They can be elected representatives (local or professional), leaders of NGOs (such as associations of stakeholders, associations for the environmental protection, consumer protection, for the defence of women's rights), public administration managers, donor representatives, etc.
Members of the group are usually selected from this category.
Skills of group members
Group members should be sufficiently informed about the topic under study and aware that their points of view will need to be presented to the evaluators and debated.
Fair representation of group member's views
The evaluation team must ensure that the various types of stakeholders and all the viewpoints relating to the activities being compared are fairly represented within the group.
Possibility to appoint several groups
Some evaluators appoint two groups: a group of specialists, who rate the activities using judgement criteria (for example, the ability of an activity to have a sustainable outcome), and a group of decision-makers in charge of the weighting of judgement criteria. In country evaluations, this type of specialisation in groups is not wide spread because it is complicated and expensive in resources.
Necessity for an agreement on the overall objective between members
The group will be effective only if its members agree on the intervention's overall objective (for example, to improve the protection of the natural environment of a specific area). If the objective is not agreed unanimously, working on specific objectives will be problematic (for example, to protect forest biodiversity), and even more challenging for operational objectives and activities at the core of these objectives (for example, to promote the certification of sustainable forest management, to create nature reserves, etc.).
STAGE 3: Choose the technical team responsible for supporting the judgement team group
The number of staff appointed to the technical team will depend on the complexity of the study.
The mediator
Main mission
The mediator is the key actor in multi-criteria analysis. Its role is crucial because a group cannot usually formulate judgements alone. His /her main tasks are:
- To focus the analysis scope and clarify the problems and issues under study
- To assist members with the reconstruction of the logical framework of the project or programme under study
- To participate in the definition of activities, choices and scenarios which will be incorporated within multi-criteria analysis
- To assist the group in the determination of judgment criteria and the drafting of weighting rules
- To assist the group in the rating of activities per criterion
- To decide, in agreement with the group, to ask experts to carry out additional studies, in order to assist the group in its decision-making
- To be responsible (if necessary) for the management of the technical assistant whose computer skills should be adapted to the specific multi-criteria analyses
- To co-ordinate the analysis process by maintaining the group's cohesion and fostering the participation of everyone
Required skills
The mediator should be selected for his/her skill in this type of intervention. Good knowledge of the field under study is an asset. The mediator should be aware of potential bias in the methodology, in order to avoid presenting misleading findings. Also, mathematical operations should not be opaque to the members of the group (for example, different classifications between two activities provided by the weighted sum and the weighted product method).
In country evaluations, the mediator should be familiar with the context of the development assistance.
The technical assistant and the experts
Technical assistant
The technical assistant should have a full knowledge of the software required for the undertaking of specific multi-criteria analyses.
Frequently, wide-scope multi-criteria analyses require the use of calculating and even cartographic software packages. The proper use of these tools requires specific knowledge. In addition, he/she should be able to present the findings in a clear and understandable way.
Experts
The group may need additional information during the analysis to carry on its work (for example, details about the impacts of a specific activity on the environment). In this case, one or more experts specialised in the relevant field will be required.
Their assistance may be limited to providing advice, but they may also be asked to undertake in-depth studies.
STAGE 4: Establish the list of competing activities to be included in the analysis
Selection of the activities to be compared
Depending of the objectives, multi-criteria analysis helps comparison of:
- Scenarios or potential solutions in a planning or ex ante evaluation (for example, measures to alleviate poverty)
- Choices for land-use planning (for example, selection of the location of a harbour, layout of a road, etc.)
- Activities implemented in a programme (for example, advice on various activities scheduled within an education programme)
The choice among these options is crucial and must be understood by all the members of the group. Some options can be presented visually (two different roads layouts drawn on a map), while others are more conceptual (various types of business assistance).
The mediator should therefore make sure that members of the group have fully understood the content of each activity.
Production of a list of activities
At the end of stage 4, a list of activities, scenarios and choices relevant to the analysis should be produced.
Definition of the content of the activities
In some methodologies, this list can be completed progressively, through an iterative process of thinking. This technique facilitates the identification of the optimal solution (or, depending on the case, the consensus solution), and progressively describes the content of these activities.
The definition of activities may be long and sophisticated. Yet, in ex ante evaluations, for example, the evaluator often proceeds to the selection of activities of a programme after the definition of the activities.
STAGE 5: Determine judgement criteria
This is the core stage of multi-criteria analysis. It can be very technical and it is defined by specific rules.
Rules for the selection of criteria
Characteristics of the criteria
Basic rules apply to the definition of criteria:
- Criteria should be defined by rules recognised and accepted by all, prior to the undertaking of the analysis.
- They should integrate all the points of view expressed by the members of the group (for example, economic interests for some members, environmental ones for others).
- They should be unique (for example, a particular interest should not be measured by several criteria).
- They should constitute a coherent whole, resulting in plausible and non-disputable findings.
Example: recruitment advertisement in an enterprise
When an enterprise needs to recruit a employee of a given skill level, the head office publishes an advertisement and uses the following criteria:
- Minimum grades in key subjects in examinations.
- Motivation and experience to be evaluated during a professional interview.
- Level of salary expected by the candidate.
In this example, criteria are easily understandable and acceptable to everyone. They are established prior to the selection, in order to remove the risk of preference or bias from members of the enterprise. The head office ensures that all views about the recruitment process have been taken into account.
Criteria should be unique, although the criteria for the expected salary and the level of experience are likely to be related. They should be coherent: if two candidates obtain the same rating in two criteria out of three, the third criterion should distinguish them without giving rise to complaint.
This voluntary simplistic example illustrates the rationale of the criteria's construction in ordinary situations.
Categories of criteria
Four categories of criteria
Criteria can be very varied. In the usual bibliography of multi-criteria analysis, four categories of criteria are often encountered, which are easily applicable to all varieties of fields:
- Economic
- Environmental
- Social and organisational
- Legal and political
Technological criteria may be added to these categories, if necessary, such as in reliability studies on specific technologies, and the programme's utility criteria.
Development assistance criteria
The following are examples for each category of criteria in the field of the development assistance:
- Economic criteria: cost of the activity; capacity of an activity to create sustainable enterprises and employment, to attract foreign currencies, to alleviate poverty, etc.
- Environmental criteria: capacity of an activity to improve water quality, to encourage waste recycling, to protect specific species, etc.
- Social criteria: social acceptability of the activity; acceptability to local customs; impact of the activity on minorities; capacity of an activity to improve the literacy level, maternal health, the ratio girl to boy at school, etc.
- Legal and political criteria: political risks inherent to an activity; conformity of the activity with the national rules, etc.
How should criteria be constructed?
The mediator often carries out this stage. He/she ensures that the rules for the criteria's selection are respected, and that the proposed system is coherent and comprehensive.
Techniques for the criteria's selection
The techniques are very varied, from brain-storming to a list prepared in advance by the mediator or the technical working parties.
At the end of the study, which may sometimes take a long time, all points of view must be incorporated within the criteria system.
Except where the group does not have a full knowledge of the topic and overlooks important components, the process of the criteria selection is judged to be comprehensive when the group does not have any further complaints. This often happens when each member of the group sees one of his/her preferred activities get the highest score.
Use of criteria categories
The evaluator can refer to the categories of criteria to ensure comprehensive use of the criteria.
Example: criteria selected for the multi-criteria analysis of an urban waste management programme
The following criteria could be chosen for the ex ante evaluation of a series of activities included in an urban waste management programme in a developing country:
- Economic criteria: cost of the collection and treatment utilities, transports costs, employees and maintenance costs, etc.
- Environmental criteria: sensitivity to pollution from storage sites, control of the chemical releases, area needed for the project, etc.
- Social and organisational criteria: necessity to move populations, participation of beneficiaries, necessity to resort to technical assistance, etc.
- Legal and political criteria: conformity of the future utilities with the country's laws and regulations, local job opportunities, acceptability of the utilities to the local population, etc.
If required, each of the criteria can be divided into sub-criteria. For example, the criterion relating to the participation of beneficiaries can be divided into willingness to pay, willingness to carry out their part of the work as counterparts, etc.
STAGE 6: Determine each criterion's relative weight
Methodology for the weighting of criteria
Objectives of the weighting
Once the criteria are established, one of the rules in multi-criteria analysis is to weight these criteria, in order to measure their relative importance for the members.
In secondary school, the coefficients allocated to each subject during the evaluation of the students' work is an example of criteria's weighting. When one subject is judged to be more important than another, its weight within the average is higher than the others.
In certain multi-criteria analyses, the criteria's weights are based on a consensus (for example, the tender evaluation grids for public calls for tenders, where the judgement rules determine in advance the weight for each criterion). The judgement group, which in this case is the tenders evaluation panel, only attributes scores per criterion for each offer.
In most cases, the criteria's weight is not the subject of consensus. Each member can give a different weight to each criterion.
This stage can therefore reveal important divergences of opinion between members, similar to the selection criteria stage.
Weighting coefficients method
The most well-known example of the use of coefficients is the evaluation of students during their school years.
In the simplest cases, the group can be asked to put the criteria on a scale, and allocate coefficients to each of them (for example, each criterion is given a coefficient scoring from 1 to 5, which reflects its weight).
The distribution of a pre-determined total of points between the various criteria by each participant (for example, a total of 20 points to be allocated among 6 criteria) can also be a simple solution. If the weighting coefficients method is adopted, a criterion cannot be given a 0.
Weighting matrix of 6 criteria using coefficients
This matrix shows how members can distribute a maximum of 20 points between various criteria. The right hand column of the matrix shows the level of consensus (or opposition) between participants for a criterion.
"Playing cards" method
In complex situations, various methods have been developed to improve the organisation of the weightings (for example, the Macbeth method, resistance matrix, etc.).
Among these methods, one of them is simple enough to be understood by everyone and sufficiently sophisticated to take account of notions such as equality, preference and net preference. This method, called "playing card", is used as follows:
- All the criteria are copied on cards such as the ones used in playing cards. Some of them can be blank (without any criterion)
- Each member of the group receives a series of cards for each criterion and as many blank cards as he/she wishes
- The rule stipulates that each participant must rank the criteria (usually, in a decreasing order). The method offers two possibilities: the participant can get a draw for several criteria; or insert several blank cards between two criteria, in order to indicate his/her preference between two criteria.
- Two basic rules must be agreed by everyone at the start: one deals with the total number of the cards accepted (for example, no more than 8 cards is authorised for the classification of the criteria); the other deals with the way to rate criteria (for example, the gap between the best and the worst score should not exceed a multiple of 5).
The outcome is a classification of criteria which precisely weights the notion of preference.
'Playing cards' method
In this example, the scores of the various levels reflect the preference gap among criteria: the 8 levels have been given scores from 5 to 1. The participant x sees the criteria C1 and C5 as the most important, but has no preference between them. C7 also has a good score but the participant indicates the difference of preference with the first two criteria by inserting a blank card between them. And so on, until C6, which is the least important criterion for this participant (so much so that he/she separates it from the other criteria with three blank cards).
Establishment of veto, indifference and preference thresholds
Definition of veto thresholds
Some criteria may have such importance that they have to be singled out. This is the case for criteria determined by a veto threshold. These thresholds can be absolute or have relative values, intervene alone or in series, etc., depending on the choices made by the group.
Example
An examination can require an average mark from the candidates higher than 10 and marks higher than 5 for each subject to pass the exam. In this example, two veto thresholds intervene in series.
Relative values may be used, for example, in the different costs in two projects or two offers. Whatever the difference of quality between projects, the chosen criterion can be that the cost of the best project should not be higher than double of the cost of the next cheapest project.
Veto threshold imposed by the regulation
Veto thresholds corresponding to regulation rules can also be used. If, for example, in the study of various motorway alternatives, detailed studies undertaken during the process of definition of the route show that one choice destroys a rare biotope, regulation/rules can act as a veto threshold and reject this choice.
Definition of preference and indifference thresholds
Preference and indifference thresholds also need to be defined, especially for long and complex analyses.
Actors without a common understanding about the subjects under study may not apply the rules the same way, which can lead to distortions.
For example, if these thresholds have not been clearly defined in advance, two actors with very similar opinions may rank two activities differently: one may put them at the same level, and the other at different levels, because preference and indifference thresholds had not been sufficiently defined.
For each criterion, the thresholds must be clearly understood by everyone. They are usually determined in relation to the degree of the detail yielded by data measuring the criterion.
Sensitivity analysis
Objectives
When the group has finished constructing the system for the preparation of the analysis, it is important to test its sensitivity. This test examines the impact of modifications to the parameters selected by the group on the findings of the analysis.
Thus, all the rules established by the group can be tested, in order to see, for example, if variations in the rating of performances, in the weighting, in the aggregation of sub-criteria within a criterion, in the definition of a threshold, etc. have an important impact on the analysis findings.
In long and complex analyses, tests for sensibility can be carried out at various stages of the process.
Usefulness of a software
These tests can be relatively easy to conduct if the analysis is undertaken with a software package which provides the findings of the simulations immediately.
Advantage of the test
In addition to checking the smooth process of the system, these tests, which should preferably be carried out with the members of the group, are informative and demonstrate the reliability of the system.
|
