Evaluation methodological approach

Why is this tool used in evaluation?

The implementation of case study reviews of one or more actual examples, in order to gain an in-depth knowledge of the topic and, if possible, to learn about the entire evaluation programme.

In complex situations, case studies are the preferred evaluation tool when "how" and "why" questions are being posed, They allow a detailed examination of the actual elements in line with the evaluation goals. The purpose of the case study is to provide a picture, which is often more reliable than the outputs from other tools in context of the scarcity of basic data (which is often the case in country evaluations).

Figure 1- Case study's components

If case studies include the analysis of documents, statistical and implementing data, they are mostly known as a field observation tool and a means to interview people directly involved in the programme, such as the officials and stakeholders.

How is a case study carried out?

What are the conditions for the use of this tool?

To ensure that a case study is credible and yields satisfactory results the geographic evaluations specific context needs to:

• find an effective local partner, who should be neutral to the topic under evaluation and competent both in the theme to be studied and in evaluation techniques
• plan for supervision procedures of the international and local working teams
• keep control of the selection of participants
• have a fair distribution between interviews with officials and beneficiary representatives

Figure 2 : How is a case study carried out? Steps involved in case study implementation

How is the instance selection undertaken?

This selection is crucial because an incorrect basis for selecting an instance can lead to a flawed evaluation outcome and can jeopardise its generalisation.

The United States General Accounting Office suggests 3 possible keys for instance selection:

• Convenience
• Purposive samples
• Probability

Example of the country's selection criteria for the European assistance evaluation in the water sector:
Be included in the list of the country's main beneficiaries of the watsan European assistance
2. Have a representative in every region of the world
3. Have water sanitation as a priority intervention sector
4. Not being influenced by the last evaluations conducted by the European evaluation Unit

How is the case study organised and planned?

A modus operandi defining how to carry out one or more case studies is always useful.

Figure 3 : A modus operandi defining how to carry out one or more case studies is always useful.

Among other advantages the carrying out a pilot case study:

• validates the methodology with a field test
• determines more precisely the categories of respondents and the basic bibliography
• completes the interview guidelines and the questionnaires
• reviews the questions asked to the evaluators, criteria and indicators, in the light of what is available on-site
• designs a standard report to be followed by the other evaluators

How are the data collected?
To ensure that the case study findings are reliable, a number of fundamental elements should be carefully taken into account:

• The data collection should include adequate longitudinal data, i.e. data covering a sufficiently long period of time in order to avoid taking an anomalous situation as a reference point.
• The data collection should be based on a principle systematically adopted in the evaluation: information verification through triangulation.
• The evaluator must ensure that the information collected is thoroughly used, and that nothing important has been overlooked. It is essential to obtain as much information as possible, especially when opinions differ among the people interviewed.
• By definition, the case study is open to any possible discoveries throughout the course of its implementation. Thus, the evaluator must know how to identify key features during the case study implementation and focus on them, even if they were not expected or scheduled in advance.
• The field stage leads to first-hand observations of what is occurring. The evaluator must also note them down carefully.

How are the results analysed and interpreted?
This is the most challenging stage of the case study. Its goal is to analyse the data that have been collected during the fieldwork and to link as far as possible the effects of the observed facts to their causes. This analysis is difficult to conduct because it is less structured than at the conception and the collection stages.

The analysis overlaps with the data collection stage, and this is particularly true for case studies in which:

• The data collection stage includes a pre-established hypothesis test that may partly modify the study content during its implementation.
• The study is large enough to allow the evaluator to review and refine his criteria for the next data collection as a result of the initial findings.

What are the preconditions for its use?

What are the advantages and limitations of the tool?