Evaluation methodological approach

What is meant by a "case study"?

The manager familiarises him/herself with the strategy, and particularly with the sectors and cross-cutting issues concerned. He/she identifies the key actors and composes the reference group in which the delegation is involved. The head of unit sends a note to the services concerned (example).

The case study is probably one of the most diversified evaluation tools, whose goals and content can vary greatly. So much so, that it is sometimes difficult for experts to be sure whether or not they are dealing with case studies.

To illustrate how case studies can benefit evaluations, two complementary definitions from the various published works can be cited:

According to the United States General Accounting Office (GAO 1970), one of the main evaluation institutions which has used and rationalised case studies in evaluation tasks, “a case study is a method for learning about a complex instance, based on a comprehensive understanding of that instance obtained by extensive description and analysis of that instance taken as a whole and in its context”.

According to R. Yin (YIN 2003), whose interest focuses on applied research in social sciences, “a case study is an empirical inquiry that investigates a contemporary phenomenon within its real-life context, especially when the boundaries between phenomenon and context are not clearly evident”.
The principal difference between research methodology and evaluation methodology is that in the former, a hypothesis is often chosen as a starting point that the case study will confirm or invalidate, whereas in the latter, the case study is a tool to study an example in detail, and generally with no predetermined views.

Thus, the implementation of case studies should start with a review of one or more actual examples, in order to gain an in-depth knowledge of the topic and, if possible, to obtain lessons for the entire evaluation programme.
In complex situations, case studies are the preferred evaluation tool when “how” and “why” questions are being posed, because they allow a detailed examination of the actual elements in line with the evaluation goals. The purpose of the case study is to provide a picture, which is often more reliable than the outputs from other tools in context of the scarcity of basic data (which is often the case in country/region evaluations).

The case study may include the examination of documents, of statistical data, or of programme implementation data, but, in particular, it includes the direct observation of an intervention and interviews with people directly concerned in its implementation and effects. This necessitates fieldwork by the evaluators, and this aspect is one of the strengths of the case study.

With its wide variety of use, the case study is a tool that can be used concurrently with almost all the stages in the evaluation process: data collection, observation, analysis and even in judgement.

This tool comes from the research field and has been incorporated subsequently into evaluation practice. Its use has expanded since the 1950s, and among research methods in social sciences (survey, experimentation, archives analysis…), the case study has established itself as a tool that can be used in specific situations where other tools have proved to be less effective.

Nonetheless, its general use has not developed easily, for the use of tools producing quantitative data (survey, mathematic tools, etc…) often compete with case studies. In this context, its advocates and users have established rigorous methods that allow them to obtain reliable results.

Fields in which this tool has been successfully used include social and political sciences, psychology, medicine, management, international relations, etc…

In the evaluation field, the use of the case study has covered a wide range of subjects. The first case studies undertaken were similar to research tools, and often carried out by a sole evaluator and essentially descriptive. Their main objective was to illustrate the implementation of a programme and its results through actual examples. They were often based on empirical methods.

Since then, the methodology has been progressively formalised and the approach more structured. However, the case study still remains a tool that allows the evaluator a degree of "freedom". The evaluator can place more emphasis on one aspect of the study, even though it was not designed that way (for example, the evaluator can decide to investigate the reasons why two groups of people have opposite views on a subject).

This progressive building of methodology and experience has led, inter alia, to the identification of various types of case study.

The illustrative case study primarily describes one or a limited number of instances. It can be depicted as a tool which allows the evaluator to start from a general perspective and then highlight a specific element. Because it is a very descriptive tool, it can go deeper into an evaluation task and record field elements, increasing the credibility of the argumentation.

This field approach, based on the observation of actual events enables the reader of an evaluation report to fully understand the context of implementation, and any gap between what should have happened and what actually happened. Its principal role is to highlight this gap and explain why it has appeared.

Based on an analysis of available data and a series of interviews, its goal is to collect:

• facts, and verification of facts,
• opinions and points of view,
• suggestions,
• and/or reactions to the evaluator's hypotheses and conclusions.

From this information, the illustrative case study helps with the understanding of events, and facilitates and supports the analysis made in the evaluation report. As with other case studies, the illustrative case study focuses on the "How" and "Why" questions. However, the limited number of case studies conducted may not be sufficient to draw generalisations from what has been illustrated.

This type of case study will probably be the most commonly used in country/region evaluation in the future. The exploratory case study mainly contributes to clarifying a situation in which information is scarce, and is undertaken before starting the field stage of an evaluation. This situation is often encountered in country/region evaluation.

The exploratory case study helps the evaluators assess the facts relating to programme implementation, the local context and, if possible, the findings, or the difficulties in identifying and measuring them. By giving this key information, the exploratory case study provides the basis for the establishment of the evaluation questions and the investigation methods that will have to be used. It should offer practical advice as to what questions should be asked, how and to whom.

This type of exploratory study can include one or several studies, depending on the variety of contexts to be taken into account. As exploratory case studies are not designed to provide conclusions, but only to prepare the ground for them, the level of investigation is less rigorous than in a descriptive case study, for example, and the chain of evidence is less well developed.

The reporting of such studies can take many forms but their execution needs less work than in other case study types, because their purpose is principally to ease the work that follows. Thus, they contribute more to the evaluation methodology than the accumulation of evidence.

This aspect is a potential pitfall for this type of study: it can lead to premature findings released as conclusions, in spite of a lack of depth in the study. Evaluators must proceed carefully and thoroughly evaluate their subsequent case studies once they are completed, in order to avoid confirming initial findings instead of testing them.

The critical instance case study examines a limited number of sites for one or two specific purposes (i.e. to investigate problematic projects). It analyses in detail cause-and-effect questions relating to the issues of concern, and confirms or invalidates hypotheses that have been formulated before the beginning of the case study. This aspect makes the critical instance case study methodology inappropriate for a country/region evaluation, because it is too focused on situations of extreme specificity: it is only mentioned here for completeness.

Some critical instance case studies may reveal broader issues. In this case, they can be used in country/region evaluation, but illustrative case studies should be used if the objective is to include in the evaluation a fair and balanced picture of the situation.

Along with programme effects case studies, programme implementation case studies are probably be the most appropriate tool in country/region evaluation. Their goal is to examine whether programme implementation:

• is consistent with the rules (utilisation of reference texts, which ones to select, how often have they changed; rules of monitoring-and-control, etc…),
• meets expectations (how the beneficiaries are informed; beneficiaries' interest in the programme; difficulties in understanding the documents; delays in implementation, etc…),
• leads to substantial or significant variations between sites (rules applied differently; differences in implementation between one site and another, etc…),
• causes problems for the programme outcome, and why.

Programme implementation case studies are designed to answer evaluation questions relating to implementation (and particularly to effectiveness and efficiency). They should be designed to make generalisations from their findings possible. In country/region assistance evaluation, multiple sites studies are therefore almost a necessity. The instance selection is consequently very important because it will be the basis for future investigations.

This is the most appropriate case study for country/region evaluation because it analyses the effects of programmes and strategies. It is designed to study the observed outcomes (positive or negative, expected or not) and to check whether or not they are the result of the implementation of programmes or strategies. The programme effects case study often includes an initial stage of implementation analysis, which corresponds to the coverage of programme implementation case studies, and is essential for a full understanding of the context. Thereafter, it focuses on the results of implementation and attempts to demonstrate how and why the programme's outcomes are positively linked to the observed changes.

As with programme implementation case studies, programme effects case studies also frequently aim at obtaining information that can easily be generalised. Therefore, multiple sites studies are almost inevitable. In order to give more weight to the studies' findings, it is useful to associate them with beneficiary surveys, which can give an interesting perspective on the evaluation, even though they do not always have statistical validity.

These studies concentrate on observed effects by relating differences between sites to explain links and causes between the observed changes and the evaluated programmes.

This type of case study brings together the findings from case studies or more general studies undertaken at different times. It aggregates information from one or more sites collected over extended period of time. Clearly, such studies are of a great interest and relevance but there are substantial difficulties in finding sufficient historical material to constitute the starting point for cumulative case studies.

The opportunity to implement these case studies during country/region evaluations is very limited, and the generalisation of the findings may not be a straightforward task. Thus, they are only mentioned here for the record.

In which situation is this tool appropriate?

The case study is an outstanding tool for addressing complex situations, mostly:

• when quantitative data are scarce or unavailable,
• when assessing the answers to 'Why' and 'How' questions is as important, or more important, than the data analysis itself.

Case studies can be used in all types of evaluation (ex-ante, intermediary and ex-post). However, even if they can be designed for ex-ante evaluations, in practice implementation is difficult because ex-ante evaluations have to be conducted rapidly. Usually, only exploratory case study can be adapted to ex-ante evaluations.

Multiple case studies provide the evaluators with the opportunity to observe and analyse situations that are of special interest for the evaluation (for example, aspects that are the most problematic, success stories, etc…). Therefore, this tool can be applied using cases as close to the evaluation questions as required.

Paradoxically, some case study advantages are cited as drawbacks by opponents to this tool, for example:

• its richness, made possible by detailed qualitative information. The context of implementation is precisely described and it is agreed that, on occasions, qualitative data are easier to understand than quantitative information.
• its relatively straightforward use, especially for evaluators already used to techniques of interview, triangulation, observation and data analysis.
• its flexibility, making continuous adaptation to various situations possible.
• its implementation stage is compatible with that of a country/region evaluation (a few months), and can be very short in some cases (for example, in exploratory case studies).
• the opportunity to obtain and understand information at a sufficiently deep level. This tool allows evaluators to become familiar with the logic of action of the various actors and the different perspectives they have about what is at stake, the problems, what is taken for granted, difficulties, etc, all of which can be linked to policies or programmes that are to be evaluated.
• its accessibility. It is one of the few tools within the reach of non-specialists, enabling them to understand complex situations.

The limitations of this tool

Although this tool has many advantages, it also has limitations beyond which it should not be used. Apart from the individual limits inherent in each type of case study (Illustrative, Exploratory, Programme implementation case study, etc…), a number of common limitations exist, such as:

• the difficulty in country/region evaluation of identifying the appropriate targets, leading to a risk of bias in the collection or treatment of information,
• the difficulty of identifying cases, setting boundaries (for example, a geographical zone, within a population, etc…) and linking them to problems as broad as those commonly addressed in a country/region evaluation (for example, the struggle against poverty, economic development, etc…),
• the difficulty of determining the number of cases that provide a sound basis for generalisation,
• the difficulty arising from generalisation to a global level (for example, a country) of themes that were studied at a local level,
• the difficulty of ensuring that data will always be available when implementing studies which extend over a significant period of time,
• the tool's cost. In multiple sites country/region evaluation, this tool should be limited to evaluations where its added value is identified beforehand (for example, case studies in several countries designed to evaluate a regional strategy),
• the fact that this tool rarely allows statistical interpretation of data, even if a valid survey can be done in each study,
• the fact that this tool is based on the judgement of one or more evaluators. This can lead to partiality, even with the most careful use of case study methods.

The case study has very flexible implementation requirements and evaluators can combine it with other tools and methods. Among them are tools which are fully incorporated within case studies, such as:

• interviews,
• basic data analysis tools.

Other tools can only be applied to specific studies, such as:

• the reconstitution of historical facts, mostly needed in cumulative case studies,
• the use of specific tools dedicated to the processing of data from multiple sites studies (for example, matrix of categories, tabulating event frequencies, etc…).

Some tools can be added to the study, such as:

• surveys, and particularly those targeting beneficiaries of the implemented programme,
• focus groups targeting beneficiaries or people working in particular economic sectors.

The case study could overlap with existing investigations if the contexts were similar, but as a general rule, the case study has a specific purpose and would not be made redundant by other interviews or focus group activities carried out during the evaluation.

What are the pre-conditions for the use of the case study in evaluation ? 

The time span

The preparation stage of case studies can take a relatively long time, especially during the identification of sites, finalisation of the content and determining the logistics. Typically, an elapsed period of 2 to 3 months should be planned for a multiple sites study, and 15 to 30 working days should be assigned to its preparation.
The implementation of studies can vary greatly, but planning should at least include:

• 3 to 4 working days allocated to the field mission, which could take 10 days or more if coupled with a survey.
• 3 to 4 working days (up to 10) allocated for data analysis (excluding any survey analysis) and report preparation.

As one expert cannot implement a large number of case studies during an evaluation (due to a lack of time), several evaluators should work concurrently on-site. In this case, a period of training of 1 to 2 days should be planned.

For multiple sites case studies, the benefits of a pilot case study, implemented before the other case studies should be strongly considered. Its time span would normally be a little longer than average.

The analysis and conclusions from multiple case studies can take 10 to 30 days, depending on the complexity of the cases and on their number.

Human resources

Case studies must be carried out by qualified people, who:

• fully understand the problems associated with the evaluation,
• have sufficient experience of interview techniques,
• speak the language of the people interviewed.

Financial resources

The full cost of case studies is very variable and depends, among other things, on:

• the complexity of the subject to be studied,
• the number of people to be met during the case study,
• the distance between sites to be visited,
• the location.

A minimum budget of €15,000 should be fixed and allocated to the case studies preparation stage.
A budget of at least €5,000 to €7,000 should be planned for each case study, not including long-distance transportation, assuming that no survey is to be undertaken.

When should a case study be used in project evaluations ?

Three types of case studies can be useful in project evaluations:

• Exploratory case studies: these are used to help specify the questions at the start of the evaluation, or to design a tool (for example, a questionnaire). The findings of a simplified case study cannot be generalised automatically (the validation of any hypotheses would require further, specific case studies).
• Illustrative case studies: these are undertaken for an in-depth study of all the categories of stakeholders, including final beneficiaries. Such a study aims at bringing out the factual elements of the project for its quality assessment without generalising. These case studies can be organised during the evaluation field.
• Case studies about the effects and impact of the project on various categories of stakeholders.

Why are case studies used in country/region evaluation?

The role of case studies in country/region evaluation

Case studies have a role to play in country/region evaluation at 4 levels:

• Exploratory case studies allow the formulation of more precise questions at the beginning of the evaluation.
• Illustrative case studies including an in-depth study reach the final beneficiaries and bring out the main features, without the obligation to establish a basis for generalisation.
• Programme implementation case studies and / or programme effects case studies can deal with a series of programmes within a country evaluation.
• Programme implementation case studies and / or programme effects case studies can also deal with a series of countries within a programme evaluation or a regional strategy evaluation.

The specific use conditions in country/region evaluation

To make the utilisation of the results of case studies as credible and useful as possible, and to derive generalised recommendations that work for all case studies' applications, the evaluators should:

• find an effective local partner, without any conflicts of interest, and competent in the subjects to be studied and evaluation methodology,
• establish procedures for the supervision of the work carried out by the local and international teams, so that the evaluators can manage issues relating to the physical distances involved,
• monitor the selection of people to interview and avoid time pressure on interviewees if there is a long list of appointments,
• ensure a balance between institutional interviews and beneficiaries' interviews.

In addition, in country/region evaluation, the main problem consists of determining what kind of detailed information to seek and who should be interviewed, in a relatively short time and on loosely defined questions. Case studies have similar problems.
Thus, one of the main tasks of country/region evaluation is to determine very precisely the types of actors that have benefited from the implemented policies. It is also necessary to analyse the strategies and logic of implementation developed by the various actors vis-a-vis the policies under evaluation.
The process of selecting and targeting the interlocutors is therefore very important, as is defining the questionnaire grid.

What kind of use for each type of case study in coutnry/region evaluation?

6 Types of Case Study and Their Possible Use in Country/Region Evaluation

What are the stages to follow when carrying out a case study?

STAGE 1: THE CASE STUDY PREPARATION

How is the methodology designed?

The design of the methodology includes the basis for analysis and conclusions, the importance of the study in the overall organisation, the general design and the sources of information.

During the design of the methodology, 5 successive steps should be taken before deciding how many case studies and which type to implement.

First step

Second step

Determine the case study role within the evaluation process (designing the methodology or giving data on the implementation process, etc.).

Third step

Determine the guidelines for the study's general approach. Stipulate the primary hypotheses (where they exist), the selected analysis units (for example, regions), the kind of information to be collected, etc. These guidelines must determine a precise process, the more so when multiple sites studies are planned. The final stage of this general approach and the case study process will be achieved in the pre-operational stage, once adequate data has been collected.

Fourth step

Propose the sources of information that have been identified before the implementation of the case study. These will be used to collect each type of information.

The documentation:

Documentation can be very varied and may include, for example:

• Data concerning the country, the region, or the programmes evaluated and their implementation within the scope of the case study,
• Previous study reports,
• Activity reports from operators and programme monitoring,
• Monitoring data from projects and programmes,
• Maps,
• Statistics,
• Newspaper articles,
• Contracts, etc.

This documentation can be collected on-site but may also be obtainable before the departure of the evaluators from the administration or the operators working in Europe and/or in the country to be evaluated. Some documentation can also be acquired from the Internet (for example, other tender evaluations).

The interviews:

This is the most important source of information in case studies and should be systematically planned. In country/region evaluation, however, one of the most difficult tasks is to decide on whom to focus. If some targets are obvious, others are less so. Thus, the list of the target group will need to be reviewed on-site and after preliminary interviews. This should also be done to avoid being restricted to interviewing officials nominated by the local authorities. Focus groups can also be conducted, especially with the beneficiaries.

Direct observation:

Direct observation is an absolute necessity in case studies because the evaluator has to have first-hand knowledge of the events he is studying. Most of the time, the observation includes:

• A simple review of the situation on-site (for instance, firms that are being helped, work in progress, classrooms, community clinics, etc.) and of the people (for example, the standard of living, the health standard, schooling, etc, and also the relations between stakeholders, with the authorities, etc.).
• If needed, physical measurements.
• A pre-determined minimum number of interviews, and particularly with beneficiaries.

The other sources of information:

Many other sources of information are at the disposal of the evaluator in case studies. Among them, the most common is the carrying out of concurrent surveys (in particular, beneficiary surveys) in order to add quantitative data to the qualitative data. Thus, a larger number of surveys within case studies gives the evaluators a better quantitative understanding of the observed events.

Instance selection

This selection is a crucial stage because a wrong basis for selecting an instance can lead to a flawed evaluation outcome and can jeopardise its generalisation.
If multiple sites case studies are scheduled, methods that have been chosen for the instance selection should be detailed. The United States General Accounting Office suggests 3 possible keys for instance selection:

• Convenience
• Purposive samples
• Probability

The instance selection varies with the combination of the 3 keys and also with the number of case studies.

Convenience

Using this key, the selection is justified because the data collection is straightforward (for example, a site near the capital where most of the fieldwork is implemented; a site of a limited size that will only need a short mission, etc…).
Although this key is not likely to be the best fit for a fully representative study, it can give an acceptable introduction to some illustrative case studies with a minimum of resources.

Purposive samples

This second key is the most used, and is sub-divided into several options, depending on the evaluation objectives:

• Bracketing: the evaluation seeks to explain why important differences occur within several situations that could have been identical. Case studies will focus on extreme outcomes, selected for their particular meaning to the evaluator who will attempt to demonstrate a point (gap between implementation costs, between effective outcomes, etc…).
• Best cases/worst cases: the case studies focus on projects or programmes of interest to the evaluation because of their successes/failures.
• Cluster: while most projects and programmes are very different, some belong to homogeneous sub-categories (for example, different projects focusing on one country in a regional programme; programmes implemented by two different operators in the same country, etc…). In this case, the instance selection includes these cases that can be clustered on points of interest for the study, forming sub-categories (for example, are programmes implemented by one operator better conducted than those implemented by another?).
• Representative or typical: the instance selection focuses on one or more points of interest to be dealt with by the study (for example, important programmes, health programmes, etc…).
• Special interest: for very specific cases in which the information is extraneous to the data organisation carried out by the evaluator. For example, the evaluation terms of reference specify that the programme should be considered as a case study because it presents serious dysfunctions.

Probability

Under this third key, instances are selected from a list with the help of traditional criteria for purposive sampling, with an equal chance of being included. This selection method is not widely used, mostly because the number of cases is insufficient to lead to a representative sample. It can also be argued that case studies focus by definition on very specific problems.

Combination of the 3 keys

Some combinations of different instance selections are possible. For example, representative elements can be coupled with a bracketing purpose, or specific cases.

Number of case studies

There is no minimum number of case studies; however, the more included, the more detailed the future analysis will be, along with the increased likelihood of making useful generalisations. For most of the time, case studies will not normally permit statistical analyses, therefore unduly multiplying the use of case studies should be avoided. Five to 10 studies for an evaluation should be enough to make generalisations. This figure can rise to 15 or so if a series of distinctive problems and situations need to be studied in various case studies. This is however seldom encountered.
It is essential to confirm with the authorities in charge of programme management that the selected cases have not already been the subject of other case studies implemented during past evaluations, and this should be done before fixing the list of case studies to be carried out. Otherwise, the target group will probably show signs of irritation that could affect the quality of the study.

STAGE 2: THE PRE-OPERATIONAL STAGE

Once the decision to carry out a case study has been taken and guidelines for its contents defined, the pre-operational stage can begin. This stage is designed to facilitate the operational stage. Several activities have to be conducted before starting the case studies:

Finalise the process

A modus operandi defining how to carry out one or more case studies is always useful. It is part of the design of the evaluation methodology but goes beyond it by additionally including operational aspects. While the modus operandi is brief for a case study conducted by only one evaluator, it has to be very detailed where multi-site case studies are concurrently carried out by different evaluators. The modus operandi generally includes:

• Data giving the evaluators a good insight into the context: the evaluation and the case studies' objectives, the logical frameworks for the programmes or the projects to be evaluated, the main implementation data, the methodology and the reasons why case studies have been selected, and the evaluation and the case study timetable (detailed phase by phase).
• Data collection methods: definition of the scope of the study, the period, the people to meet, the data to be collected (analysed by type: documentation, interviews, focus groups, observation, surveys, etc…) before or during the case study fieldwork, and the time to spend on each phase. The methodology should always be based on the chain of evidence techniques that are being used, i.e. based on explicit links between evaluation questions given to the evaluator, those formulated by him, the data he collects, and the evidence and analysis produced to arrive at his conclusions.
• Actual elements of methodology on which the evaluator can rely: guides for interview by type of actor, questionnaires, and field measurements if they are useful, etc.
• Questions which the case study has to answer and possible interpretations, methodologies used to provide the answer, evaluation criteria, and indicators that are sought. In order to ensure that the multi-site case studies contribute to the general synthesis, it is essential that each study responds to an identical series of questions.
• The framework for the case study report, including the presentation of the findings (the template sheet, for example) and the content of the annexes. At this stage, the development of a pilot case study before the other case studies is a great advantage: the pilot will be an example of the size and the content that other evaluators will have to manage in their studies. Also, the synthesis of all the studies' findings will be easier to make.
• The names and the contact details of the main source, who can be asked to supply explanations to the evaluator at any time.

Choosing the teams to implement the case studies

Concurrent with the finalisation of the organisation stage (if it has not already been done in the consultant's answer to the call for tenders), a list of the evaluators in charge of the case studies and their supporting staff (for example, the investigators) should be agreed with the client.
At this stage, two essential requirements should be fulfilled before proceeding with the implementation of these studies:

• The selected people must have the required competence for the various tasks that have been defined. Case studies should always be directed by one of the evaluators belonging to the team (national or international).
• The selected people should speak the language of the country, region or area. Interpretation is always possible but should be limited to situations in which all the other solutions have failed.

Once this list has been established, the consultants should ask their client for a formal notification of mission, describing their intervention objectives and asking the people they are going to meet to assist them as much as they can.
In certain cases, junior assistants may carry out part or the whole of a case study, under the evaluator's supervision and responsibility. It is however more sensible to limit the task to straightforward collection of information (for example, statistics, bibliography, etc.). Interviews with beneficiaries and the authorities in charge of implementation should be conducted by the evaluator, whenever possible.

Scheduling the case studies

Scheduling a case study is a very detailed operation that needs to be carried out well in advance in order to get the required appointments on-site. In country/region assistance, the use of a local representative is almost obligatory in order to:

• find the co-ordinates of the people to meet,
• give them a meeting request form, a copy of the interview guide, the client's notification of mission and, if necessary, basic project or programme documentation,
• organise the interviews and the meetings (reservation of vehicles, local meeting rooms and offices, accommodation, etc…),

give assistance with translations or with the conduct of some parts of the study (for example, surveys) where required

If possible conduct a pilot study

There are many reasons to justify the conducting of a pilot case study before starting a multiple sites case study. Indeed, this good practice allows, inter alia, the:

• validation of the methodology within which the studies are to be conducted, by testing it on-site,
• informed reduction of the list of the people to meet and of the scope of the core bibliography,
• finalisation of the interview guides and the questionnaires,
• review of the questions given to the evaluators, the criteria, and the indicators in respect of what is available on-site,
• writing of a standard report that will be followed by other evaluators. It is often more understandable than a study production guide.

The pilot site may be chosen because of its practicability (accessibility, sufficiency of data or interlocutors' availability). However, as the evaluator who will supervise all the case studies often carries out the pilot case study, it will usually be selected on the basis of its potential contribution to subsequent case studies. These criteria could be:

• a study that identifies a large range of problems and contexts that other studies will have to deal with (those elements will therefore be used as a reference point for other case studies),
• the most complex or difficult study to conduct, regarding logistics. This will ensure that the other case studies are viable.

Train the evaluation team when multiple sites case studies are scheduled

Arranging for one evaluator to implement all the case studies can be very difficult or impracticable, in particular because of delays in completing the studies. As a consequence, case studies are usually conducted by several evaluators, which inevitably risks an "evaluator effect" in the conduct of the case study and in the interpretation of the findings. This means that the evaluator's subjectivity should be controlled as much as possible.
To do so, in addition to the standardisation of methodology and practice described in the previous stage, the evaluator in charge of the case studies should train all field evaluators in order to get their understanding and agreement on the objectives, the methods and what the reports are expected to achieve.
The content of this training consists of a group review of elements of methodology and the reporting of standards, with a view to getting a consensus on the content of the work.

This training should be linked with permanent monitoring, with each evaluator being able to contact the person in charge of the case studies quickly to get information and advice concerning their conduct or reporting. If the studies have to be written in several languages (for example, where regional evaluations include case studies from many countries), this information should also cover the meaning of particular concepts which could be ambiguous or unclear in translation.

STAGE 3: HOW ARE THE DATA COLLECTED?

This stage, which can start before the fieldwork with the bibliographical data collection or preliminary meetings, is crucial, especially when the evaluator goes on-site. To ensure that the case study findings are reliable, a number of fundamental elements should be carefully taken into account:

• The data collection should include adequate longitudinal data, i.e. data covering a sufficiently long period of time in order to avoid taking an anomalous situation as a reference point.
• The data collection should be based on a principle systematically adopted in the evaluation: information verification through triangulation. This means that the data are collected from different sources and combined (for example, interviews, data taken from the implementation and surveys, or many interview sources, such as both programme administrators and programme beneficiaries).
• The evaluator must exercise care in collecting and presenting the needed to support his argumentation. This evidence (similar to the judiciary field) can be:
  • facts that he himself has observed,
  • facts observed by people or institutions above suspicion (for example, public statistics),
  • written documentation whose value in terms of evidence can be relatively variable (for example, meeting reports),
  • the acknowledgement of mistakes made by certain institutions or individuals (for example, during the programme implementation),
  • evidence against or for someone. However, in the context of an evaluation, the veracity of such testimonies should be checked out through triangulation,
  • experts' opinions,
  • other material evidence, such as video tapes, audio cassettes, etc…
• The evaluator must make sure that all the relevant sources are properly used: statistical data relating to the context, implementation data, existing reports, interviews with the appropriate persons taking account of mixed categories of actors (administrators, operators, research and NGO and control staff, beneficiaries…), and surveys.
• The evaluator must ensure that information collected is thoroughly used, and that nothing important has been overlooked. It is essential to obtain as much information as possible, especially when opinions differ among the interviewed people. If this is the case, the evaluator should request their argumentation, examples and evidence. This type of inquiry gives validity to data collected which will be used at the analysis stage, and supports the chain of evidence.
• By definition, the case study is open to any possible discoveries throughout the course of its implementation. Thus, the evaluator must know how to identify key features during the case study implementation and focus on them, even if they were not expected or scheduled in advance. More particularly, the data collection stage should be designed to progress the analysis process in order to test alternative interpretations put forward to explain a situation.
• The reporting of the study findings is a fundamental element that increases the credibility of what has been found. Thus, the evaluator must carefully make notes on all the information he has collected, and especially everything dealing with the context, the relationship between stakeholders, and the reasons for the study's success or failure. To do so, it is important during the interviews to clearly dissociate the information collected (facts that can be checked) from points of view, analyses and opinions, in order to present them properly in the report.
• The field stage leads to first-hand observations of what is occurring. The evaluator must also note them down carefully.

The language in which the case study is to be implemented is also an important point. Thus, the evaluator can be locally recruited or assisted by an interpreter for the whole study or only a part of it (for example, during exchanges with the beneficiaries).

STAGE 4: HOW ARE THE FINDINGS ANALYSED AND INTERPRETED?

This is the most challenging stage of the case study. Its goal is to analyse the data that have been collected during the fieldwork and to link as far as possible the effects of the observed facts to their causes. This analysis is difficult to conduct because it is less structured than at the conception and the collection stages.

The two important analysis patterns

Two main analysis patterns are linked to the commencement of the case study itself:

• If the case study is implemented to check out a hypothesis: the analysis is focused on elements that confirm or invalidate this hypothesis. This type of case study is more often used in the research field than in evaluation.
• If the case study starts with no preconceived ideas, then its implementation will progressively build up the evidence to explain the findings, predictable or otherwise. This constitutes the chain of evidence.

Process and link to other stages

The analysis overlaps the data collection stage, and this is particularly true for case studies in which:

• The data collection stage includes a pre-established hypotheses test that may partly modify the study content during its implementation.
• The study is large enough to allow the evaluator to review and refine his criteria for the next data collection as a result of the initial findings.

If the findings are to be investigated by other tools after the fieldwork stage, with a view to assembling and comparing data (for example, survey analysis, benchmarking, etc.), the analysis must always be based on the construction of a chain of evidence.
The analysis stage must include a number of steps, some being concurrent with the data collection stage:

• The preparation of a database for the case study,
• The process must always be iterative, i.e. based on a search for information sufficient for the construction of a chain of evidence,
• The analysis must test all possible explanations for a situation in order to retain a limited number of probable explanations, and sufficient justification to eliminate others.
• The evaluator must make sure that the analysis findings are reproducible, i.e. make sure another evaluator would come to the same conclusions when confronted with the same information.
• For multiple sites evaluation, a generalisation is one of the main objective to be achieved. This can only be reached if the previous stages are properly implemented, and give the findings a consistency and plausibility to enable the evaluator to draw firm conclusions.

The particular case of the multiple sites case studies

The analysis methods for multiple sites case studies use different types of tools which make the management of various qualitative data easier. Among these techniques are:

• Matrix of categories that places similar elements from different sites (for example, types of programme administration or number of educational programmes implemented in each site, etc.) into a table. In this technique, a coding scheme is established prior to data collection.
• Data tables that emphasise the frequencies of occurrence of an event. These tables are particularly suitable for case studies and derived from surveys conducted during the implementation. They make comparisons between the findings of the surveys from the different areas considered in the study. However, as a case study rarely includes the construction of a statistically valid comparison, other means of data comparison should be found without resorting to classical statistical representations, such as the average, the standard deviation or the variance.
• Graphic data displays with or without computer programmes. When they are being formalised, these techniques require a meeting between the evaluators conducting the case studies to validate or exclude hypotheses on each topic being studied, as illustrated by the graphics. This will enable the evaluators to progress towards their conclusions. The use of video-conferences and emailing between a limited number of evaluators could be an alternative and cost-effective solution to such meetings.
• Time-series analyses that compares, site by site, the chronological events in order to draw conclusions. These series may be a useful technique to explain why timing leads to the success of some programmes, and failure in others.

This list of tools is not exhaustive. Each evaluator must be free to show the findings from several of his case studies by using any tools providing a chain of evidence, the ultimate goal. With the benefit of studies carried out at various sites, the chains of evidence are reinforced when similar effects happen and are identified in different sites at the same time.

STAGE 5: THE REPORTING

This stage is crucial because the reader of the report must get the same insight from the evaluation as the evaluator. Thus, the importance of this section of the case study should not be underestimated. For the presentation, the report must include:

• the objectives of the evaluation and the case studies,
• the selected sites, and the reasons for selection, (for a multi-site case study),
• the collecting and analysis methods which have been used and their limitations,
• basic data for each case study (statistics, implementation data, survey findings, etc…) and the sources of information,
• factual descriptions and, particularly, those relating to implementation that will ease the reader's insight,
• the answers to questions dealing with:
  • comparisons between different sites and the case studies' content, summarised in a readable way,
  • the chains of evidence leading to well-argued findings.
• the building of conclusions designed to lead as far as possible to useful generalisations (mostly for the programme implementation case studies and programme effects case studies).

Ideally, the in-mission report should be distinguished from the final report in order to avoid any bias from empathy. Thus, the author of the final report should not normally be responsible for the production of the in-mission report. However, this can be difficult to achieve in country/region evaluation, where time and budget constraints are significant.

The editing style of case study reports largely depends on the role that the evaluators want them to play. This role should be defined in advance, for example, in a context of:

• a single illustrative case study. This should give a detailed description of the observed event, in order to bring factual elements to the evaluation.
• a multiple sites case study with numerous sites. The summary analysis report will not be as detailed as in the single illustrative case study because it has to be readable. A sites' comparison should be made, and each study report should be included in the annex.

The target group (to whom the case study findings are directed) must also be identified in advance. In country/region evaluation, the target group will mostly be the evaluation users. Report production will therefore be essentially focused on the answers to the evaluation questions, and on the conclusions and recommendations directly linked to programme management.

If the case study is addressed to a general audience, reporting should reflect this by taking a less technical approach to the content. Giving a non-technical summary at the beginning of the report can also be helpful to non-specialists.

What are the pitfalls to avoid?

Point 1: inapropriate design of the study

Within the general design of the study, the evaluator must always be able to answer to the question "How?" and, wholly or partially, "Why?". Its validity must be tested once the design stage begins and must include:

• the questions at stake,
• the spatial and temporal analysis units that have been selected,
• the collection and data analysis process, linked to the questions at stake,
• findings' interpretation criteria and their reporting type.

If one of these criteria is poorly conceived, doubts will be raised about the validity of the whole case study. Thus, checks must be made as to whether:

• The questions are directly linked to the evaluation questions, and will lead to all or part of the answers sought.
• The units selected cover the problems and the context (for example, is the selected period long enough? Does the area studied sufficiently cover the problems and contexts at stake? etc…).
• The methodology deals with all the study sections, from the data collection to the report. Does it lead to the establishment of a chain of evidence? Is the process iterative and is it implemented with a view to validating/excluding alternative interpretations?
• The findings' interpretation criteria are explicit and sufficiently argued?

It is important to bear in mind that in such a difficult field as in country/region evaluation:

• It is a sensible precaution to plan several studies during the case study design because a single study may lead to bias.
• Multiple sites case studies should be designed with the same methodology to make comparisons between sites possible and provide a sound basis for generalisation, unless good reasons otherwise are produced.
• Despite the required rigour in the implementation of case studies, the study design must always include the flexibility to "explore" a new field if the evaluator thinks it justified. When this occurs, the reasoning should be validated with the manager of all the case studies.

Point 2: poorly selected sites

The sheet "preparation for case studies" gives details of the various choices for site selection assisting evaluators to implement case studies.
Therefore, selections must be carefully made and be the result of:

• detailed argumentation in an intermediary report illustrating the choices and proposed selections, directly linked to the evaluation questions (for example, when to choose a bracketing (extreme outcomes) case study or a worst case study. Why choose 5 sites rather than 3 or 7?),
• a process of discussion/validation with the client, including local consultation if possible (with a delegation, host country counterparts, etc…), in order to validate that the proposed sites fit the purpose of the study and do not present any specific problems (for example, sites already subjected to evaluation, dangerous sites, etc…),
• the drafting of the final document, integrating the different stages and stipulating the final selections.

Point 3: insufficient information collection and weak argumentation

Case studies are mostly used because they offer the opportunity for a detailed examination of a situation. That is their main justification. To be decisive, the information collection must be focused. As a consequence, there must be a systematic search for exhaustiveness and quality when planning and carrying out a case study. The collection and the analysis stages detail the points that have to be met in order to reach this objective.

If the evaluation design has been well conducted, there is less chance of insufficient information collection and weak argumentation; however, some precautions may improve the case studies content, such as:

• producing criteria to secure a good data validity (on the selection of information sources, complementarities between them, etc…),
• requiring the lead evaluator devise and enforce quality assurance procedures in order to check the accuracy and plausibility of the multiple sites case studies' findings produced by other evaluators.

Starting the process of production of the report, by drafting appropriate sections during the collection stage, is an excellent way to identify any lack of information. This is because the evaluator's judgement is of the utmost importance.

In any case, an assertion without any evidence is a pitfall to be avoided at all costs in a case study evaluation. The analysis and production stages must always reflect impartiality, which is the only guarantee of the findings' credibility. This is because the only element eventually taken into account is the evaluator's judgement of the evidence. This judgement can be tested, particularly by submitting the case study report to the people from whom data were collected to obtain their views. This leads to:

• checking the accuracy of the data used in the report (for example, eliminating factual errors).
• correcting any subjective assertions.
• limiting any analytical bias.

Point 4: Excessive generalisation

Generalisation from findings is one of the most valuable achievements of an evaluation. But, it can also be the case studies' weakest point if they are badly conducted, especially when case studies are contrasted with tools based on statistical analysis. Thus, being able to generalise from case studies implies bringing enough evidence to be convincing. Excessive generalisation must be avoided. This is also often linked to the following points:

• a bad design (that can make the findings unusable),
• an incorrect site selection (one that does not demonstrate what the evaluator is looking for),
• poor argumentation (insufficient evidence to support the conclusions, let alone to generalise).

Usually, generalisation is possible when case studies have an external validity, for example, as demonstrate by a survey (if the interviewees are representative). This external validity is easier to deduce from several case studies rather than from one. Questions on the external validity may also concern only specific points of the study, and not the whole case study (for example, conclusions about implementation problems can be generalised but not those about the programme impacts).

The pitfall to avoid in terms of generalisation is to "imitate" surveys when implementing case studies. In a case study, generalisation is derived from analysis, not statistics, as with experimentation. Generalisation can only be used if the case studies' selection, and the way they are carried out, supports it. In particular, multiple sites case studies can produce useful, comparative analyses, from which common conclusions are derived. In order to know if a generalisation is valid, it is possible to use the replication technique (as in the scientific field), which tests whether replication is possible on a second site, or a second series of sites, and if a general theory is emerging which can be universally applied.

• Examples

• Case study Evaluation, November 1990. GAO/PEMD-91-10.1.9.
• Case study research. Design and methods. Third Edition. Robert K Yin. Vol 5. 2003. www.sagepublications.com
• Embedded case study Methods. Integrating quantitative and qualitative knowledge. Roland W. Scholz - Olaf Tiedje. 2001. www.sagepublications.com
• Cahiers MEANS Volume 3 "Principales techniques et outils d'évaluation". Commission Européenne. 1999. Office des publications officielles des Communautés Européennes.