Evaluation methodological approach

WHAT DOES A PROBLEM DIAGRAM REPRESENT?

What is the definition of a problem?

In development assistance, projects and programmes have two targets: to satisfy priority needs or achieve a specific goal, and in this view to solve a range of hindering problems. Generally, the success of a policy or strategy depends on the resolution of difficult issues for the partners and potential beneficiaries.

Problems seldom appear separately. Cause-and-effect relationships often link them and constitute a system, which can be presented as a tree-like diagram.

The importance of problems varies. It is thus theoretically possible to identify a core (or central) problem and derive from it a range of causes and effects.

Chart of the problem diagram

When a core problem is identified, the diagram takes the shape of a complete tree, provided with a trunk (the core problem), roots (the causes) and branches (the consequences and impacts).

Standard charts illustrate the core problem in the centre of the diagram, causes at the bottom, and consequences at the top. The diagram is read from the bottom upward.

Standard chart: the complete problem tree

Image

A strategy, programme or project is all the more effective when it studies the fundamental causes of the problem(s) to be solved. Thus, once the core problem is identified, the diagram only represents the causes and is displayed as an inverted tree.

The diagram below illustrates a simple case, where three levels of problems are identified: fundamental causes, intermediary causes and the core problem.

Simplified problem diagram

Image

In the context of the European Commission's development assistance, the problem diagram illustrates the core problem with its immediate and in-depth causes. It may also depict the core problem's main consequences which are addressed by the assistance strategy (defined in the strategy or policy papers and the corresponding programmes).

WHERE DOES THE PROBLEM DIAGRAM COME FROM?

The problem diagram comes from objective-based management whose use expanded during the 1960s in the American economic sphere and in Western economies.

Also called cause-and-effect diagram, it has been developed for a variety of fields, such as medical diagnosis, prevention and research on the causes of accidents, and total quality management.

It is used as a country assistance decision tool. It is linked to the objectives tree in the context of Goal Oriented Project Planning (GOPP, originally called ZOPP in German for Zielorientierte Projektplanung) which derives from the logical framework developed by GTZ at the start of the 1980's.

Used as a problem tree, it is:

• A useful tool for the elaboration and wording of projects
• Part of a planning methodology contributing to the development
  of an objectives tree and a logical framework
• A participative decision tool which embraces the main stakeholders of the project and
  whose determination is the first stage of the project cycle sequence

Following GTZ's initiative, various public donors and operators (agencies and NGOs) have used the problem tree as a component of project planning and management. However, its application as a participatory and ownership tool does not always play the crucial role defined by the ZOPP designers.

WHAT SHAPE CAN IT TAKE?

Tree shape or complex diagram

Complex and simplified problem trees are considered to be standard charts. They are subject to the following basic rules:

• Determination of a single core problem
• Subdivision of the problem of a given row into 2 or more problems of the row below
• Connection of a defined problem with a single problem of the row above
• No horizontal, or feedback links (i.e. effects becoming causes and vice versa)

This simplification of reality is understandable in the context of the objectives-oriented participative planning process. It becomes a handicap in an evaluation because it avoids the complexity of reality and the rational limitations inherent in strategy and planning papers. Thus, the standard chart has been abandoned and the term "tree" replaced by "diagram".

Complex problem diagram

Image

This diagram represents a complex problem system.

• One of the intermediate causes in row (2) is directly linked to the core problem and not to another intermediate cause (1).
• Two intermediate causes in row (3) are interrelated.
• One of these causes is linked to two causes of the row above (intermediate 2).

Examples of more complex problem systems can be found, having two core problems and subdivisions of intermediate causes. In any case, such problem diagrams do not illustrate feedback links.

Vertical or horizontal diagram

Diagrams can either be vertically or horizontally oriented.

In their vertical shape, they read from top to bottom, or from bottom to top. Usually, the core problem is located in the upper part of the diagram.

In their horizontal shape, diagrams read from right to left, or left to right.

Other shapes

Other shapes have been developed, such as the fishbone diagram (also called Ishikawa diagram). Dedicated to the study of the cause resulting from a selected impact, it is also used in quality control process, medical diagnoses and accident prevention.

WHAT IS A LOGICAL LINK?

Definition

Problems and causes illustrated in the diagram are related to each other with horizontal or vertical links. These links are called "logical" when expressing an inference relationship (induction or deduction) which has been validated by experience. They highlight the fact that:

• The introduction of a problem or cause at a certain row implies that of more deep-rooted problems or causes
• A problem (cause) of a certain row can be deduced from a problem (cause) of an upper row
• Two problems (causes) of the same row share a synergic relationship

The logic of the links between problems (causes) is essentially based on experience and development theories (originating from this experience). Experiments and theories usually provide good indicators for the determination of subordinate problems (or causes) in which a particular problem (or cause) in a certain row has its origin.

However, in terms of development, experiments do not always result in the same conclusions and, as a consequence, should not be considered universal. This can be explained by the dependency of the problems on various factors, which are often not well-known. Even when experts agree on the relevance of a core problem, they may disagree on the definition of intermediate and in-depth causes. The content of core problems, such as poverty alleviation, is continuously debated and evolving.

Verification of the diagram's logic

The logical links between problems (i.e. the problem diagram's logic) can be simultaneously checked with:

• Recognised experts who have worked in different countries and situations
• Designers and implementing managers of programmes included in the evaluation scope

These two groups of actors can validate the diagram's logic in whole or in part. Conversely, a link which is contradicted by the experience of the experts and operators can be classed as illogical. The borderline between the logical and illogical nature of a situation is determined when there is a doubt, and contradictory or ambiguous experiments.

WHEN SHOULD A PROBLEM DIAGRAM BE USED?

Evaluation of standard projects and programmes

As a programme aims at solving a range of problems affecting potential beneficiaries (individuals or groups of beneficiaries), the evaluation should be concerned with the validity of its analysis. When a logical framework (ZOPP, Project Cycle Management) supports the establishment of the programme, its drafting usually includes the analysis of the problems. The evaluators should therefore check:

• The validity of the procedure: how have the problems been identified and classified?
• The apparent coherence of the problems' linkage: are their causal links relevant?

The assessment of the analysis of the problem and its corresponding diagram is a means to test the validity of the objectives system and its corresponding diagram, and to draw conclusions from them.

If the programme does not have a logical framework, at least the problems to be addressed should have been analysed. In this context, it may be useful to develop a rational diagram based on a study of the definition of the programme and programming documentation.

Evaluation of complex strategies and programmes

In complex strategies and programmes, the evaluation usually deals with a range of activities (projects and programmes) which lack explicit justification within a logical framework. The evaluation team may not be provided with an explicit and logically structured presentation of the objectives targeted by the donor, nor with thorough analysis of the difficulties which are supposed to be solved as a result of adopting such objectives.

The available documentation is not always clear about how the various types of problems have been taken into account. The evaluation team must therefore reflect upon questions such as:

• Are the problems belonging to the context?
• Or, are they selected because of the strategy's main objectives?
• Why have the reasons for the problem's selection not been clearly stated?

In addition, the methodology which was used to analyse the situation and select the problems should be presented, which is not always done prior to the evaluation. Answers to the following questions should be considered as part of the evaluation:

• Does this analysis result from documentary research?
• Or, does it result from the conclusions of workshops and seminars?
• Has a single author drafted it?
• Or, has it been the result of a collective work?
• Is/are the author(s) of the analysis also responsible for the drafting of the strategic orientations and the intervention programme?

Evaluation of the analysis of the initial situation

Programmes and strategies depend on the analysis of the situation. This analysis presents the primary data relative to the problems addressed by the programmes and strategies, and the information about the context (economic, social, cultural, institutional, environmental) in which they will be implemented. This crucial analysis identifies:

• The problems to be addressed by the activities in order of priority
• The positive elements on which activities can rely
• The actors to be involved in the activities
• The structural and dynamic constraints which may hinder the success of the activities

Therefore, the evaluation team can be asked to assess the quality of the analysis, and the conformity between the analysis of the situation and the adopted strategy (or programme). The problem diagram, as an ex post construction, can be one of the tools used to check the coherence and the relevance of the analysis in respect of the main contextual problems.

The evaluation team should not underestimate the amount of work, nor the level and variety of competences required (Human resources and working arrangement) to construct the problem diagram. Indeed, the evaluation team should not simply present a diagram with the problems developed in the documentation, but also reconstruct the problem diagram which should have been prepared during the development of the objectives diagram (and therefore, during the drafting of the strategy)

PROBLEM OR OBJECTIVE: WHICH COMES FIRST?

In project cycle

In the planning process and project management field (ZOPP, PCM), specialists place the preparation of the problem diagram before the objectives diagram. The typical sequence can be pictured as followed:

Analysis of the problems in a simplified sequence of the project cycle

Image

Problems analysis may be undertaken before stakeholder analysis, but should always precede the objectives analysis. Indeed, in Project Cycle Management (PCM), objectives are always deduced from the problem analysis.

In the strategy policy co-ordination cycle

If the definition of the donor's activities in bilateral co-operation does not systematically result from the analysis of the problems, it will target one or more overall objectives (strategic and philosophical) which are explicitly or implicitly mentioned in the documentation. The donor (alone or in partnership) defines the main fields of the development assistance in accordance with these objectives.

The chart below illustrates the logical position of the problem analysis within the strategy policy co-ordination cycle and, particularly, in relation with the definition of the objectives.

Analysis of the problems in a simplified sequence of the strategy policy co-ordination cycle

Image

The problem analysis and the resulting diagram derive from the definition of the intervention scope, and thus from the overall objectives of the donor (the European Commission). They also form the basis for the definition of specific objectives of the development assistance, which is to be provided during the period covered by the strategy.

WHAT ARE ITS ADVANTAGES AND LIMITATIONS?

Its advantages

Presentation of the problems

The diagram presents the various problems and their relationship with the core problem through a system of rows. It displays the causal logical links between them or, conversely, the poor logic of these links.

Main problems and contextual problems

In the analysis of the situation, the diagram distinguishes the problems relating to the activity's context from the problems to be solved by the strategy and the planning. As a consequence, its construction requires the highlighting of priorities of the development assistance, and explains why certain problems are considered as important features of the strategy while others are not.

Definition of the objectives

The problem diagram enables the evaluator to:

• Reveal the implicit objectives of the strategy (or the programme)
• Check the validity of the objectives presented in the strategy and programming documentation

It contributes to the organisation of the evaluation around a crucial question, which should be systematically answered:

To what extent have the objectives been achieved?

Its limitations

The diagram rationally organises the main difficulties on the basis of the analysis of the situation. It may entail the following limitations.

Access to information

The lack of data or difficult access to information may lower the quality of the analysis of the situation. The evaluator should therefore reflect upon the subsequent questions:

• Is the analysis sufficiently comprehensive and does it cover the main aspects of the situation?
• Is it topical, i.e. based on sufficiently updated data (which documents should be referred to?)?
• Is it dynamic, i.e. does it take into account the observable trends and deep-rooted changes in the society?

Quality of the analysis

The methodology for the analysis does not guarantee high quality of the data, for neither the methodology, nor the sources of information are usually mentioned in the strategic, political and programming papers. The evaluator must therefore systematically enlarge the assessment to include the sources of the analysis displayed in these documents. The methodology used, the nature of the main sources of information and the identity of the authors should also be noted.

Main problem / contextual problem

The determination of the priorities for each problem results in their classification into two categories: contextual problems and main problems. Two questions must be addressed:

• Is the determination of these priorities explained?
• Does the available data enable the evaluator to assess the relevance of such an explanation?

Selection of the main problem

The selection of the main problem, which is crucial for the construction of the diagram, is particularly challenging when the objectives of the activities are general and the whole range of the country's problems (or the region's) are to be considered. The documentation may show two main problems which lack links between them, or may appear insufficient for the determination of a single core problem.

Tree-like illustration

The standard setting of the challenges into a tree diagram illustrates a straightforward classification (which types of problem diagrams can be used?), which does not always highlight the complexity of the situation and the interactions between issues. Indeed, the construction of diagrams depends on graphical conventions, such as:

• There should be no illustration of interactions between problems of the same row
• There should be no illustration of feedback links (effects become causes and vice versa)
• Several boxes can illustrate several problems' single cause

More sophisticated representations should therefore be tried.

Knowledge of the situation in the country or the region

The evaluation team may lack a sufficient understanding of the country or the region under consideration to assess the relevance of the analysis undertaken, the determination and the logical setting of the main problem.

WITH WHICH TOOLS CAN IT BE COMBINED?

Objectives diagram

The problem diagram is usually associated with the objectives analysis (and as a consequence, with the objectives diagram) and the development of the logical framework.

The problem diagram is used for the construction of the objectives diagram, often through a simple transposition.

Decision diagram

The analysis of the programme's development process (decision diagram) can be used to explain the reasons underpinning the determination of the main problems and the core problem.

In intermediary evaluations of programmes, the evaluator may be asked to construct a decision and an objectives diagram, in addition to the problem diagram itself.

Effect diagram

In an ex post evaluation or in an evaluation with ex post components, the initial analysis of the situation and the selection of the main problems can be compared with the analysis of the situation undertaken at the end of the period assessed. They can also be compared to an effect diagram resulting from the strategy's implementation, in order to assess the contribution of the development assistance to the resolution of problems.

Other analytical tools

In the context of an ex post reconstitution of the events, the problem diagram may be supported by other analytical tools, such as:

• SWOT analysis
• Socio-cultural analysis

WHAT ARE THE PRECONDITIONS FOR ITS USE?

Available information

When the problem diagram is found in the basic documentation of the programme, or when it can be directly deduced from a logical framework, it must be presented with a study of the choices underpinning its construction. This requires the consultation of the documentation which has supported its construction, and the interviewing of the actors directly involved in its development.

When key actors and documentation are not available

Subsequent to the consultation with specialists, if required, the evaluator may present conclusions about the relevance of the problems system (illustrated in the diagram).

When the diagram has been developed on the basis of the documentation at the disposal of the evaluator, the latter should:

• check the validity of the documentation, and enlarge the scope of the analysis to additional sources, if needed
• test the validity of the hierarchical links and the problems' interrelationship with documentary research and/or interviews

If neither the archives documentation, nor the key actors are available, the evaluator must explain the hypotheses at the basis of the diagram's construction, by providing the reader with precise sources (quotations and comprehensive references of the documentation) underpinning the hypotheses.

STAGE 1: HOW TO IDENTIFY THE PROBLEMS

Record the references to the problems found in the documentation

Quotations from the evaluation's baseline documents are used to identify problems which may not be systematically depicted as such in the texts.

Sometimes, problems are identified as assistance objectives or impacts targeted by these objectives.

Problems are thus expressed as:

• problems (presented as such)
• objectives, whose goal is to resolve explicitly or implicitly a problem
• expected impacts of the CSP assistance

What is to be recorded in the documentation?

Image

Image

The preliminary analysis of quotations can reveal:

• Repetitions, partly due to the numerous presentations of the situation (executive summary, strategy, programme, annex)
• Imprecise formulation (for example, generic "economic and social development" which could be more specific)

A thorough selection should therefore be undertaken.

Distinguish context problems from main problems

Among the various problems, the documentation often identifies problems which should be solved by the intervention. The evaluation team should therefore identify the problems related to the intervention in the analysis of the situation, in order to incorporate them into the diagram. The remaining problems should be considered part of the context.

The problems directly targeted by the intervention may not be explicitly identified. Main problems and context problems can be intermingled, which complicates the construction of the diagram. The evaluator will only be able to differentiate the two types of problems after the completion of the diagram.

Whatever the situation, the evaluator should present the list of all the problems under consideration in the annexes, split into two categories:

• Context of the intervention
• Object of the intervention

Identify the core problem

The crucial stage of the process is the identification of the core problem among the variety of the selected problems.

Three situations can be encountered.

The core problem is mentioned in the documentation
It is usually a wide and consensus problem, such as structural poverty, rebuilding after a conflict or a natural disaster.

The core problem is not clearly mentioned, whereas the overall objective is
In this case, the evaluator can deduce the core problem from the high-order objective.

These two situations do not need further developments: the evaluator records the core problem explicitly or implicitly mentioned in the documentation, whatever his/her opinion about the problem selected.

Neither the core problem, nor the overall objective are explicitly mentioned
The evaluator plays the role of the planning manager, and decides which of the problems will be the core problem. He/She may:

• Take the decision, on the basis of his/her rationale
• Benefit from the assistance of specialists during interviews or working sessions

Whatever the procedure, the selection of the core problem should be conducted concurrently with the classification of the problems into levels. Indeed, the selection of the core problem should be supported and justified by the coherence of the whole diagram.

Proceed to a first classification of the problems by rows

Several situations must be considered:

• The row is explicit
• The cause-and-effect link between two problems is explicit, which eases the location of one in relation to another
• No indication is given about the problem's row (per se or relative). The members of evaluation team must draw their own conclusions and/or ask the assistance of an expert specialised in the fields where they lack the required competence.

Each of the problem's classification by row must be justified (with interpretations presented as explicit assumptions, if needed).

One of the main difficulties in the graphical illustration of problems is to distinguish between "short-cuts" and inconsistencies.

In a "short-cut", the succession of causal links is pictured as a causal link between the first and the last problem of the causal chain.

For example, the chain illustrating the impact of inadequacies of the road infrastructure on poverty can be illustrated as follows:

Image

In the same document, quotations can be illustrated as follows:

Image

Or:

Image

This example does not portray inconsistencies but stylistic short-cuts.

STAGE 2: HOW TO CONSTRUCT A PROBLEM DIAGRAM

Establish a temporary diagram

The diagram may intend to faithfully reflect the problems system as stated in the strategic and planning documents with missing elements and inconsistencies. If no faithfully constructed diagram is explicitly requested, the evaluators will consider it as a step to the construction of a logically reconstructed problem diagram.

Stages for the drafting of temporary diagram

Image

Should the diagram's construction start with the core problem or the in-depth causes?

Usage highlights the fact that these two types of problems/causes are the easiest to identify, whereas intermediary causes are the hardest to determine and classify. Thus, it is recommended that the development of the diagram starts with its extremities at the same time.

Identify the authors who have analysed the situation

Before testing the temporary diagram, the evaluator should identify the main and secondary authors who have contributed to the analysis of the situation (writing of a chapter, sector-based contributions, participation in working sessions, etc.).
It may be useful to check the impact of this analysis on the definition of the strategy and programme operational objectives.

Test the temporary diagram

Where possible, the authors of the documentation referred to above should test the diagram in order to validate the classification of the problems by rows and links. The aim is to check that the evaluator's interpretation reflects the authors' intention correctly. If the authors are not available or, in order to complement their original contribution, the evaluator should consider asking for the participation of other actors responsible for the drafting process.

Contacting the authors is usually possible when the documentation is recent and the authors are still in their position or contactable in one of the services. This task is challenging when the documentation is old and its authors are not easily identifiable or contactable.

Develop the final version of the problem diagram

The final version takes into account the opinions collected during the testing of the temporary diagram.

It is an accurate account of the initial intentions logically reconstructed of the European Commission, taken from official documentation.

HOW MANY DIAGRAMS MUST BE DEVELOPED?

A single diagram

If the analysis of the situation does not radically evolve during the evaluation and if the activities focus on a single core problem, one diagram is enough.

Several diagrams

• When the country situation (the country's own or its strategic context) evolves during the evaluation period, and the analyses clearly reflect these changes.
• When the evaluation team cannot identify a single core problem and when various core problems result from a fairly independent causes system, it is appropriate to establish as many diagrams as core problems. All the verification procedures must be used to support the various diagrams, and to prove the existence of a range of overall objectives.
• If the information is sufficient, a diagram can be established for each of the priorities or sectors of concern of the assistance.

HOW SHOULD PROBLEM DIAGRAMS BE USED?

When it is impossible to directly establish objectives diagrams, problem diagrams play an essential role in the organisation stage of the evaluation.

Conversion into an objectives diagram

Problem diagrams present a summarised vision of the situation to be improved, partially at least, by the strategy. The classified objectives included in the strategy should be deduced from the problems diagrams. 

Planning stage

During the planning stage (of a project or a programme), the conversion of the problem diagram into an objectives diagram is not automatic, because each problem integrated into the diagram does not have the same priority.

Evaluation stage

During the evaluation stage, the reconstruction of the problem diagram includes a step which differentiates between context problems and intervention problems. As a consequence, the diagram resulting from this selection should be completely convertible into a logically reconstructed objectives diagram, i.e. the higher-order overall objective corresponds to the core problem, and each row of subordinated objectives to its equivalent row of problems.

Conversion of the problem diagram into an objectives diagram

Extract from 'A summary of the theory behind the LFA method', The Logical Framework Approach (LFA), SIDA - Swedish International Development Co-operation Agency, Draft, June 2002.

Image

The objectives diagram displays the classification of objectives, from the higher-order objective to the projects and programmes that are designed to achieve the overall objective. The diagram can provoke comments about:

• Its internal coherence
• The relevance of the objectives, regarding the overall context (i.e. the overall strategy of the European Union in terms of country assistance, the situation of eligible countries and the objectives of major actors, the intervention scope and the objectives of other donors)
• The appropriateness of the means allocated to achieve these objectives

These comments must be presented in the reports and notes of the evaluation organisation stage. They must also appear as an annex in the final report.

Definition of the themes for evaluation questions

The previous analyses raise questions about:

• The consistency of the strategy with the European Union's overall objectives of its country assistance
• The relevance of the strategy to the situation of eligible countries and the policies of the major actors (governments in particular)
• The need for complementarities and synergies with other donors
• The consequences of a lack of coherence in the objectives system
• The appropriateness of the means to the objectives

These questions lead to the determination of a range of themes which could be investigated during the subsequent stages of the evaluation, particularly through evaluation questions.

However, evaluation questions cannot be automatically deduced from these analyses. Other questions may emerge during the organisation stage, formulated by the strategy implementing operators. Such questions can relate to the difficulties encountered during the planning and implementation stages. However, the number of evaluation questions is limited and additional questions should be the subject of a selection process.

HOW ARE THE FINDINGS PRESENTED?

Notes and reports of the organisation stage

Problem diagrams are established during the organisation stage, where the reports and notes should be provided. At this stage, the construction of the diagram must be precisely described.

The precise sources identifying the problems (exact quotations, references to original documents) must be provided. References to specific documents, interviews, and expertise must support the location of the problems in the diagram, and the assumptions developed during the diagram's construction must be explained.

Final report

The problem diagram should be incorporated into the final report because it is not only a stage in the development of the evaluation, but also an illustration of a possible analysis of the evaluation's strategy.

The report itself may include a short presentation of the diagrams, in addition to its development in the annexes.

Verbal presentation

The evaluation team will need to present its work (methodology and findings) to different groups of people (the evaluation reference group, participants in the debriefing session, etc.). The problem diagram is a very efficient tool for these situations, providing that it is readable without being over-simplistic.

To do so, a main diagram and several sub-diagrams developing fundamental sections of the latter should be presented, each of them not exceeding more than 20 items.

WHAT ARE THE PRECONDITIONS FOR THE DEVELOPMENT OF THE PROBLEM DIAGRAM?

Human resources and working arrangements

The need for human resources varies with the tasks to be achieved and the situation encountered: reconstruction of the problem diagram with the analysis of the situation displayed in the documentation; retrospective reconstruction of the problem diagram as it should have been established to support the relevance of the strategy or programme under assessment.

The right column of the table below illustrates a wide range of working days.

Type of work required for the design of a problem diagram (usual situation)

Travelling expenses

Strategy papers are drafted under the responsibility of the head office (DG RELEX or DG DEV), which is also responsible for the planning stage. EuropeAid is in charge of project drafting, which explains why the majority of the useful documentation (including the complete baseline documentation) is in Brussels. Some tasks may be devolved to Delegations in the future, whose responsibilities in this area will grow.

In a country evaluation context, the retrospective construction of the diagram requires at least 2 or 3 experts working on a mission of 5 to 10 days in the country under consideration.

Computer devices

Most of the graphic problems can be solved with software like MS PowerPoint.

The evaluator can also benefit from specific software designed to assist decision-making, such as MS Visio, TeamUp-PCM for Windows, and Palisade Precision Tree.

WHEN SHOULD IT BE USED?

Donors must explicitly or implicitly explain the decision to provide assistance to a country or a region in a given timeframe. For this purpose, they must determine the objectives of the assistance designed to solve a range of problems (the objectives can be the donor's or the beneficiary's).

The purpose of an evaluation is not to analyse the problems and establish the related diagram, but rather to reconstitute the analysis which has already been undertaken and convert it into a diagram.

Consequently, in country evaluations, the problem diagram is particularly appropriate for the following two situations.

To examine the explicit problems included in the documentation

The conversion of the analysis into a diagram (Stage 2: how to construct the problem diagram) enables the evaluator to check the coherence of the analysis and its relevance to the objectives. For this purpose, a problem and an objectives diagram should be reconstructed (how should problem diagrams be used?) and be compared.

To analyse the non-explicit problems in the documentation

A study of the analysis of the situation facilitates the establishment of a problem diagram, whose identified problems should be solved by the assistance. This stage should ease the reconstitution of an objective diagram, or as a minimum, validate the draft of the diagram established in accordance with the documentation.

WHICH PROBLEMS MUST BE TAKEN INTO ACCOUNT?

In a bilateral assistance strategy, the European Commission is the donor, and the States and national public institutions are usually its partners. The European Commission and the States have their own overall objectives, corresponding to their own issues. Yet, to a certain extent and degree of detail, the problems become common.

The donor's range of problems

This type of problem corresponds to strategic issues meant to be resolved with financial assistance (for example, regional instability, weak absorption capacity of the local market, uncontrolled immigration).

The country beneficiary's range of problems

This range of problems is related to major political issues to which the States are confronted (macro-economic unbalances, social instability, weak growth, social or geographical disparities, etc).

The range of problems common to the donor and the country beneficiary

This type of problem has common origins which are relevant for both the donor and the country beneficiary, and are relative to the fields of the assistance scope (for example, health, SME, irrigation and environment).

Except for participatory evaluations, the evaluation team will prioritise the first and third type of problems, because they are directly relevant to the donor.

WHICH DOCUMENTATION SHOULD BE REFERRED TO?

Where can documents analysing problems be found?

Documents related to the European Commission strategy
Country Strategy Papers (CSP) do not systematically describe long-term problems, whose partial solution is to be found through the implementation of the European Commission's strategy. These problems are sometimes described in the presentation of the strategy objectives, or can be found in the European Council's political papers expressing the institution's interest in the country situation.

Documents related to development assistance programmes
The problems analysed by development assistance projects and programmes constitute the core of the chapter "Analysis of the situation" in the CSPs. This chapter, however, does not always distinguish the problems included in the context of the projects and programmes from the problems at the origin of such projects and programmes.

Thus, the construction of a problem diagram requires the evaluator to distinguish context problems from main problems.

Which documents should be selected?

Successive versions of the analysis of the situation can seldom be found (in strategy or programme papers). After ensuring that previous drafts do not yield more in-depth and valid analyses, the evaluation team mainly works on the final version.

The baseline documentation comprises the (CSP) and the corresponding National Indicative Paper (NIP), covering the timeframe of the evaluation scope. They are usually provided to the evaluation team at the start of the mission.

Often, the evaluation period (defined in the Terms of Reference) is covered by several strategy and programming papers, each of them including analyses and information about the situation of the country, specific sectors or regions.

Usually, the situation and problems are analysed at the start of the period covered by the strategy and/or the programme under assessment. Yet, the analysis may be affected by difficulties concerning the information, such as:

• Late publication of the data
• Difficult access to information
• Poor or uncertain quality of the data

These difficulties, which may even weaken the relevance of the objectives, should not be underestimated by the evaluation team.

WHICH TYPES OF PROBLEM DIAGRAMS CAN BE USED?

Complete problem diagram

Used in participative planning, the complete problem diagram sketches on each side of the core problem the problems identified by the stakeholders.

Simplified problem diagram

The evaluation only focuses on the lower part of the diagram, because the projects usually deal with the causes of the problems.

Complex problem diagram

Usually, the evaluation team limits its task to the reconstruction of a simple tree shape diagram (simplified or complex, depending on the case) which is derived from the identification of the core problem.

Network diagram

This type of diagram is faithful to the complexity of an actual situation. Yet, its reconstitution requires a wealth of information which is seldom available. Furthermore, the conversion of such information into a diagram represents a challenging task.

• Examples

The logical framework and the objectives tree

• Manuel de Gestion du cycle du projet, Commission Européenne, EuropeAid, mars 2001. [FR]
• Guide récapitulatif des Formations - Gestion du Cycle de Projet, Commission Européenne, EuropeAid, Itad Ltd, Particip GmbH. version 1.1, février 2001. [FR]
• 'Logical framework analysis', BOND Guidance Notes No 4, March 2003.
• 'The Logical Framework Approach (LFA)', A summary of the theory behind the LFA method, Sida - Swedish International Development Co-operation Agency, Draft, June 2002.
• 'The Logical Framework Approach', Handbook for Objectives-oriented Planning, NORAD, 4th edition, January 1999.
• 'The logical framework approach', Australian agency for international aid (AusAid), AusGuide, AusGuidelines, 2000.
• 'Logical frameworks: a critical assessment', Working paper # 264, Des Gasper, Institute of Social Studies, the Hague, December 1997.

Use of the logical framework in evaluations:

• 'Using the Engendered Logframe for Monitoring and Evaluation', Helen Hambly Odame, ISNAR, August 2001.
• 'Project appraisal 2 (4)', Logical Framework Approach to the Monitoring and Evaluation of Agricultural and Rural Development Projects, Coleman G., 1987, p. 251-259.

The other approaches:

• Program Theory in Evaluation: Challenges and Opportunities, New Directions for Evaluation, a publication of the American Evaluation Association, Fall 2000.