Evaluation methodological approach

The analysis of problems is a means to test the validity of the objectives of a project, a programme or a strategy. As a programme aims at solving a range of problems, the evaluation should be concerned with the validity of its analysis. The evaluators should therefore check:

• The validity of the procedure: how have the problems been identified and classified?
• The apparent coherence of the problems' linkage: are their causal links relevant?

Programmes and strategies depend on the analysis of the situation. This analysis presents the primary data relative to the problems addressed by the programmes and strategies, and the information about the context (economic, social, cultural, institutional, environmental) in which they will be implemented.

This crucial analysis identifies:

• The problems to be addressed by the activities in order of priority
• The positive elements on which activities can rely
• The actors to be involved in the activities
• The structural and dynamic constraints which may hinder the success of the activities

Therefore, the evaluation team can be asked to assess the quality of the analysis, and the conformity between the analysis of the situation and the adopted strategy (or programme). The problem diagram, as an ex post construction, can be one of the tools used to check the coherence and the relevance of the analysis in respect of the main contextual problems.

When it is impossible to directly establish objectives diagrams, problem diagrams play an essential role in the organisation stage of the evaluation. Problem diagrams present a summarised vision of the situation to be improved, partially at least, by the strategy. The classified objectives included in the strategy should be deduced from the problems diagrams.

The reconstruction of the problem diagram includes a step which differentiates between context problems and intervention problems. As a consequence, the diagram resulting from this selection should be completely convertible into a logically reconstructed objectives diagram, i.e. the higher-order overall objective corresponds to the core problem, and each row of subordinated objectives to its equivalent row of problems.

Stage 1: How to identify the problems?

Record the references to the problems found in the documentation: quotations from the evaluation's baseline documents are used to identify problems which may not be systematically depicted as such in the texts. Sometimes, problems are identified as assistance objectives or impacts targeted by these objectives. Problems are thus expressed as:

• problems (presented as such)
• objectives, whose goal is to resolve explicitly or implicitly a problem
• expected impacts of the CSP assistance

The problems directly targeted by the intervention may not be explicitly identified. Main problems and context problems can be intermingled, which complicates the construction of the diagram. The evaluator will only be able to differentiate the two types of problems after the completion of the diagram. 

The crucial stage of the process is the identification of the core problem among the variety of the selected problems. Three situations can be encountered:

• The core problem is mentioned in the documentation
• The core problem is not clearly mentioned, whereas the overall objective is. In this case, the evaluator can deduce the core problem from the higher-order objective.
• Neither the core problem, nor the overall objective are explicitly mentioned

In the last situation, the evaluator plays the role of the planning manager, and decides which of the problems will be the core problem. He/She may:

• Take the decision, on the basis of his/her rationale
• Benefit from the assistance of specialists during interviews or working sessions

The selection of the core problem should be conducted concurrently with the classification of the problems into levels. Indeed, the selection of the core problem should be supported and justified by the coherence of the whole diagram.

Stage 2: How to construct a problem diagram?

Usage highlights the fact that these two types of problems/causes are the easiest to identify, whereas intermediary causes are the hardest to determine and classify. Thus, it is recommended that the development of the diagram starts with its extremities at the same time.

Test the temporary diagram

Where possible, the authors of the documentation referred to above should test the diagram in order to validate the classification of the problems by rows and links. The aim is to check that the evaluator's interpretation reflects the authors' intention correctly. If the authors are not available or, in order to complement their original contribution, the evaluator should consider asking for the participation of other actors responsible for the drafting process. 

Contacting the authors is usually possible when the documentation is recent and the authors are still in their position or contactable in one of the services. This task is challenging when the documentation is old and its authors are not easily identifiable or contactable.

Develop the final version of the problem diagram

The final version takes into account the opinions collected during the testing of the temporary diagram. It is an accurate account of the initial intentions logically reconstructed of the European Commission, taken from official documentation.

The need for human resources varies with the tasks to be achieved and the situation encountered: reconstruction of the problem diagram with the analysis of the situation displayed in the documentation; retrospective reconstruction of the problem diagram as it should have been established to support the relevance of the strategy or programme under assessment. 

Most of the graphic problems can be solved with software like MS PowerPoint. 

The evaluator can also benefit from specific software designed to assist decision-making, such as MS Visio, TeamUp-PCM for Windows, and Palisade PrecisionTree.