This section is structured as follows:
- What is a problem diagram?
- Why and when?
- How should the problem diagram be constructed?
- In country evaluation
This section is structured as follows:
WHAT IS A PROBLEM DIAGRAM?
WHAT DOES A PROBLEM DIAGRAM REPRESENT?
What is the definition of a problem?
In development assistance, projects and programmes have two targets: to satisfy priority needs or achieve a specific goal, and in this view to solve a range of hindering problems. Generally, the success of a policy or strategy depends on the resolution of difficult issues for the partners and potential beneficiaries.
Problems seldom appear separately. Cause-and-effect relationships often link them and constitute a system, which can be presented as a tree-like diagram.
The importance of problems varies. It is thus theoretically possible to identify a core (or central) problem and derive from it a range of causes and effects.
Chart of the problem diagram
When a core problem is identified, the diagram takes the shape of a complete tree, provided with a trunk (the core problem), roots (the causes) and branches (the consequences and impacts).
Standard charts illustrate the core problem in the centre of the diagram, causes at the bottom, and consequences at the top. The diagram is read from the bottom upward.
Standard chart: the complete problem tree
A strategy, programme or project is all the more effective when it studies the fundamental causes of the problem(s) to be solved. Thus, once the core problem is identified, the diagram only represents the causes and is displayed as an inverted tree.
The diagram below illustrates a simple case, where three levels of problems are identified: fundamental causes, intermediary causes and the core problem.
Simplified problem diagram
In the context of the European Commission's development assistance, the problem diagram illustrates the core problem with its immediate and in-depth causes. It may also depict the core problem's main consequences which are addressed by the assistance strategy (defined in the strategy or policy papers and the corresponding programmes).
WHERE DOES THE PROBLEM DIAGRAM COME FROM?
The problem diagram comes from objective-based management whose use expanded during the 1960s in the American economic sphere and in Western economies.
Also called cause-and-effect diagram, it has been developed for a variety of fields, such as medical diagnosis, prevention and research on the causes of accidents, and total quality management.
It is used as a country assistance decision tool. It is linked to the objectives tree in the context of Goal Oriented Project Planning (GOPP, originally called ZOPP in German for Zielorientierte Projektplanung) which derives from the logical framework developed by GTZ at the start of the 1980's.
Used as a problem tree, it is:
Following GTZ's initiative, various public donors and operators (agencies and NGOs) have used the problem tree as a component of project planning and management. However, its application as a participatory and ownership tool does not always play the crucial role defined by the ZOPP designers.
WHAT SHAPE CAN IT TAKE?
Tree shape or complex diagram
Complex and simplified problem trees are considered to be standard charts. They are subject to the following basic rules:
This simplification of reality is understandable in the context of the objectives-oriented participative planning process. It becomes a handicap in an evaluation because it avoids the complexity of reality and the rational limitations inherent in strategy and planning papers. Thus, the standard chart has been abandoned and the term "tree" replaced by "diagram".
Complex problem diagram
This diagram represents a complex problem system.
Examples of more complex problem systems can be found, having two core problems and subdivisions of intermediate causes. In any case, such problem diagrams do not illustrate feedback links.
Vertical or horizontal diagram
Diagrams can either be vertically or horizontally oriented.
In their vertical shape, they read from top to bottom, or from bottom to top. Usually, the core problem is located in the upper part of the diagram.
In their horizontal shape, diagrams read from right to left, or left to right.
Other shapes have been developed, such as the fishbone diagram (also called Ishikawa diagram). Dedicated to the study of the cause resulting from a selected impact, it is also used in quality control process, medical diagnoses and accident prevention.
WHAT IS A LOGICAL LINK?
Problems and causes illustrated in the diagram are related to each other with horizontal or vertical links. These links are called "logical" when expressing an inference relationship (induction or deduction) which has been validated by experience. They highlight the fact that:
The logic of the links between problems (causes) is essentially based on experience and development theories (originating from this experience). Experiments and theories usually provide good indicators for the determination of subordinate problems (or causes) in which a particular problem (or cause) in a certain row has its origin.
However, in terms of development, experiments do not always result in the same conclusions and, as a consequence, should not be considered universal. This can be explained by the dependency of the problems on various factors, which are often not well-known. Even when experts agree on the relevance of a core problem, they may disagree on the definition of intermediate and in-depth causes. The content of core problems, such as poverty alleviation, is continuously debated and evolving.
Verification of the diagram's logic
The logical links between problems (i.e. the problem diagram's logic) can be simultaneously checked with:
These two groups of actors can validate the diagram's logic in whole or in part. Conversely, a link which is contradicted by the experience of the experts and operators can be classed as illogical. The borderline between the logical and illogical nature of a situation is determined when there is a doubt, and contradictory or ambiguous experiments.
WHY AND WHEN?
WHEN SHOULD A PROBLEM DIAGRAM BE USED?
Evaluation of standard projects and programmes
As a programme aims at solving a range of problems affecting potential beneficiaries (individuals or groups of beneficiaries), the evaluation should be concerned with the validity of its analysis. When a logical framework (ZOPP, Project Cycle Management) supports the establishment of the programme, its drafting usually includes the analysis of the problems. The evaluators should therefore check:
The assessment of the analysis of the problem and its corresponding diagram is a means to test the validity of the objectives system and its corresponding diagram, and to draw conclusions from them.
If the programme does not have a logical framework, at least the problems to be addressed should have been analysed. In this context, it may be useful to develop a rational diagram based on a study of the definition of the programme and programming documentation.
Evaluation of complex strategies and programmes
In complex strategies and programmes, the evaluation usually deals with a range of activities (projects and programmes) which lack explicit justification within a logical framework. The evaluation team may not be provided with an explicit and logically structured presentation of the objectives targeted by the donor, nor with thorough analysis of the difficulties which are supposed to be solved as a result of adopting such objectives.
The available documentation is not always clear about how the various types of problems have been taken into account. The evaluation team must therefore reflect upon questions such as:
In addition, the methodology which was used to analyse the situation and select the problems should be presented, which is not always done prior to the evaluation. Answers to the following questions should be considered as part of the evaluation:
Evaluation of the analysis of the initial situation
Programmes and strategies depend on the analysis of the situation. This analysis presents the primary data relative to the problems addressed by the programmes and strategies, and the information about the context (economic, social, cultural, institutional, environmental) in which they will be implemented. This crucial analysis identifies:
Therefore, the evaluation team can be asked to assess the quality of the analysis, and the conformity between the analysis of the situation and the adopted strategy (or programme). The problem diagram, as an ex post construction, can be one of the tools used to check the coherence and the relevance of the analysis in respect of the main contextual problems.
The evaluation team should not underestimate the amount of work, nor the level and variety of competences required (Human resources and working arrangement) to construct the problem diagram. Indeed, the evaluation team should not simply present a diagram with the problems developed in the documentation, but also reconstruct the problem diagram which should have been prepared during the development of the objectives diagram (and therefore, during the drafting of the strategy)
PROBLEM OR OBJECTIVE: WHICH COMES FIRST?
In project cycle
In the planning process and project management field (ZOPP, PCM), specialists place the preparation of the problem diagram before the objectives diagram. The typical sequence can be pictured as followed:
Analysis of the problems in a simplified sequence of the project cycle
Problems analysis may be undertaken before stakeholder analysis, but should always precede the objectives analysis. Indeed, in Project Cycle Management (PCM), objectives are always deduced from the problem analysis.
In the strategy policy co-ordination cycle
If the definition of the donor's activities in bilateral co-operation does not systematically result from the analysis of the problems, it will target one or more overall objectives (strategic and philosophical) which are explicitly or implicitly mentioned in the documentation. The donor (alone or in partnership) defines the main fields of the development assistance in accordance with these objectives.
The chart below illustrates the logical position of the problem analysis within the strategy policy co-ordination cycle and, particularly, in relation with the definition of the objectives.
Analysis of the problems in a simplified sequence of the strategy policy co-ordination cycle
The problem analysis and the resulting diagram derive from the definition of the intervention scope, and thus from the overall objectives of the donor (the European Commission). They also form the basis for the definition of specific objectives of the development assistance, which is to be provided during the period covered by the strategy.
WHAT ARE ITS ADVANTAGES AND LIMITATIONS?
Presentation of the problems
The diagram presents the various problems and their relationship with the core problem through a system of rows. It displays the causal logical links between them or, conversely, the poor logic of these links.
Main problems and contextual problems
In the analysis of the situation, the diagram distinguishes the problems relating to the activity's context from the problems to be solved by the strategy and the planning. As a consequence, its construction requires the highlighting of priorities of the development assistance, and explains why certain problems are considered as important features of the strategy while others are not.
Definition of the objectives
The problem diagram enables the evaluator to:
It contributes to the organisation of the evaluation around a crucial question, which should be systematically answered:
To what extent have the objectives been achieved?
The diagram rationally organises the main difficulties on the basis of the analysis of the situation. It may entail the following limitations.
Access to information
The lack of data or difficult access to information may lower the quality of the analysis of the situation. The evaluator should therefore reflect upon the subsequent questions:
Quality of the analysis
The methodology for the analysis does not guarantee high quality of the data, for neither the methodology, nor the sources of information are usually mentioned in the strategic, political and programming papers. The evaluator must therefore systematically enlarge the assessment to include the sources of the analysis displayed in these documents. The methodology used, the nature of the main sources of information and the identity of the authors should also be noted.
Main problem / contextual problem
The determination of the priorities for each problem results in their classification into two categories: contextual problems and main problems. Two questions must be addressed:
Selection of the main problem
The selection of the main problem, which is crucial for the construction of the diagram, is particularly challenging when the objectives of the activities are general and the whole range of the country's problems (or the region's) are to be considered. The documentation may show two main problems which lack links between them, or may appear insufficient for the determination of a single core problem.
The standard setting of the challenges into a tree diagram illustrates a straightforward classification (which types of problem diagrams can be used?), which does not always highlight the complexity of the situation and the interactions between issues. Indeed, the construction of diagrams depends on graphical conventions, such as:
More sophisticated representations should therefore be tried.
Knowledge of the situation in the country or the region
The evaluation team may lack a sufficient understanding of the country or the region under consideration to assess the relevance of the analysis undertaken, the determination and the logical setting of the main problem.
WITH WHICH TOOLS CAN IT BE COMBINED?
The problem diagram is usually associated with the objectives analysis (and as a consequence, with the objectives diagram) and the development of the logical framework.
The problem diagram is used for the construction of the objectives diagram, often through a simple transposition.
The analysis of the programme's development process (decision diagram) can be used to explain the reasons underpinning the determination of the main problems and the core problem.
In intermediary evaluations of programmes, the evaluator may be asked to construct a decision and an objectives diagram, in addition to the problem diagram itself.
In an ex post evaluation or in an evaluation with ex post components, the initial analysis of the situation and the selection of the main problems can be compared with the analysis of the situation undertaken at the end of the period assessed. They can also be compared to an effect diagram resulting from the strategy's implementation, in order to assess the contribution of the development assistance to the resolution of problems.
Other analytical tools
In the context of an ex post reconstitution of the events, the problem diagram may be supported by other analytical tools, such as:
WHAT ARE THE PRECONDITIONS FOR ITS USE?
When the problem diagram is found in the basic documentation of the programme, or when it can be directly deduced from a logical framework, it must be presented with a study of the choices underpinning its construction. This requires the consultation of the documentation which has supported its construction, and the interviewing of the actors directly involved in its development.
When key actors and documentation are not available
Subsequent to the consultation with specialists, if required, the evaluator may present conclusions about the relevance of the problems system (illustrated in the diagram).
When the diagram has been developed on the basis of the documentation at the disposal of the evaluator, the latter should:
If neither the archives documentation, nor the key actors are available, the evaluator must explain the hypotheses at the basis of the diagram's construction, by providing the reader with precise sources (quotations and comprehensive references of the documentation) underpinning the hypotheses.
HOW SHOULD THE PROBLEM DIAGRAM BE CONSTRUCTED?
STAGE 1: HOW TO IDENTIFY THE PROBLEMS
Record the references to the problems found in the documentation
Quotations from the evaluation's baseline documents are used to identify problems which may not be systematically depicted as such in the texts.
Sometimes, problems are identified as assistance objectives or impacts targeted by these objectives.
Problems are thus expressed as:
What is to be recorded in the documentation?
The preliminary analysis of quotations can reveal:
A thorough selection should therefore be undertaken.
Distinguish context problems from main problems
Among the various problems, the documentation often identifies problems which should be solved by the intervention. The evaluation team should therefore identify the problems related to the intervention in the analysis of the situation, in order to incorporate them into the diagram. The remaining problems should be considered part of the context.
The problems directly targeted by the intervention may not be explicitly identified. Main problems and context problems can be intermingled, which complicates the construction of the diagram. The evaluator will only be able to differentiate the two types of problems after the completion of the diagram.
Whatever the situation, the evaluator should present the list of all the problems under consideration in the annexes, split into two categories:
Identify the core problem
The crucial stage of the process is the identification of the core problem among the variety of the selected problems.
Three situations can be encountered.
The core problem is mentioned in the documentation
The core problem is not clearly mentioned, whereas the overall objective is
These two situations do not need further developments: the evaluator records the core problem explicitly or implicitly mentioned in the documentation, whatever his/her opinion about the problem selected.
Neither the core problem, nor the overall objective are explicitly mentioned
Whatever the procedure, the selection of the core problem should be conducted concurrently with the classification of the problems into levels. Indeed, the selection of the core problem should be supported and justified by the coherence of the whole diagram.
Proceed to a first classification of the problems by rows
Several situations must be considered:
Each of the problem's classification by row must be justified (with interpretations presented as explicit assumptions, if needed).
One of the main difficulties in the graphical illustration of problems is to distinguish between "short-cuts" and inconsistencies.
In a "short-cut", the succession of causal links is pictured as a causal link between the first and the last problem of the causal chain.
For example, the chain illustrating the impact of inadequacies of the road infrastructure on poverty can be illustrated as follows:
In the same document, quotations can be illustrated as follows:
This example does not portray inconsistencies but stylistic short-cuts.
STAGE 2: HOW TO CONSTRUCT A PROBLEM DIAGRAM
Establish a temporary diagram
The diagram may intend to faithfully reflect the problems system as stated in the strategic and planning documents with missing elements and inconsistencies. If no faithfully constructed diagram is explicitly requested, the evaluators will consider it as a step to the construction of a logically reconstructed problem diagram.
Stages for the drafting of temporary diagram
Should the diagram's construction start with the core problem or the in-depth causes?
Usage highlights the fact that these two types of problems/causes are the easiest to identify, whereas intermediary causes are the hardest to determine and classify. Thus, it is recommended that the development of the diagram starts with its extremities at the same time.
Identify the authors who have analysed the situation
Before testing the temporary diagram, the evaluator should identify the main and secondary authors who have contributed to the analysis of the situation (writing of a chapter, sector-based contributions, participation in working sessions, etc.).
It may be useful to check the impact of this analysis on the definition of the strategy and programme operational objectives.
Test the temporary diagram
Where possible, the authors of the documentation referred to above should test the diagram in order to validate the classification of the problems by rows and links. The aim is to check that the evaluator's interpretation reflects the authors' intention correctly. If the authors are not available or, in order to complement their original contribution, the evaluator should consider asking for the participation of other actors responsible for the drafting process.
Contacting the authors is usually possible when the documentation is recent and the authors are still in their position or contactable in one of the services. This task is challenging when the documentation is old and its authors are not easily identifiable or contactable.
Develop the final version of the problem diagram
The final version takes into account the opinions collected during the testing of the temporary diagram.
It is an accurate account of the initial intentions logically reconstructed of the European Commission, taken from official documentation.
HOW MANY DIAGRAMS MUST BE DEVELOPED?
A single diagram
If the analysis of the situation does not radically evolve during the evaluation and if the activities focus on a single core problem, one diagram is enough.
HOW SHOULD PROBLEM DIAGRAMS BE USED?
When it is impossible to directly establish objectives diagrams, problem diagrams play an essential role in the organisation stage of the evaluation.
Conversion into an objectives diagram
Problem diagrams present a summarised vision of the situation to be improved, partially at least, by the strategy. The classified objectives included in the strategy should be deduced from the problems diagrams.
During the planning stage (of a project or a programme), the conversion of the problem diagram into an objectives diagram is not automatic, because each problem integrated into the diagram does not have the same priority.
During the evaluation stage, the reconstruction of the problem diagram includes a step which differentiates between context problems and intervention problems. As a consequence, the diagram resulting from this selection should be completely convertible into a logically reconstructed objectives diagram, i.e. the higher-order overall objective corresponds to the core problem, and each row of subordinated objectives to its equivalent row of problems.
Conversion of the problem diagram into an objectives diagram
Extract from 'A summary of the theory behind the LFA method', The Logical Framework Approach (LFA), SIDA - Swedish International Development Co-operation Agency, Draft, June 2002.
The objectives diagram displays the classification of objectives, from the higher-order objective to the projects and programmes that are designed to achieve the overall objective. The diagram can provoke comments about:
These comments must be presented in the reports and notes of the evaluation organisation stage. They must also appear as an annex in the final report.
Definition of the themes for evaluation questions
The previous analyses raise questions about:
These questions lead to the determination of a range of themes which could be investigated during the subsequent stages of the evaluation, particularly through evaluation questions.
However, evaluation questions cannot be automatically deduced from these analyses. Other questions may emerge during the organisation stage, formulated by the strategy implementing operators. Such questions can relate to the difficulties encountered during the planning and implementation stages. However, the number of evaluation questions is limited and additional questions should be the subject of a selection process.
HOW ARE THE FINDINGS PRESENTED?
Notes and reports of the organisation stage
Problem diagrams are established during the organisation stage, where the reports and notes should be provided. At this stage, the construction of the diagram must be precisely described.
The precise sources identifying the problems (exact quotations, references to original documents) must be provided. References to specific documents, interviews, and expertise must support the location of the problems in the diagram, and the assumptions developed during the diagram's construction must be explained.
The problem diagram should be incorporated into the final report because it is not only a stage in the development of the evaluation, but also an illustration of a possible analysis of the evaluation's strategy.
The report itself may include a short presentation of the diagrams, in addition to its development in the annexes.
The evaluation team will need to present its work (methodology and findings) to different groups of people (the evaluation reference group, participants in the debriefing session, etc.). The problem diagram is a very efficient tool for these situations, providing that it is readable without being over-simplistic.
To do so, a main diagram and several sub-diagrams developing fundamental sections of the latter should be presented, each of them not exceeding more than 20 items.
WHAT ARE THE PRECONDITIONS FOR THE DEVELOPMENT OF THE PROBLEM DIAGRAM?
Human resources and working arrangements
The need for human resources varies with the tasks to be achieved and the situation encountered: reconstruction of the problem diagram with the analysis of the situation displayed in the documentation; retrospective reconstruction of the problem diagram as it should have been established to support the relevance of the strategy or programme under assessment.
The right column of the table below illustrates a wide range of working days.
Type of work required for the design of a problem diagram (usual situation)
Strategy papers are drafted under the responsibility of the head office (DG RELEX or DG DEV), which is also responsible for the planning stage. EuropeAid is in charge of project drafting, which explains why the majority of the useful documentation (including the complete baseline documentation) is in Brussels. Some tasks may be devolved to Delegations in the future, whose responsibilities in this area will grow.
In a country evaluation context, the retrospective construction of the diagram requires at least 2 or 3 experts working on a mission of 5 to 10 days in the country under consideration.
Most of the graphic problems can be solved with software like MS PowerPoint.
The evaluator can also benefit from specific software designed to assist decision-making, such as MS Visio, TeamUp-PCM for Windows, and Palisade Precision Tree.
IN COUNTRY EVALUATION
WHEN SHOULD IT BE USED?
Donors must explicitly or implicitly explain the decision to provide assistance to a country or a region in a given timeframe. For this purpose, they must determine the objectives of the assistance designed to solve a range of problems (the objectives can be the donor's or the beneficiary's).
The purpose of an evaluation is not to analyse the problems and establish the related diagram, but rather to reconstitute the analysis which has already been undertaken and convert it into a diagram.
Consequently, in country evaluations, the problem diagram is particularly appropriate for the following two situations.
To examine the explicit problems included in the documentation
The conversion of the analysis into a diagram (Stage 2: how to construct the problem diagram) enables the evaluator to check the coherence of the analysis and its relevance to the objectives. For this purpose, a problem and an objectives diagram should be reconstructed (how should problem diagrams be used?) and be compared.
To analyse the non-explicit problems in the documentation
A study of the analysis of the situation facilitates the establishment of a problem diagram, whose identified problems should be solved by the assistance. This stage should ease the reconstitution of an objective diagram, or as a minimum, validate the draft of the diagram established in accordance with the documentation.
WHICH PROBLEMS MUST BE TAKEN INTO ACCOUNT?
In a bilateral assistance strategy, the European Commission is the donor, and the States and national public institutions are usually its partners. The European Commission and the States have their own overall objectives, corresponding to their own issues. Yet, to a certain extent and degree of detail, the problems become common.
The donor's range of problems
This type of problem corresponds to strategic issues meant to be resolved with financial assistance (for example, regional instability, weak absorption capacity of the local market, uncontrolled immigration).
The country beneficiary's range of problems
This range of problems is related to major political issues to which the States are confronted (macro-economic unbalances, social instability, weak growth, social or geographical disparities, etc).
The range of problems common to the donor and the country beneficiary
This type of problem has common origins which are relevant for both the donor and the country beneficiary, and are relative to the fields of the assistance scope (for example, health, SME, irrigation and environment).
Except for participatory evaluations, the evaluation team will prioritise the first and third type of problems, because they are directly relevant to the donor.
WHICH DOCUMENTATION SHOULD BE REFERRED TO?
Where can documents analysing problems be found?
Documents related to the European Commission strategy
Documents related to development assistance programmes
Thus, the construction of a problem diagram requires the evaluator to distinguish context problems from main problems.
Which documents should be selected?
Successive versions of the analysis of the situation can seldom be found (in strategy or programme papers). After ensuring that previous drafts do not yield more in-depth and valid analyses, the evaluation team mainly works on the final version.
The baseline documentation comprises the (CSP) and the corresponding National Indicative Paper (NIP), covering the timeframe of the evaluation scope. They are usually provided to the evaluation team at the start of the mission.
Often, the evaluation period (defined in the Terms of Reference) is covered by several strategy and programming papers, each of them including analyses and information about the situation of the country, specific sectors or regions.
Usually, the situation and problems are analysed at the start of the period covered by the strategy and/or the programme under assessment. Yet, the analysis may be affected by difficulties concerning the information, such as:
These difficulties, which may even weaken the relevance of the objectives, should not be underestimated by the evaluation team.
WHICH TYPES OF PROBLEM DIAGRAMS CAN BE USED?
Complete problem diagram
Used in participative planning, the complete problem diagram sketches on each side of the core problem the problems identified by the stakeholders.
Simplified problem diagram
The evaluation only focuses on the lower part of the diagram, because the projects usually deal with the causes of the problems.
Complex problem diagram
Usually, the evaluation team limits its task to the reconstruction of a simple tree shape diagram (simplified or complex, depending on the case) which is derived from the identification of the core problem.
This type of diagram is faithful to the complexity of an actual situation. Yet, its reconstitution requires a wealth of information which is seldom available. Furthermore, the conversion of such information into a diagram represents a challenging task.
The logical framework and the objectives tree
Use of the logical framework in evaluations:
The other approaches: