Do more through capacity4dev
This section is structured as follows:
- What is an objectives diagram and what is an effect diagram?
- Why and when?
- How should the objectives and effect diagram be constructed?
- How are the diagrams used?
- How are the findings presented?
- What are the preconditions for establishment of an objective diagram and an effect diagram?
- In country evaluations
WHAT DO THEY REPRESENT?
Objectives and effects: definitions
Development assistance (in terms of projects, programmes and strategies) usually focuses on an objective to be achieved, or an intended effect.
The objective is expressed in terms of an intervention, whereas the impact is expressed in terms of consequence of the intervention.
Various glossaries of evaluations provide definitions for these two terms, such as in the OECD's glossary (Glossary of Key Terms in Evaluation and Result-Based Management).
The objective of the external assistance covers a variety of aspects, from factual interventions (roads, training, etc.) to macro-economic and social changes. In evaluation, the term impact conveys the idea of wide-scope and long-term effects. By convention, medium term effects are called outcomes, and short-term effects results. The tool will refer to the term effect, so as to include the development assistance's range of outputs (results, outcomes and effects).
When the diagram is used as a structuring tool, the effects presented in the diagram are the effects intended from the assistance.
The diagram can also be used as an evaluation tool for the analysis of the programme's outcomes. The effects presented in the diagram are the observed effects.
At the planning stage, the assistance goals are usually expressed in terms of objectives. At the evaluation stage, the concepts of objectives and effects can be used either way.
The role of a diagram in the planning process
The objectives system
Usually, a long term and global strategic objective, assumed to be a first level objective, is fulfilled through the completion of a range of second level objectives. This is true even where the first level objective is straightforward. Each second level objective depends on the completion of several third level objectives, and so on, down to operational objectives (the intervention projects).
Therefore, an objective is usually understood as a means to achieve a superior level aim, while depending on the completion of subordinate means or objectives.
The objectives system is the presentation of all the objectives of all levels with their respective links.
The effect system
The outputs of the implementation of a project, a programme or an assistance strategy include direct results, the short-term outcomes which are linked to these results, and the longer term impacts (direct and indirect). Such outputs can be called 'effects' and are linked together in a range of causal relations or synergies at the basis of the effect system.
The objectives tree
The need for a logical classification of the range of objectives in objective-based planning is at the origin of the drafting process of programmes and projects. The objectives system is usually presented as a system of roots and a trunk, hence its common name, the objectives tree.
Such tree-like illustrations, however, are subject to some basic rules:
The level expresses the place of the objectives in the cause-and-effect system. In the diagram, the level is represented by the row.
When these rules are applied, objectives trees are a simplified illustration of the objectives system defined during the drafting of the strategy (or the programme).
In the context of the European Commission development assistance, objectives trees show the classification of objectives to be achieved in a geographical (region or country), sector-based or thematic strategy. They range from the European Union's long-term global objective, down to the activities carried out in operational programmes.
The role of the objectives diagram and effect diagram in an evaluation
What actually happens in thematic, sector-based or geographical evaluations is more complex than that illustrated in objectives and effect systems. Indeed, strategies and programmes under evaluation do not systematically result from objective-based planning. Even if they do, the logical classification of objectives may result from decisions based on circumstances rather than from a rational selection derived from the fundamental issues.
This situation may result in:
The use of standard objectives trees in evaluations assumes that the definition of the objectives is rational and the strategic scope can be reduced to a simplified illustration.
These limitations explain why the use of objectives diagrams, whose shape can fluctuate more, is favoured to strictly codified objectives trees.
The effect diagram
International development assistance has tended to evolve progressively in the fields of planning and management from an objective-based approach to a result-based approach. Thus, the evaluation must take into account the effect system linked to the programme or the strategy to be assessed, and organise it. The term effect diagram is used to describe the theoretical organisation of the effect system (outputs, short-term results and intermediate impacts) which leads to the overall intended impact.
The effect diagram displays the classification of the results, outcomes and impacts of what is intended from the implementation of the objectives system. Its tree-shape connects the actual activities which have been planned, and the outputs, which should produce direct results, to the medium term intermediate impacts and the long-term global impacts. The tree can also be read from the bottom to the top, the long-term overall impact being reached after the implementation of intermediate impacts, results, outputs and interventions.
WHERE DOES THE OBJECTIVES DIAGRAM COME FROM?
The objectives diagram comes from objective-based management whose use expanded during the 1960s in the economic sphere of the United States and in Western economies.
It was first adapted to the requirements of the United States Department of Defense and, in the late sixties, to the USAID country assistance management as a component of the logical framework.
Thus, the first well-known use of objectives diagram (including USAID), relates to the drafting of logical frameworks.
The objectives diagram belongs to a series of country assistance decision tools, and can be combined with the problem diagram in the context of Goal Oriented Project Planning (GOPP, originally called ZOPP in German for Zielorientierte Projektplanung).
At the start of the 1980s, GTZ (German Technical Cooperation) conducted a methodological revision of ZOPP, to be used as an educational tool to determine the stages of the participatory process which produces the logical framework.
The typical sequence can be illustrated as followed:
Analysis of the problems of a simplified sequence in the project cycle
In this context, the objectives diagram is:
Following GTZ's initiative, various public donors and operators (agencies and NGOs) have used the problem tree as a component of project planning and management.
Its use as a participatory and appropriation tool has not always played the paramount role defined by the ZOPP designers, indeed, in many cases, the stakeholder analysis was insufficiently comprehensive and the participation procedures were extremely formal, resulting in the problem tree becoming a formal exercise, even a manipulation tool. The logical framework was therefore limited to rigid project management tool, ignoring the fact that it is first and foremost a learning and negotiation tool. As a consequence, emphasis was put on over-simplistic wording, forgetting that these corresponded to the educational objective of the tool.
The biggest criticism of the objectives diagram is the assumption that a consensus can be reached for development objectives and strategies among stakeholders (donors, operators and beneficiaries).
Inadequacies and modifications at the programme drafting stage should be examined during the evaluation, which should focus particularly on the drafting process of the objectives system and avoid tools requiring or yielding a simplistic representation of the system.
WHAT SHAPES CAN AN OBJECTIVES DIAGRAM AND AN EFFECT DIAGRAM TAKE?
Tree shape or complex diagram
The standard objectives tree is subject to the following basic rules:
When trees are drafted in the context of objectives-oriented planning, the first row objective is usually called the "global objective". The lower row objectives have various names, but preferably "intermediate objectives". Several rows can be dedicated to intermediate objectives. The objectives located at the bottom of the tree are called "operational objectives".
The objectives diagram set out below is a complex objectives system:
Complex objectives diagram
Examples of more complex objectives systems can be found, having two global objectives and subdivisions of intermediate objectives, however, such objectives diagrams cannot present feedback links.
A structured presentation of the effect system (the most recent one) has taken into account the limitations of the tree shaped representation and adopted the flexible shape of the diagram. As a consequence, the systems of results, outcomes and effects can have the same degree of complexity as the objectives system, and can be illustrated by similar diagrams.
Vertical or horizontal diagram
Diagrams can either be vertically or horizontally oriented.
WHAT IS A LOGICAL LINK?
The objectives and effects illustrated in the diagrams are related to each other with horizontal or vertical links. These links are called "logical" when expressing an inference relation (induction or deduction) which has been validated by experience. They highlight the fact that:
The logical nature of the links between objectives/effects are essentially based on experience and development theories (derived from this experience). Experience and theory can determine which subordinated objectives/effects should result from the implementation of objectives/effects of a certain level, in order to be properly fulfilled.
However, in terms of development, experiments do not always result in the same conclusions and, as a consequence, should not be considered universally applicable. This can be accounted for by the dependency of the objectives/effect on various factors, often not well-understood.
For example, even when experts agree on the relevance of a global objective, they may disagree on the definition of intermediate objectives, and in particular, operational objectives. The content of overall objectives such as poverty alleviation is continuously debated and evolving.
Verification of the logical nature of the diagram
The logical links between objectives/effects (i.e. the logical nature of the objectives diagram and effect diagram) can simultaneously be checked with:
These two groups of actors may validate the logical nature of the diagram unanimously or individually.
The evaluation should clearly present the process of the validation and its outcome.
WHY AND WHEN ?
WHEN SHOULD AN OBJECTIVES DIAGRAM BE USED?
Three fundamentals in development assistance programmes
An assistance programme can be viewed from different perspectives: its results, internal procedures, partnerships, etc. In terms of efficiency, its immediate results and/or impacts should be understood in relation to the various objectives and/or effects defined in the assistance programme. This implies that the objectives must be identified, or easily recognisable, and classified, or easily put into a hierarchy.
In programmes established on the basis of the logical framework
When a logical diagram has supported the drafting of a programme, it clearly states the various levels of objectives which are targeted by the programme. The logical framework is consequently a reference point for the evaluation, as a consequence of the presentation of the objectives diagram's basic constituents. Although the establishment of result-based logical frameworks has not yet been generalised, the effect diagram can usually be deduced from the logical framework of the intervention's objectives. The evaluator should reproduce the objectives diagram included in the programme, or, if it is not available, reconstruct it with the help of the logical framework's first column, and, if necessary, convert it into an effect diagram.
In more complex policies and strategies
The objectives diagram can also be used in evaluations for projects and programmes whose rationale is not explicit in the logical framework. In practice, the objectives and intended effects of complex policies and strategies often lack explicit presentation and logical structure, whereas the justification for an evaluation is to be able to answer the following questions:
In order to carry out the evaluation, the evaluator must determine and rank the objectives/effects of the strategy or the policy. In this context, an objectives diagram combined with an effect diagram is an effective tool for the reconstruction and representation of the objectives system and/or effect system.
For programmes established on the basis of the logical framework, as well as in more complex policies and strategies, the construction of the objectives diagram and effect diagram should be one of the evaluation's first tasks. The diagram orients the first stage of the information collection and the undertaking of interviews, whose goal is to develop an in-depth knowledge of the policy or programme to be assessed.
WHAT ARE ITS ADVANTAGES AND LIMITATIONS?
Logical classification of the objectives and the effects
The diagram explicitly illustrates the objectives/effects and their various rows, from the global objective (more or less long term), down to the range of activities that have already been implemented or are to be undertaken.
It reveals the logical links between objectives/effects, or the lack of a linkage.
Definition of the evaluation questions
Through the diagram, the evaluator examines a range of questions whose aim is to help answer the crucial question:
To what extent have the stated objectives and the intended effects been achieved?
Presentation of the strategy
When the objectives diagram is well structured and clearly presented, it is a valuable educational tool which facilitates the understanding of the strategy.
Replacing the tree illustration for the diagram avoids most of the construction constraints and their inherent limitations (with one exception: the representation of retroactive links).
A simplified representation of reality
As a chart, the diagram is a simplified representation of reality, and its educational value depends on the selection of a sensible degree of simplification.
In order to avoid an over-simplification of the facts, the evaluator can develop sub-diagrams focusing on specific parts of the main diagram.
A would-be coherence
The diagram establishes a logical link between an objective and another belonging to the row above, or between an effect and another belonging to the row above. Each subordinate objective is presented as logically dependent on a higher row objective, and each effect is presented as logically dependent on a higher row effect. The outputs of the activities implemented or scheduled by the programme appear as a contribution to the global objective/impact and support the coherence of the objectives and effect systems.
Yet, the objectives diagram and the effect diagram (whether they are directly drawn from the programme's logical framework or reconstructed from explanatory documentation) conceal a series of choices. Each objective or effect of a given row is achieved through the implementation of subordinate objectives or effects whose selection must be explained. Indeed, a strategy or a programme seldom address the full scope of the overall objective, which is limited to choices for each row of objectives or effects. Unless the evaluation can find an explicit explanation of the choices made in strategic documentation, it must provide an answer to series of questions:
Selected and rejected objectives
Selected and rejected effects
If the evaluator does not investigate such questions, the evaluation of relevance and coherence of a programme or strategy may be superficial.
WITH WHICH TOOLS CAN IT BE COMBINED?
The objectives diagram often comes (but not systematically) from the problem tree.
Problem diagrams illustrate the overall aspects of a situation which should at least be partially improved by the strategy, whose ranked objectives should be deduced from the diagram.
In the planning process, such a deduction is not systematic. Indeed, for various reasons, all the problems listed in the diagram cannot be analysed with the same priority.
The relationship between the problem and the objectives diagrams
In evaluations, the reconstruction of the problem diagram includes a stage during which the problems caused by the context are distinguished from those following the implementation of activities. The resulting diagram should be easily translated into an objectives diagram: the focal problem should correspond to the global objective, and each level of problems should be matched by an equivalent row of objectives.
In strategy or planning papers, the objectives diagram is often linked to the development of the logical framework, which provides to the diagram its left column and general structure.
The analysis of the programme's development (decision diagram) can be used to explain the reasons for the selection of objectives for various rows (and particularly the highest row objectives).
In mid-term programme evaluations, the evaluator may be asked to re-establish some of the tools which have been previously mentioned in this methodology, in addition to the objectives diagram.
WHAT ARE THE PRECONDITIONS FOR ITS USE?
Study of the initial documentation
When the objectives diagram and the effect diagram are found in the initial documentation of the programme, or when they can be drawn from the logical framework, they should be presented, with the study of the choices made during their construction. This implies a study of the documentation which was used in the construction of the diagrams and interviews with stakeholders directly involved in their drafting.
In sector-based evaluations, baseline documents can be found in all the sector-based strategy documentation originating from the European Commission (such as communication papers, guidelines, handbooks, etc.), from the partner States and other donors.
Baseline documents also include other strategic documents (such as CSPs) and their corresponding national indicative programmes (NIPs) which are relevant to the evaluation scope under study, as well as bilateral agreements such as Association Agreement, Agreement Cooperation and their monitoring documents.
When key actors and documentation are not available
When neither the documentation nor stakeholders are available, the evaluator should discuss the relevance of the objectives system presented in the diagram with specialists, and formulate a judgement.
When the evaluator has reconstructed the diagram on the basis of the available documentation, the evaluation team must:
If archives and stakeholders are not accessible, the evaluation team must explain and describe their own assumptions underpinning the construction of the diagram. The team must therefore provide the reader of the diagram with the sources (quotations and documentary references) on which the diagram is based.
HOW CAN THE OBJECTIVES DIAGRAM BE USED IN PROJECT EVALUATIONS?
A means to concentrate on the intervention rationale of the project
Usually, projects are provided with a logical framework. The analysis should focus on this logical framework, and study the vertical and horizontal coherence of the matrix with the context. In the absence of a logical framework, the intervention rationale should be reconstructed on the basis of the project's documentation (Stage 2: identify the objectives and the impacts).
There may be a delay between the design of the project and its start. In this case, the relevance of the logical framework should be checked when the project is launched.
Any other interventions in the field of application of the project should be taken into account, in order to assess the coherence of the project's interventions with the general context of the assistance.
Which objectives should be taken into account?
The evaluation studies the objectives presented in the logical framework, or in the reconstituted logical framework.
The ownership of the project's objectives should be measured for the various types of stakeholders, particularly in the context of a participative evaluation.
Once completed, the logically reconstructed objectives diagram results in two concurrent evaluative approaches:
HOW SHOULD THE OBJECTIVES AND EFFECT DIAGRAM BE CONSTRUCTED?
The evaluation should first identify the outcomes of the assistance and compare them with the objectives to be achieved. The structuring stage requires the analysis of the objectives, from which the intended effects are deduced.
After determining the evaluation scope, the evaluators should construct a faithfully reconstructed objectives diagram from the strategy and planning documents. A logically reconstructed objectives diagram and an effect diagram (intended effects) will be derived from the faithfully reconstructed objectives diagram.
STAGE 1: HOW IS THE EVALUATION SCOPE DELIMITED
Usually, the evaluation scope is strictly defined in the terms of reference which includes information about the timeframe and the financial tools to be assessed.
In any case (concerning the evaluation timeframe or the financial tools), the managing authority must provide the evaluation team with a precise definition of the scope, although the evaluation team must feel free to warn the latter about the consequences of a too restrictive scope.
Period of the evaluation
The period to be assessed fully or partially corresponds to the period covered by the strategy and/or planning papers.
However, during this timeframe, projects and programmes which are being implemented may have already been described and authorised in previous documentation.
What importance should be given to the objectives or the effects of such projects and programmes?
Two situations should be considered, depending on the nature of the projects and programmes implemented during the period under study (i.e. their objectives are in continuity with or differing from the objectives planned in previous projects and programmes).
Projects and programmes whose objectives differ
The evaluator should undertake a general analysis of the differing topics in the overall strategy and type of activities, and avoid the study of specific objectives (or effects) included in these projects and programmes.
Projects and programmes whose objectives are in continuity
In this case, the study of their specific objectives (or effects) should be included in the analysis of the objectives.
Explicit terms of reference
Terms of reference can focus the evaluation on the activities carried out by a specific institution (a General Directorate of the European Commission) with a specific financial tool. The scope is strictly limited. The evaluation team should however examine other institutions, include them in the category "other donors", and seek the elements of coherence in their activities which could be missing from the objectives system or the effect system of the institution under evaluation.
The evaluation team should also examine the coordination between activities under assessment and others, and formulate its conclusion on the possible effects of insufficient coordination.
Inexplicit terms of reference
When terms of reference do not determine the evaluation scope, the analysis of the objectives or the effects should cover the range of activities, whatever the implementing institution and the financial tool implemented.
STAGE 2: HOW TO IDENTIFY THE OBJECTIVES AND EFFECTS
Collect the documentation
As a first step, the documentation needed for the construction of the diagram is collected in the relevant Unit of DG RELEX or DG DEV. It comprises:
The documentation varies with the type of evaluation (geographical, sector-based and thematic).
Record the references to the objectives in the baseline documentation
The task should begin with a careful analysis of the baseline documentation, to assemble a record of all references (quotations, sources) to the objective (understood as the target of the European Commission's assistance) and any explicit mention of the links between objectives.
The documentation seldom uses the term "objectives" and only introduces the term in expressions such as "with a view to", "in order to", "so as to", etc. Objectives can be called "priorities", "fields of intervention", or sometimes "focal sectors".
A precise table of such terminology must be established.
Distinguish strategic from background objectives
Les citations relevées dans les documents de référence sont traduites en objectifs. Chacun des objectifs est assorti d'un numéro d'identification renvoyant aux citations.
The documentation identifies the objectives which should be targeted by the assistance. The remaining objectives are treated as background items.
In an annex to the reports (how are the findings presented?), the evaluation team should present the list of all the objectives, marshalled into two categories: strategic and background objectives.
STAGE 3: HOW TO CONSTRUCT A FAITHFULLY RECONSTRUCTED OBJECTIVES DIAGRAM
Classify the objectives by rows
Several situations can be encountered:
Each of the classifications by row should be explained by means of an interpretation expressed as an assumption.
A provisional classification could be carried out on the basis of the distinction between three levels of objectives:
The definition of the logical relationship is paramount in the classification. This task is a question of experience, because the decision of the evaluation team that two events are logically connected depends on the judgement of experts in the field under study, and managers responsible for the implementation of the strategy and policies.
Establish a temporary diagram
The first faithfully reconstructed objectives diagram can be drawn from this first classification. It usually reveals incoherencies that will have to be corrected in accordance with the evaluator's interpretation (these corrections must be indicated in the diagram).
Should the construction of the diagram start with the overall objective
Usage has shown that these two types of objectives are more easily identified than intermediate objectives, which are also more difficult to rank. Thus, it is recommended that the roots of the diagram and the extremities of the branches are developed concurrently.
In the diagram, each objective is presented as a box with a heading. The evaluator should also indicate the identification number which refers to the quotation list. Providing the boxes with colours corresponding to the level to which the objective belongs may also be helpful.
Each link should refer to one or more quotations (in accordance with the identification number).
The diagram should also mention the specific means (financial and non financial) used in the implementation of each operational objective.
Identify the authors of the objectives' wording
Before testing the temporary diagram, the evaluator should identify the main and secondary authors of the objectives' wording (writing of a chapter, sector-based contributions, participation in working meetings, etc.).
It may be useful to check the consequence of the wording on the definition of the operational objectives of the programme planning.
Test the temporary diagram
The authors of baseline documentation and additional documents should test the diagram in order to validate the classification of the objectives by rows and links. The aim is to check that the evaluator's interpretations accurately reflect the authors' intentions. If the authors are not available or, with a view to complement their contribution, the evaluator can seek the participation of other actors responsible for the drafting process (both from the writing and the discussion process).
Contacting the authors is usually possible when the documentation is recent and the authors are still in position or contactable in one of the services. This task is challenging when the documentation is old and its authors are not easily identifiable, nor reachable.
At the end of the interviews, the evaluator may also encounter difficulties evaluating what was actually done during the drafting process and the justifications of respondents for the decisions taken.
Establish the final version of the faithfully reconstructed objectives diagram
This version takes into account the opinions collected during the test of the temporary diagram.
It should be considered an accurate report of the initial intentions of the European Commission, taken from the official documentation.
STAGE 4: HOW TO CONVERT THE FAITHFULLY RECONSTRUCTED OBJECTIVES DIAGRAM INTO A LOGICALLY RECONSTRUCTED OBJECTIVES DIAGRAM
Advantages and limitations of a faithfully reconstructed objectives diagram
Advantages of the faithfully reconstructed objectives diagram
Some faithfully reconstructed objectives diagrams display a strong internal coherence: apart from the overall objective or the overall expect impact, each objective 'x' leads to an higher (immediate or not) objective and is supported by one or more subordinate objectives. This is particularly the case when faithfully reconstructed objectives diagrams are extracted from a carefully established logical framework. Faithfully reconstructed objectives diagrams can be deemed as completely logical and stand as logically reconstructed objectives diagrams.
Logical links of a given objective 'x' (in objectives diagrams)
The establishment of a logically reconstructed objectives diagram is not required when the objectives system of the strategy and planning documents is thoroughly and consistently presented.
Limitations of the faithfully reconstructed objectives diagram
Some faithfully reconstructed objectives diagrams may reveal logical defects in strategy or political papers, such as:
In these cases, the definition of the evaluation thematic scope and the evaluation questions is difficult. The team should therefore draft a logically reconstructed objectives diagram.
Preparation of the logically reconstructed objectives diagram
In order to prepare a comprehensive and coherent objectives diagram, the evaluation team will need all available documentation, its own expertise and, if required, that of experts. This task should be completely transparent (how are the findings presented?): whenever possible, the diagram should display the reclassification of objectives, the changes affecting links between objectives, and the introduction of missing intermediate objectives. Each of these rationalisation operations should be explained in a technical note.
Stage 5: how to construct an effect diagram
Objectives diagrams and intended effect diagrams share the same rules of construction.
The effect diagram is constructed from the conversion of each of the objectives into the corresponding intended effect.
Objectives diagrams and effect diagrams are a crucial tool in the organisation stage of the evaluation. They play many roles:
Description and analysis of the strategy
Objectives diagrams demonstrate a synthesis of the geographical strategy displayed in the official documentation. When the evaluation scope covers one or more strategy papers (geographical) or strategic policies (sector-based, thematic), it is recommended that one diagram per document is created (unless there is a logical continuity in the strategy or the policy).
They illustrate the objectives/effects classification, from the overall objective/impact to projects and programmes scheduled to implement the objective/effects. The logically reconstructed objectives diagramand the effect diagram are informative means to reflect upon:
The faithfully reconstructed objectives diagram displays the objectives system and provides the evaluator with a first approach to the strategy and policies inner quality. Indeed, an unclear, incomplete and incoherent diagram means a lack of relevance in the forthcoming planning or a lack of faithfulness to the initial objectives system.
These analyses must be presented in the notes and reports of the evaluation organisation stage. They must also appear as an abstract in the final report.
Definition of the themes for evaluation questions
The previous analyses reveal questions about:
These questions result in the determination of a range of themes which could be investigated during the following stages of the evaluation, particularly through evaluation questions.
However, evaluation questions cannot be automatically deduced from these analyses. Other questions can emerge during the organisation stage, formulated by the main strategy implementation operators (since they are the most well-informed actors about the problems encountered during the planning and the implementation stages). However, the number of evaluation questions is limited and additional questions would have to be the subject of a selection process.
Notes and reports of the organisation stage
Objectives diagrams are established during the organisation stage, where reports and notes should be provided. At this stage, the diagram's construction must be precisely described.
For the faithfully reconstructed objectives diagram, the sources of the objectives/effects (quotations, references to the original documentation) must be provided. References to documentation, interviews and expertise must support the objectives' location in the diagram, and the assumptions developed during the construction of the diagram must be explained.
The process through which the logically reconstructed objectives diagram has been extracted from the faithfully reconstructed objectives diagram must also be clearly explained.
Diagrams should be incorporated into the final report because they are not only stages in the development of the wording of evaluation questions, but also an illustration of a possible analysis of the evaluation strategy.
The report may include a short presentation of the diagrams, in addition to a full explanation included in the annexes.
The evaluation team will need to present its work (methodology and findings) to different types of people (the evaluation reference group, participants in the debriefing session). The objectives diagram or the effect diagram are very efficient tools for this purpose, providing that they are readable without being over-simplistic.
To do so, a main diagram, and several sub-diagrams developing fundamental sections of the main diagram, should be presented, each of them not exceeding 20 items.
Human resources and working arrangement
Type of work required for the design of an objectives diagram
Strategy papers have been drafted under the responsibility of the head office (DG RELEX or DG DEV). The head office is also responsible for the planning stage. EuropeAid is in charge of the project drafting. This implies that the majority of the useful documentation (the whole baseline documentation) can be found in Brussels. This situation may devolve to Delegations in the future whose responsibilities in this area will grow.
The current needs (2004) in travelling expenses to Brussels
These elements (and the corresponding budgetary components) will have to be reviewed when the Delegations take broader responsibilities for strategy development (2007).
Most of the graphic problems can be addressed with software such as MS PowerPoint
IS THE USE OF AN OBJECTIVES DIAGRAM AND AN EFFECT DIAGRAM COMPULSORY?
Every donor must implicitly or explicitly explain the decision to provide assistance to a country or a region in a given timeframe. For this purpose, the objectives to be achieved or the intended effects which form the basis for the assistance are described in documents which are regularly published and present the overall objectives and priorities. Such documentation can either display orientations, an individual programme, or both.
Such documents may present programmes which are jointly drafted by the donor and beneficiary. In this case, the interventions under study should be that of the European Commission and of the country partner. The outcomes and intended impacts are the outputs of the implementation of these two types of interventions.
In national programmes
National Indicative Programmes (NIP) can be inserted in the logical framework which can be a single document or correspond to each main sector of intervention. In this case, the evaluation uses the logical frameworks (i.e. objectives and implementation indicators) as the reference for efficiency criteria. Its goal will be to check in what extent the objectives and effect systems have been completed during the implementation of the programme.
In country strategies
Country Strategy Papers (CSP) do not usually rely on an overall logical framework because they cover a long timeframe and are implemented through various successive programmes, each of them supported by a logical framework. Consequently, the evaluators should assemble the objectives and effects system, and draft a diagram for each programme.
WHICH OBJECTIVES AND EFFECTS MUST BE TAKEN INTO ACCOUNT?
In strategies and complex bilateral assistance programmes, the European Commission is the donor and the States and national public institutions are usually its partners. The European Commission and the States have their own overall objectives, corresponding to their own issues. Yet, to a certain extent and degree of detail, the objectives overlap.
In a bilateral assistance contract, three objectives or effect systems can be developed:
Except for participatory evaluations, the evaluation team will prioritise the first and third objectives system.
WHAT ROLE DO OBJECTIVES DIAGRAM AND EFFECT DIAGRAMS PLAY IN COUNTRY EVALUATIONS?
Once completed, the logically reconstructed objectives diagram and the effect diagram provide two parallel approaches to the country strategy evaluation:
These approaches are illustrated in the following table. The main branch of the logically reconstructed objectives diagram splits into two: the right branch deals with evaluation questions, the left branch with the overall analysis of the strategy.
Role of the objectives diagram in the conduct of an evaluation structured by evaluation questions
Two main aspects of relevance are examined:
The evaluation questions connect the intended effects (results, outcomes, secondary impacts) with the specific and general means which are used to achieve the objectives.
The overall analysis studies the strategy building, i.e.:
WHICH TYPES OF OBJECTIVES DIAGRAMS OR EFFECT DIAGRAMS CAN BE USED FOR COUNTRY EVALUATIONS?
Two types of diagram, setting the framework for the evaluation, illustrate the objectives of the programme or the strategy to be assessed.
The faithfully reconstructed objectives diagram
The faithfully reconstructed objectives diagram is useful for two reasons:
The logically reconstructed objectives diagram
The logically reconstructed objectives diagram determines the thematic scope of the evaluation.
It establishes the reconstructed classification of the objectives system as they are presented in logical frameworks or reconstituted in the faithfully reconstructed objectives diagram. It is also the logical reconstitution of the objectives' classification implied in strategy papers (CSP, RSP) and planning papers (NIP, RIP).
It contributes to the assessment of strengths and weaknesses on which the evaluation questions should focus:
The logically reconstructed objectives diagram is particularly needed because the strategy papers and documentation concerning the planning of geographic assistance do not provide the evaluator with a systematic presentation (with or without a diagram) of the objectives system.
It should be summarised within the final report of the evaluation, and fully presented in the annexes.
Diagrams are not the only tools available to analyse objectives and effects. For example, evaluators frequently use a matrix.
HOW MANY EFFECT DIAGRAMS MUST BE ESTABLISHED?
A single diagram
If the analysis of the situation does not radically evolve during the evaluation and if the activities focus on a single intended overall impact, a single diagram is enough.
Many diagrams, depending on the context
When the strategy evolves, effect diagrams should be constructed for each change in strategy of the European Commission.
When the evaluator cannot determine a single overall impact and when each of the global impact depends on an independent effect system, the evaluator should establish as many diagrams as overall impacts.
Intermediate diagrams, corresponding to each of the assistance priorities (or sectors) for the period(s) covered by the evaluation, can be useful for the construction of an objectives diagram or effect diagram. They are particularly adapted for programmes with a logical framework provided for each of the priorities.
Intermediate diagrams can be useful for the determination and justification of evaluation questions.
When overall diagrams are very complex, their readability and information quality may suffer. A simplified overall diagram should be prepared, and detail diagrams should be drawn from the overall diagrams, in order to emphasize some of their specific parts.
WHICH DOCUMENTATION SHOULD BE REFERRED TO IN COUNTRY EVALUATIONS?
The baseline documentation includes strategy papers (CSP) and the related indicative programmes (NIP) relevant to the period covered in the evaluation scope.
Bilateral agreements, such as the Association Agreement, Cooperation Agreement and their monitoring documents should also be included, as well as multilateral agreements (the Lome and Cotonou agreements) and their related documents (for example, the Barcelona Declaration).
In most cases, they should be provided to the evaluator at the start of the evaluation.
However, the evaluator should check that:
Three types of documents must be available :
Other informative documentation
Other documentation may be useful, such as successive versions of the baseline documentation (the date of the final version must be checked), including the various contributions and reactions of the Commission's services, their national partners and other donors (particularly Member States). It is valuable, but often difficult to identify and seldom available or accessible.
The logical framework and the objectives tree
Use of the logical framework in evaluations
The other approaches