Detailed presentation

WHAT IS AN OBJECTIVES DIAGRAM AND WHAT IS AN EFFECT DIAGRAM?

.

WHAT DO THEY REPRESENT?

Objectives and effects: definitions

Development assistance (in terms of projects, programmes and strategies) usually focuses on an objective to be achieved, or an intended effect.

The objective is expressed in terms of an intervention, whereas the impact is expressed in terms of consequence of the intervention.

Various glossaries of evaluations provide definitions for these two terms, such as in the OECD's glossary (Glossary of Key Terms in Evaluation and Result-Based Management).

The objective of the external assistance covers a variety of aspects, from factual interventions (roads, training, etc.) to macro-economic and social changes. In evaluation, the term impact conveys the idea of wide-scope and long-term effects. By convention, medium term effects are called outcomes, and short-term effects results. The tool will refer to the term effect, so as to include the development assistance's range of outputs (results, outcomes and effects).

When the diagram is used as a structuring tool, the effects presented in the diagram are the effects intended from the assistance.

The diagram can also be used as an evaluation tool for the analysis of the programme's outcomes. The effects presented in the diagram are the observed effects.

At the planning stage, the assistance goals are usually expressed in terms of objectives. At the evaluation stage, the concepts of objectives and effects can be used either way.

The role of a diagram in the planning process

The objectives system

Usually, a long term and global strategic objective, assumed to be a first level objective, is fulfilled through the completion of a range of second level objectives. This is true even where the first level objective is straightforward. Each second level objective depends on the completion of several third level objectives, and so on, down to operational objectives (the intervention projects).

Therefore, an objective is usually understood as a means to achieve a superior level aim, while depending on the completion of subordinate means or objectives.

The objectives system is the presentation of all the objectives of all levels with their respective links. 

The effect system

The outputs of the implementation of a project, a programme or an assistance strategy include direct results, the short-term outcomes which are linked to these results, and the longer term impacts (direct and indirect). Such outputs can be called 'effects' and are linked together in a range of causal relations or synergies at the basis of the effect system. 

The objectives tree

The need for a logical classification of the range of objectives in objective-based planning is at the origin of the drafting process of programmes and projects. The objectives system is usually presented as a system of roots and a trunk, hence its common name, the objectives tree.

Such tree-like illustrations, however, are subject to some basic rules:

• There can only be one first row objective, also called the global objective
• Each subordinate objective is related to one objective in the row above
• Interactions between objectives in the same row and feedback links (i.e. effects becoming causes and vice versa) are not represented

The level expresses the place of the objectives in the cause-and-effect system. In the diagram, the level is represented by the row.
Standard objectives tree

When these rules are applied, objectives trees are a simplified illustration of the objectives system defined during the drafting of the strategy (or the programme).

In the context of the European Commission development assistance, objectives trees show the classification of objectives to be achieved in a geographical (region or country), sector-based or thematic strategy. They range from the European Union's long-term global objective, down to the activities carried out in operational programmes.

The role of the objectives diagram and effect diagram in an evaluation

What actually happens in thematic, sector-based or geographical evaluations is more complex than that illustrated in objectives and effect systems. Indeed, strategies and programmes under evaluation do not systematically result from objective-based planning. Even if they do, the logical classification of objectives may result from decisions based on circumstances rather than from a rational selection derived from the fundamental issues.

This situation may result in:

• Two or more first row objectives which cannot be reduced to a single global objective
• Objectives of a given row which are linked to several objectives of the immediate row above
• Objectives of a given row which are related to an objective belonging to 2 rows above
• Interactive objectives at the same level

The use of standard objectives trees in evaluations assumes that the definition of the objectives is rational and the strategic scope can be reduced to a simplified illustration.

These limitations explain why the use of objectives diagrams, whose shape can fluctuate more, is favoured to strictly codified objectives trees.

The effect diagram

International development assistance has tended to evolve progressively in the fields of planning and management from an objective-based approach to a result-based approach. Thus, the evaluation must take into account the effect system linked to the programme or the strategy to be assessed, and organise it. The term effect diagram is used to describe the theoretical organisation of the effect system (outputs, short-term results and intermediate impacts) which leads to the overall intended impact.

The effect diagram displays the classification of the results, outcomes and impacts of what is intended from the implementation of the objectives system. Its tree-shape connects the actual activities which have been planned, and the outputs, which should produce direct results, to the medium term intermediate impacts and the long-term global impacts. The tree can also be read from the bottom to the top, the long-term overall impact being reached after the implementation of intermediate impacts, results, outputs and interventions.

WHERE DOES THE OBJECTIVES DIAGRAM COME FROM?

The objectives diagram comes from objective-based management whose use expanded during the 1960s in the economic sphere of the United States and in Western economies.

It was first adapted to the requirements of the United States Department of Defense and, in the late sixties, to the USAID country assistance management as a component of the logical framework.

Thus, the first well-known use of objectives diagram (including USAID), relates to the drafting of logical frameworks.

The objectives diagram belongs to a series of country assistance decision tools, and can be combined with the problem diagram in the context of Goal Oriented Project Planning (GOPP, originally called ZOPP in German for Zielorientierte Projektplanung). 

At the start of the 1980s, GTZ (German Technical Cooperation) conducted a methodological revision of ZOPP, to be used as an educational tool to determine the stages of the participatory process which produces the logical framework.

The typical sequence can be illustrated as followed:

Analysis of the problems of a simplified sequence in the project cycle

In this context, the objectives diagram is:

• a useful tool for the drafting of projects
• a planning methodology tool, which comes from the problem tree and contributes to the logical framework
• a participative decision tool which embraces the main stakeholders of the project, whose determination is the first stage of the project cycle sequence

Following GTZ's initiative, various public donors and operators (agencies and NGOs) have used the problem tree as a component of project planning and management.

Its use as a participatory and appropriation tool has not always played the paramount role defined by the ZOPP designers, indeed, in many cases, the stakeholder analysis was insufficiently comprehensive and the participation procedures were extremely formal, resulting in the problem tree becoming a formal exercise, even a manipulation tool. The logical framework was therefore limited to rigid project management tool, ignoring the fact that it is first and foremost a learning and negotiation tool. As a consequence, emphasis was put on over-simplistic wording, forgetting that these corresponded to the educational objective of the tool.

The biggest criticism of the objectives diagram is the assumption that a consensus can be reached for development objectives and strategies among stakeholders (donors, operators and beneficiaries).

Inadequacies and modifications at the programme drafting stage should be examined during the evaluation, which should focus particularly on the drafting process of the objectives system and avoid tools requiring or yielding a simplistic representation of the system.

WHAT SHAPES CAN AN OBJECTIVES DIAGRAM AND AN EFFECT DIAGRAM TAKE?

Tree shape or complex diagram

The standard objectives tree is subject to the following basic rules:

• A single global objective
• Subdivision of the objective of a given row into 2 or more objectives of a row below (level 2)
• Relation of a defined objective with a single objective of a row above (level 3)
• No horizontal links
• No feedback links (i.e. effects become causes and vice versa)

When trees are drafted in the context of objectives-oriented planning, the first row objective is usually called the "global objective". The lower row objectives have various names, but preferably "intermediate objectives". Several rows can be dedicated to intermediate objectives. The objectives located at the bottom of the tree are called "operational objectives".

The objectives diagram set out below is a complex objectives system:

• One of the intermediate objectives in row (2) is directly linked to the global objective and not to another intermediate objective (1).
• Two intermediate objectives in row (3) are interrelated.
• One of these objectives is linked to two objectives of the row above (intermediate 2).
• Row 2 of the intermediate objectives appears incomplete (lack of subdivisions to the lower row objectives).

Complex objectives diagram

Examples of more complex objectives systems can be found, having two global objectives and subdivisions of intermediate objectives, however, such objectives diagrams cannot present feedback links.

A structured presentation of the effect system (the most recent one) has taken into account the limitations of the tree shaped representation and adopted the flexible shape of the diagram. As a consequence, the systems of results, outcomes and effects can have the same degree of complexity as the objectives system, and can be illustrated by similar diagrams.

Effect diagram

Vertical or horizontal diagram

Diagrams can either be vertically or horizontally oriented.
In their vertical shape, they read from top to bottom, or from bottom to top.
In their horizontal shape, they read from right to left, or left to right.

WHAT IS A LOGICAL LINK?

Definition

The objectives and effects illustrated in the diagrams are related to each other with horizontal or vertical links. These links are called "logical" when expressing an inference relation (induction or deduction) which has been validated by experience. They highlight the fact that:

• The adoption of an objective/effect at a certain row implies that of subordinated objectives/effects
• An objective/effect of a certain row can be deduced from an objective/effect of a row above (immediate or not)
• Two objectives/effects of the same row share a synergetic relationship

The logical nature of the links between objectives/effects are essentially based on experience and development theories (derived from this experience). Experience and theory can determine which subordinated objectives/effects should result from the implementation of objectives/effects of a certain level, in order to be properly fulfilled.

However, in terms of development, experiments do not always result in the same conclusions and, as a consequence, should not be considered universally applicable. This can be accounted for by the dependency of the objectives/effect on various factors, often not well-understood.

For example, even when experts agree on the relevance of a global objective, they may disagree on the definition of intermediate objectives, and in particular, operational objectives. The content of overall objectives such as poverty alleviation is continuously debated and evolving.

Verification of the logical nature of the diagram

The logical links between objectives/effects (i.e. the logical nature of the objectives diagram and effect diagram) can simultaneously be checked with:

• Recognised experts who have worked in different countries and situations
• Designers and implementing managers of programmes included in the evaluation scope

These two groups of actors may validate the logical nature of the diagram unanimously or individually.

The evaluation should clearly present the process of the validation and its outcome.

.

WHY AND WHEN ?

.

WHEN SHOULD AN OBJECTIVES DIAGRAM BE USED?

Three fundamentals in development assistance programmes

An assistance programme can be viewed from different perspectives: its results, internal procedures, partnerships, etc. In terms of efficiency, its immediate results and/or impacts should be understood in relation to the various objectives and/or effects defined in the assistance programme. This implies that the objectives must be identified, or easily recognisable, and classified, or easily put into a hierarchy.

In programmes established on the basis of the logical framework

When a logical diagram has supported the drafting of a programme, it clearly states the various levels of objectives which are targeted by the programme. The logical framework is consequently a reference point for the evaluation, as a consequence of the presentation of the objectives diagram's basic constituents. Although the establishment of result-based logical frameworks has not yet been generalised, the effect diagram can usually be deduced from the logical framework of the intervention's objectives. The evaluator should reproduce the objectives diagram included in the programme, or, if it is not available, reconstruct it with the help of the logical framework's first column, and, if necessary, convert it into an effect diagram.

In more complex policies and strategies

The objectives diagram can also be used in evaluations for projects and programmes whose rationale is not explicit in the logical framework. In practice, the objectives and intended effects of complex policies and strategies often lack explicit presentation and logical structure, whereas the justification for an evaluation is to be able to answer the following questions:

• Have the objectives or intended effects of the policy or the strategy under assessment been achieved?
• To what extent have these objectives or intended effects been achieved?

In order to carry out the evaluation, the evaluator must determine and rank the objectives/effects of the strategy or the policy. In this context, an objectives diagram combined with an effect diagram is an effective tool for the reconstruction and representation of the objectives system and/or effect system.

For programmes established on the basis of the logical framework, as well as in more complex policies and strategies, the construction of the objectives diagram and effect diagram should be one of the evaluation's first tasks. The diagram orients the first stage of the information collection and the undertaking of interviews, whose goal is to develop an in-depth knowledge of the policy or programme to be assessed.

WHAT ARE ITS ADVANTAGES AND LIMITATIONS?

Its advantages

Logical classification of the objectives and the effects

The diagram explicitly illustrates the objectives/effects and their various rows, from the global objective (more or less long term), down to the range of activities that have already been implemented or are to be undertaken.

It reveals the logical links between objectives/effects, or the lack of a linkage. 

Definition of the evaluation questions

Through the diagram, the evaluator examines a range of questions whose aim is to help answer the crucial question:

To what extent have the stated objectives and the intended effects been achieved?

Presentation of the strategy

When the objectives diagram is well structured and clearly presented, it is a valuable educational tool which facilitates the understanding of the strategy.

Its limitations

Replacing the tree illustration for the diagram avoids most of the construction constraints and their inherent limitations (with one exception: the representation of retroactive links). 

A simplified representation of reality

As a chart, the diagram is a simplified representation of reality, and its educational value depends on the selection of a sensible degree of simplification.

In order to avoid an over-simplification of the facts, the evaluator can develop sub-diagrams focusing on specific parts of the main diagram. 

A would-be coherence

The diagram establishes a logical link between an objective and another belonging to the row above, or between an effect and another belonging to the row above. Each subordinate objective is presented as logically dependent on a higher row objective, and each effect is presented as logically dependent on a higher row effect. The outputs of the activities implemented or scheduled by the programme appear as a contribution to the global objective/impact and support the coherence of the objectives and effect systems.

Yet, the objectives diagram and the effect diagram (whether they are directly drawn from the programme's logical framework or reconstructed from explanatory documentation) conceal a series of choices. Each objective or effect of a given row is achieved through the implementation of subordinate objectives or effects whose selection must be explained. Indeed, a strategy or a programme seldom address the full scope of the overall objective, which is limited to choices for each row of objectives or effects. Unless the evaluation can find an explicit explanation of the choices made in strategic documentation, it must provide an answer to series of questions:

• Do these choices represent the only possibilities?
• Are the objectives the most relevant ones for the row 'x'?
• What external factors have led to this selection?
• What reasons have led to the elimination of certain objectives?

Selected and rejected objectives

Selected and rejected effects

If the evaluator does not investigate such questions, the evaluation of relevance and coherence of a programme or strategy may be superficial.

WITH WHICH TOOLS CAN IT BE COMBINED?

Problem diagram

The objectives diagram often comes (but not systematically) from the problem tree.

Problem diagrams illustrate the overall aspects of a situation which should at least be partially improved by the strategy, whose ranked objectives should be deduced from the diagram.

In the planning process, such a deduction is not systematic. Indeed, for various reasons, all the problems listed in the diagram cannot be analysed with the same priority.

The relationship between the problem and the objectives diagrams

In evaluations, the reconstruction of the problem diagram includes a stage during which the problems caused by the context are distinguished from those following the implementation of activities. The resulting diagram should be easily translated into an objectives diagram: the focal problem should correspond to the global objective, and each level of problems should be matched by an equivalent row of objectives.

Logical framework

In strategy or planning papers, the objectives diagram is often linked to the development of the logical framework, which provides to the diagram its left column and general structure.

Decision diagram

The analysis of the programme's development (decision diagram) can be used to explain the reasons for the selection of objectives for various rows (and particularly the highest row objectives).

In mid-term programme evaluations, the evaluator may be asked to re-establish some of the tools which have been previously mentioned in this methodology, in addition to the objectives diagram.

WHAT ARE THE PRECONDITIONS FOR ITS USE?

Study of the initial documentation

When the objectives diagram and the effect diagram are found in the initial documentation of the programme, or when they can be drawn from the logical framework, they should be presented, with the study of the choices made during their construction. This implies a study of the documentation which was used in the construction of the diagrams and interviews with stakeholders directly involved in their drafting.

In sector-based evaluations, baseline documents can be found in all the sector-based strategy documentation originating from the European Commission (such as communication papers, guidelines, handbooks, etc.), from the partner States and other donors.

Baseline documents also include other strategic documents (such as CSPs) and their corresponding national indicative programmes (NIPs) which are relevant to the evaluation scope under study, as well as bilateral agreements such as Association Agreement, Agreement Cooperation and their monitoring documents.

When key actors and documentation are not available

When neither the documentation nor stakeholders are available, the evaluator should discuss the relevance of the objectives system presented in the diagram with specialists, and formulate a judgement.

When the evaluator has reconstructed the diagram on the basis of the available documentation, the evaluation team must:

• check the validity of the documentation and, if required, complete it
• test the validity of the links and interrelationships with additional documentary research and interviews

If archives and stakeholders are not accessible, the evaluation team must explain and describe their own assumptions underpinning the construction of the diagram. The team must therefore provide the reader of the diagram with the sources (quotations and documentary references) on which the diagram is based.

HOW CAN THE OBJECTIVES DIAGRAM BE USED IN PROJECT EVALUATIONS?

A means to concentrate on the intervention rationale of the project

Usually, projects are provided with a logical framework. The analysis should focus on this logical framework, and study the vertical and horizontal coherence of the matrix with the context. In the absence of a logical framework, the intervention rationale should be reconstructed on the basis of the project's documentation (Stage 2: identify the objectives and the impacts).

There may be a delay between the design of the project and its start. In this case, the relevance of the logical framework should be checked when the project is launched.

Any other interventions in the field of application of the project should be taken into account, in order to assess the coherence of the project's interventions with the general context of the assistance.

Which objectives should be taken into account?

The evaluation studies the objectives presented in the logical framework, or in the reconstituted logical framework.

The ownership of the project's objectives should be measured for the various types of stakeholders, particularly in the context of a participative evaluation.

Once completed, the logically reconstructed objectives diagram results in two concurrent evaluative approaches:

• The analysis of the project's quality criteria (relevance, efficiency, effectiveness, sustainability) in the various levels of intervention (activities, outcomes, specific objectives).
• The internal consistency of the project's strategy (vertical coherence) and its relevance with the European Union's overall objectives, the Member State's policies, and the International donors' position.

.

HOW SHOULD THE OBJECTIVES AND EFFECT DIAGRAM BE CONSTRUCTED?

.

The evaluation should first identify the outcomes of the assistance and compare them with the objectives to be achieved. The structuring stage requires the analysis of the objectives, from which the intended effects are deduced. 

After determining the evaluation scope, the evaluators should construct a faithfully reconstructed objectives diagram from the strategy and planning documents. A logically reconstructed objectives diagram and an effect diagram (intended effects) will be derived from the faithfully reconstructed objectives diagram.

STAGE 1: HOW IS THE EVALUATION SCOPE DELIMITED

Usually, the evaluation scope is strictly defined in the terms of reference which includes information about the timeframe and the financial tools to be assessed.

In any case (concerning the evaluation timeframe or the financial tools), the managing authority must provide the evaluation team with a precise definition of the scope, although the evaluation team must feel free to warn the latter about the consequences of a too restrictive scope.

Period of the evaluation

The period to be assessed fully or partially corresponds to the period covered by the strategy and/or planning papers.

However, during this timeframe, projects and programmes which are being implemented may have already been described and authorised in previous documentation.

What importance should be given to the objectives or the effects of such projects and programmes?

Two situations should be considered, depending on the nature of the projects and programmes implemented during the period under study (i.e. their objectives are in continuity with or differing from the objectives planned in previous projects and programmes).

Projects and programmes whose objectives differ

The evaluator should undertake a general analysis of the differing topics in the overall strategy and type of activities, and avoid the study of specific objectives (or effects) included in these projects and programmes. 

Projects and programmes whose objectives are in continuity

In this case, the study of their specific objectives (or effects) should be included in the analysis of the objectives.

Financial tools

Explicit terms of reference

Terms of reference can focus the evaluation on the activities carried out by a specific institution (a General Directorate of the European Commission) with a specific financial tool. The scope is strictly limited. The evaluation team should however examine other institutions, include them in the category "other donors", and seek the elements of coherence in their activities which could be missing from the objectives system or the effect system of the institution under evaluation.

The evaluation team should also examine the coordination between activities under assessment and others, and formulate its conclusion on the possible effects of insufficient coordination. 

Inexplicit terms of reference

When terms of reference do not determine the evaluation scope, the analysis of the objectives or the effects should cover the range of activities, whatever the implementing institution and the financial tool implemented.

STAGE 2: HOW TO IDENTIFY THE OBJECTIVES AND EFFECTS

Collect the documentation

As a first step, the documentation needed for the construction of the diagram is collected in the relevant Unit of DG RELEX or DG DEV. It comprises:

• Policies, strategies and programmes baseline documentation
• Complementary official documentation

The documentation varies with the type of evaluation (geographical, sector-based and thematic).

Record the references to the objectives in the baseline documentation

The task should begin with a careful analysis of the baseline documentation, to assemble a record of all references (quotations, sources) to the objective (understood as the target of the European Commission's assistance) and any explicit mention of the links between objectives.

The documentation seldom uses the term "objectives" and only introduces the term in expressions such as "with a view to", "in order to", "so as to", etc. Objectives can be called "priorities", "fields of intervention", or sometimes "focal sectors".

A precise table of such terminology must be established.

Distinguish strategic from background objectives

Les citations relevées dans les documents de référence sont traduites en objectifs. Chacun des objectifs est assorti d'un numéro d'identification renvoyant aux citations.
The quotations taken from the reference documentation are converted into objectives and each is given a unique identification number.

The documentation identifies the objectives which should be targeted by the assistance. The remaining objectives are treated as background items.

In an annex to the reports (how are the findings presented?), the evaluation team should present the list of all the objectives, marshalled into two categories: strategic and background objectives.

STAGE 3: HOW TO CONSTRUCT A FAITHFULLY RECONSTRUCTED OBJECTIVES DIAGRAM

Classify the objectives by rows

Several situations can be encountered:

• The row of the objective is explicit. This is often the case for the objective of the first row, also called the overall or long-term objective.
• The cause-and-effect link between two objectives is explicit, which facilitates the understanding of the interconnections between them.
• No indication is given about the objective's row (per se or in relation to others). The evaluator should refer to additional documentation in order to find an indication of a cause-and-effect link; if this task fails, it should be deduced and/or the evaluator should ask the assistance of experts (human resources and working arrangement).

Each of the classifications by row should be explained by means of an interpretation expressed as an assumption.

A provisional classification could be carried out on the basis of the distinction between three levels of objectives:

• Overall objectives
• Short-term and medium-term intermediate objectives
• Operational objectives

The definition of the logical relationship is paramount in the classification. This task is a question of experience, because the decision of the evaluation team that two events are logically connected depends on the judgement of experts in the field under study, and managers responsible for the implementation of the strategy and policies.

Establish a temporary diagram

The first faithfully reconstructed objectives diagram can be drawn from this first classification. It usually reveals incoherencies that will have to be corrected in accordance with the evaluator's interpretation (these corrections must be indicated in the diagram).

Should the construction of the diagram start with the overall objective
or the operational objectives (objectives describing the attended results)?

Usage has shown that these two types of objectives are more easily identified than intermediate objectives, which are also more difficult to rank. Thus, it is recommended that the roots of the diagram and the extremities of the branches are developed concurrently.

In the diagram, each objective is presented as a box with a heading. The evaluator should also indicate the identification number which refers to the quotation list. Providing the boxes with colours corresponding to the level to which the objective belongs may also be helpful.

Each link should refer to one or more quotations (in accordance with the identification number).

The diagram should also mention the specific means (financial and non financial) used in the implementation of each operational objective.

Identify the authors of the objectives' wording

Before testing the temporary diagram, the evaluator should identify the main and secondary authors of the objectives' wording (writing of a chapter, sector-based contributions, participation in working meetings, etc.).

It may be useful to check the consequence of the wording on the definition of the operational objectives of the programme planning.

Test the temporary diagram

The authors of baseline documentation and additional documents should test the diagram in order to validate the classification of the objectives by rows and links. The aim is to check that the evaluator's interpretations accurately reflect the authors' intentions. If the authors are not available or, with a view to complement their contribution, the evaluator can seek the participation of other actors responsible for the drafting process (both from the writing and the discussion process).

Contacting the authors is usually possible when the documentation is recent and the authors are still in position or contactable in one of the services. This task is challenging when the documentation is old and its authors are not easily identifiable, nor reachable.

At the end of the interviews, the evaluator may also encounter difficulties evaluating what was actually done during the drafting process and the justifications of respondents for the decisions taken.

Establish the final version of the faithfully reconstructed objectives diagram

This version takes into account the opinions collected during the test of the temporary diagram.

It should be considered an accurate report of the initial intentions of the European Commission, taken from the official documentation.

STAGE 4: HOW TO CONVERT THE FAITHFULLY RECONSTRUCTED OBJECTIVES DIAGRAM INTO A LOGICALLY RECONSTRUCTED OBJECTIVES DIAGRAM

Advantages and limitations of a faithfully reconstructed objectives diagram

Advantages of the faithfully reconstructed objectives diagram

Some faithfully reconstructed objectives diagrams display a strong internal coherence: apart from the overall objective or the overall expect impact, each objective 'x' leads to an higher (immediate or not) objective and is supported by one or more subordinate objectives. This is particularly the case when faithfully reconstructed objectives diagrams are extracted from a carefully established logical framework. Faithfully reconstructed objectives diagrams can be deemed as completely logical and stand as logically reconstructed objectives diagrams.

Logical links of a given objective 'x' (in objectives diagrams)

The establishment of a logically reconstructed objectives diagram is not required when the objectives system of the strategy and planning documents is thoroughly and consistently presented.

Limitations of the faithfully reconstructed objectives diagram

Some faithfully reconstructed objectives diagrams may reveal logical defects in strategy or political papers, such as:

• No definition of the overall objective
• Weak relevance of the causal links between objectives
• The objectives of a given row do not result in any objectives or are not linked to subordinate objectives which would have provided information about the implementation process
• Several unclassified objectives

In these cases, the definition of the evaluation thematic scope and the evaluation questions is difficult. The team should therefore draft a logically reconstructed objectives diagram.

Preparation of the logically reconstructed objectives diagram

In order to prepare a comprehensive and coherent objectives diagram, the evaluation team will need all available documentation, its own expertise and, if required, that of experts. This task should be completely transparent (how are the findings presented?): whenever possible, the diagram should display the reclassification of objectives, the changes affecting links between objectives, and the introduction of missing intermediate objectives. Each of these rationalisation operations should be explained in a technical note.

Stage 5: how to construct an effect diagram

Objectives diagrams and intended effect diagrams share the same rules of construction. 

The effect diagram is constructed from the conversion of each of the objectives into the corresponding intended effect.

.

HOW ARE THE DIAGRAMS USED?

Objectives diagrams and effect diagrams are a crucial tool in the organisation stage of the evaluation. They play many roles:

• The objectives provide a description of the strategy (geographic) or the policies (sector-based, thematic)
• The diagrams determine themes for evaluation questions
• They present the strengths and weaknesses of the objectives system to be achieved or the intended effects system in an informative and transparent way

Description and analysis of the strategy

Objectives diagrams demonstrate a synthesis of the geographical strategy displayed in the official documentation. When the evaluation scope covers one or more strategy papers (geographical) or strategic policies (sector-based, thematic), it is recommended that one diagram per document is created (unless there is a logical continuity in the strategy or the policy). 

They illustrate the objectives/effects classification, from the overall objective/impact to projects and programmes scheduled to implement the objective/effects. The logically reconstructed objectives diagramand the effect diagram are informative means to reflect upon:

• Its internal coherence
• The relevance of the objectives/effects to the context (the European Union's overall strategy for country assistance, the main actors' themes or sectors and objectives, the activity's scope and objectives of other donors)
• The appropriateness of the means dedicated to carry out these objectives/effects

The faithfully reconstructed objectives diagram displays the objectives system and provides the evaluator with a first approach to the strategy and policies inner quality. Indeed, an unclear, incomplete and incoherent diagram means a lack of relevance in the forthcoming planning or a lack of faithfulness to the initial objectives system. 

These analyses must be presented in the notes and reports of the evaluation organisation stage. They must also appear as an abstract in the final report.

Definition of the themes for evaluation questions

The previous analyses reveal questions about:

• The faithfulness of the strategy or the policies to the European Union's overall objectives for its country assistance
• The relevance of the strategy and policies to the country's situation, the evaluation's sector or theme, and the main actors' policies (governments in particular)
• The need for complementarities and synergies with other donors
• The consequences of a lack of coherence in the objectives system
• The relevance of the means to the objectives

These questions result in the determination of a range of themes which could be investigated during the following stages of the evaluation, particularly through evaluation questions. 

However, evaluation questions cannot be automatically deduced from these analyses. Other questions can emerge during the organisation stage, formulated by the main strategy implementation operators (since they are the most well-informed actors about the problems encountered during the planning and the implementation stages). However, the number of evaluation questions is limited and additional questions would have to be the subject of a selection process.

HOW ARE THE FINDINGS PRESENTED?

Notes and reports of the organisation stage

Objectives diagrams are established during the organisation stage, where reports and notes should be provided. At this stage, the diagram's construction must be precisely described. 

For the faithfully reconstructed objectives diagram, the sources of the objectives/effects (quotations, references to the original documentation) must be provided. References to documentation, interviews and expertise must support the objectives' location in the diagram, and the assumptions developed during the construction of the diagram must be explained. 

The process through which the logically reconstructed objectives diagram has been extracted from the faithfully reconstructed objectives diagram must also be clearly explained.

Final report

Diagrams should be incorporated into the final report because they are not only stages in the development of the wording of evaluation questions, but also an illustration of a possible analysis of the evaluation strategy. 

The report may include a short presentation of the diagrams, in addition to a full explanation included in the annexes.

Verbal presentations

The evaluation team will need to present its work (methodology and findings) to different types of people (the evaluation reference group, participants in the debriefing session). The objectives diagram or the effect diagram are very efficient tools for this purpose, providing that they are readable without being over-simplistic. 

To do so, a main diagram, and several sub-diagrams developing fundamental sections of the main diagram, should be presented, each of them not exceeding 20 items.

WHAT ARE THE PRECONDITIONS FOR THE ESTABLISHMENT OF AN OBJECTIVES DIAGRAM AND AN EFFECT DIAGRAM?

Human resources and working arrangement

Type of work required for the design of an objectives diagram

Observation :
The number of working days mainly depends on the scope of the evaluation, and thus on the number of programmes to study and diagrams to construct.

Travelling expenses

Strategy papers have been drafted under the responsibility of the head office (DG RELEX or DG DEV). The head office is also responsible for the planning stage. EuropeAid is in charge of the project drafting. This implies that the majority of the useful documentation (the whole baseline documentation) can be found in Brussels. This situation may devolve to Delegations in the future whose responsibilities in this area will grow.

The current needs (2004) in travelling expenses to Brussels

These elements (and the corresponding budgetary components) will have to be reviewed when the Delegations take broader responsibilities for strategy development (2007).

Computer devices

Most of the graphic problems can be addressed with software such as MS PowerPoint

IN COUNTRY EVALUATIONS

.

IS THE USE OF AN OBJECTIVES DIAGRAM AND AN EFFECT DIAGRAM COMPULSORY?

Every donor must implicitly or explicitly explain the decision to provide assistance to a country or a region in a given timeframe. For this purpose, the objectives to be achieved or the intended effects which form the basis for the assistance are described in documents which are regularly published and present the overall objectives and priorities. Such documentation can either display orientations, an individual programme, or both.

Such documents may present programmes which are jointly drafted by the donor and beneficiary. In this case, the interventions under study should be that of the European Commission and of the country partner. The outcomes and intended impacts are the outputs of the implementation of these two types of interventions.

In national programmes

National Indicative Programmes (NIP) can be inserted in the logical framework which can be a single document or correspond to each main sector of intervention. In this case, the evaluation uses the logical frameworks (i.e. objectives and implementation indicators) as the reference for efficiency criteria. Its goal will be to check in what extent the objectives and effect systems have been completed during the implementation of the programme.

In country strategies

Country Strategy Papers (CSP) do not usually rely on an overall logical framework because they cover a long timeframe and are implemented through various successive programmes, each of them supported by a logical framework. Consequently, the evaluators should assemble the objectives and effects system, and draft a diagram for each programme.

WHICH OBJECTIVES AND EFFECTS MUST BE TAKEN INTO ACCOUNT?

In strategies and complex bilateral assistance programmes, the European Commission is the donor and the States and national public institutions are usually its partners. The European Commission and the States have their own overall objectives, corresponding to their own issues. Yet, to a certain extent and degree of detail, the objectives overlap.

In a bilateral assistance contract, three objectives or effect systems can be developed:

• The European Commission's own system, corresponding to strategic issues meant to be resolved with financial assistance (for example, regional instability, weak absorption capacity of the local market, uncontrolled immigration)
• The country partner's own system, relating to major political issues to which the State is confronted (macro-economic unbalances, social instability, weak growth, social or geographical disparities, etc.)
• A common system for the European Commission and its country partner, where similar issues are relevant for both of them and to the fields of the assistance scope (for example, health, SME, irrigation, environment)

Except for participatory evaluations, the evaluation team will prioritise the first and third objectives system.

WHAT ROLE DO OBJECTIVES DIAGRAM AND EFFECT DIAGRAMS PLAY IN COUNTRY EVALUATIONS?

Once completed, the logically reconstructed objectives diagram and the effect diagram provide two parallel approaches to the country strategy evaluation:

• The wording of evaluation questions, which mainly deals with the effectiveness and the effect of programme implementation and indirectly with the efficiency and sustainability of the outputs and their effects
• The internal coherence of the strategy and its relevance relating to the European Union's overall objectives, State policies and the positions taken by international donors

These approaches are illustrated in the following table. The main branch of the logically reconstructed objectives diagram splits into two: the right branch deals with evaluation questions, the left branch with the overall analysis of the strategy.

Role of the objectives diagram in the conduct of an evaluation structured by evaluation questions

Footnotes

Two main aspects of relevance are examined:

• (a) in relation to the overall objectives of the European Union and the European Commission
• (b) in relation to the needs and /or demands of the beneficiary(ies)

The evaluation questions connect the intended effects (results, outcomes, secondary impacts) with the specific and general means which are used to achieve the objectives.

The overall analysis studies the strategy building, i.e.:

• The choice of the intended impact
• The choice of results, outcomes, and secondary impact
• Their relevance regarding the country situation and the main actors' policies
• The efforts to gain complementarity and synergy with other donors

WHICH TYPES OF OBJECTIVES DIAGRAMS OR EFFECT DIAGRAMS CAN BE USED FOR COUNTRY EVALUATIONS?

Two types of diagram, setting the framework for the evaluation, illustrate the objectives of the programme or the strategy to be assessed.

The faithfully reconstructed objectives diagram

The faithfully reconstructed objectives diagram is useful for two reasons:

• It accurately reproduces the objectives system of the logical framework, when it exists. It also highlights, as closely as possible, the objectives system which is more or less explicit in the documentation without a logical framework. Thus, it represents a crucial stage in the development of the logically reconstructed objectives diagram.
• It underlines, when required, the limitation of the objectives system's coherence (see the limitation of the faithfully reconstructed objectives diagram). It fills the gap in ex ante evaluation, and thus appears as an information tool, benefiting the authors of strategy and planning papers.
• It is a schematic illustration of the declarations included in the strategy (CSP, Regional Strategy Papers, RSP) and planning (NIP, RIP) papers relative to the targeted objectives and the various links between them.

The logically reconstructed objectives diagram

The logically reconstructed objectives diagram determines the thematic scope of the evaluation.

It establishes the reconstructed classification of the objectives system as they are presented in logical frameworks or reconstituted in the faithfully reconstructed objectives diagram. It is also the logical reconstitution of the objectives' classification implied in strategy papers (CSP, RSP) and planning papers (NIP, RIP).

It contributes to the assessment of strengths and weaknesses on which the evaluation questions should focus:

• Relevance of the objectives' selection
• Quality of the wording
• Logical links within the hierarchy of objectives

The logically reconstructed objectives diagram is particularly needed because the strategy papers and documentation concerning the planning of geographic assistance do not provide the evaluator with a systematic presentation (with or without a diagram) of the objectives system.

It should be summarised within the final report of the evaluation, and fully presented in the annexes.

Additional observation

Diagrams are not the only tools available to analyse objectives and effects. For example, evaluators frequently use a matrix.

HOW MANY EFFECT DIAGRAMS MUST BE ESTABLISHED?

A single diagram

If the analysis of the situation does not radically evolve during the evaluation and if the activities focus on a single intended overall impact, a single diagram is enough.

Many diagrams, depending on the context

When the strategy evolves, effect diagrams should be constructed for each change in strategy of the European Commission.

When the evaluator cannot determine a single overall impact and when each of the global impact depends on an independent effect system, the evaluator should establish as many diagrams as overall impacts.

Intermediate diagrams

Intermediate diagrams, corresponding to each of the assistance priorities (or sectors) for the period(s) covered by the evaluation, can be useful for the construction of an objectives diagram or effect diagram. They are particularly adapted for programmes with a logical framework provided for each of the priorities.

Intermediate diagrams can be useful for the determination and justification of evaluation questions.

Magnifier diagrams

When overall diagrams are very complex, their readability and information quality may suffer. A simplified overall diagram should be prepared, and detail diagrams should be drawn from the overall diagrams, in order to emphasize some of their specific parts.

WHICH DOCUMENTATION SHOULD BE REFERRED TO IN COUNTRY EVALUATIONS?

Baseline documentation

The baseline documentation includes strategy papers (CSP) and the related indicative programmes (NIP) relevant to the period covered in the evaluation scope.

Bilateral agreements, such as the Association Agreement, Cooperation Agreement and their monitoring documents should also be included, as well as multilateral agreements (the Lome and Cotonou agreements) and their related documents (for example, the Barcelona Declaration).

In most cases, they should be provided to the evaluator at the start of the evaluation.

However, the evaluator should check that:

• All the successive programmes, including the modifications afterwards, are provided
• The latest version provided

Additional documentation

Three types of documents must be available :

• Strategy and planning papers related to European Commission activities at the regional level (RSP and RIP) in the case of a country evaluation
• Official documentation concerning other non-geographic budgetary resources at the disposal of the European Commission's geographic assistance (such as 'horizontal' budget lines and other contributions)
• Specific planning papers dealing with the main areas of the assistance ('concentration' or 'assimilated' sectors)

Other informative documentation

Other documentation may be useful, such as successive versions of the baseline documentation (the date of the final version must be checked), including the various contributions and reactions of the Commission's services, their national partners and other donors (particularly Member States). It is valuable, but often difficult to identify and seldom available or accessible.

.

EXAMPLES

.

• Examples

BIBLIOGRAPHY

.

The logical framework and the objectives tree

• Manuel de Gestion du cycle du projet, Commission Européenne, EuropeAid, mars 2001. [FR]
• Guide récapitulatif des Formations - Gestion du Cycle de Projet, Commission Européenne, EuropeAid, Itad Ltd, Particip GmbH. version 1.1, février 2001. [FR]
• 'Logical framework analysis', BOND Guidance Notes No 4, March 2003.
• 'The Logical Framework Approach (LFA)', A summary of the theory behind the LFA method, Sida - Swedish International Development Co-operation Agency, Draft, June 2002.
• 'The Logical Framework Approach', Handbook for Objectives-oriented Planning, NORAD, 4th edition, January 1999.
• 'The logical framework approach', Australian agency for international aid (AusAid), AusGuide, AusGuidelines, 2000.
• 'Logical frameworks: a critical assessment', Working paper # 264, Des Gasper, Institute of Social Studies, the Hague, December 1997.

Use of the logical framework in evaluations

• 'Project appraisal 2 (4)', Logical Framework Approach to the Monitoring and Evaluation of Agricultural and Rural Development Projects, Coleman G., 1987, p. 251-259.

The other approaches

• Program Theory in Evaluation: Challenges and Opportunities, New Directions for Evaluation, a publication of the American Evaluation Association, Fall 2000.