published
Evaluation methodological approach
Evaluation methodological approach > EN: Methodological bases and approach

EN: Methodological bases and approach

The EU has developed and formalised a methodology for evaluating its external assistance in which priority is given to results and impacts. This page presents the methodological guidelines designed to facilitate the move towards an evaluation practice focused on programmes and strategies in EC Development Cooperation.

  1. This first chapter presents the methodological bases of the evaluation approach:

1421399525_admin-icon-star.png  Methodological bases or what, when and why to evaluate and who is a part of the evaluation:

Subject, timing of the evaluation and intervention cycle, utilisation and roles, as well as specificity of the evaluation compared to monitoring and auditing are described in details.

1421399525_admin-icon-star.png  Evaluation methods or how to evaluate

Intervention strategy, evaluation questions, judgement references, methodological design, data collection, analysis, judgement and quality assurance are presented and explained.

1421399525_admin-icon-star.png  Evaluation tools

The rationale and the use of each evaluation tool are systematically presented.

 

The second and third chapters of these guidelines contain respectively:

  1. The detailed guidelines for Project and programme evaluations

  2. The detailed guidelines for Geographic, thematic and other evaluation

The way development aid is provided is constantly evolving. To reflect this changing context, DEVCO develops specific methodological tools, often in cooperation with international institutions such as the OECD and the World Bank.

In this context, three specific methodological approaches to evaluate budget support programmes, capacity development interventions and gender as a cross cutting issue have been developed:

Evaluation methodological approach > EN: Methodological bases and approach > Disseminating Evaluations

Disseminating Evaluations

Creative Communications for Evaluation Dissemination

This space is created to help you promote evaluation findings in a way that is attractive and communicative by exploring different options, guides and good practice examples at the EU level and externally.

An INTPA/ESS study on EU evaluation dissemination suggests that the most innovative ways to communicate evaluation results, besides organising seminars or webinars, include infographics, briefs, videos, blogs and podcasts, among others.

Please find below some ‘Good Practice Examples’ from different EU offices, findings of the INTPA/ESS study in a dynamic report, and How-To Guides on creating different Dissemination Knowledge Products.

 




 

Background of the study: In the first semester of 2020, the Evaluation Support Service of DG DEVCO conducted a study to examine and document the current situation regarding how the knowledge generated from evaluation of the international development and co-operation interventions is being translated and disseminated to a wide variety of stakeholders to ensure ‘evaluations influence change’, in line with DG DEVCO’s Evaluation Policy (Evaluation Matters, 2014). Related to this, the study captures examples of good dissemination practice within DG DEVCO and non-EU institutions such as the Asian Development Bank, African Development Bank, International Fund for Agricultural Development, World Food Programme, Global Environment FacilityGreen Climate Fund, InsighShareUN Women, UNICEF, UNFPA, Y Care International and Oxfam.

A detailed report on the findings of the study and its recommendations is also available on request. Please write to the ESS team to have a copy of the full report.

Further support: Should you need any support in the management of your evaluations, including advice on ways to best disseminate their results, the ESS helpdesk remains at your full disposal and can be reached at helpdesk@evaluationsupport.eu. Please do not hesitate to contact them and share your examples of evaluation dissemination: we would be happy to publish them in this C4D space!

 

Evaluation methodological approach > EN: Methodological bases and approach > Disseminating Evaluations > Disseminating knowledge generated by evaluations. Some examples from the EU.

Disseminating knowledge generated by evaluations. Some examples from the EU.

This space has been created to showcase good-practice examples of Evaluation Dissemination Knowledge Products (EDKPs) by different EU Delegations and Headquarters.

These knowledge products are produced to further encourage the use of knowledge generated through evaluations by key decision-makers in the EU and overseas, and to support our aim to be more transparent and accountable to the wider EU public. These knowledge products are created in easily digestible and simple language and the full reports can be provided upon request. 

For EU staff members: we would like to invite you to help us populate this section by sharing your evaluation dissemination knowledge products with the Helpdesk@evaluationsupport.eu

Examples of these knowledge products can be accessed by clicking on the different categories shown below.

 





 

 

 

Evaluation methodological approach > EN: Methodological bases and approach > Disseminating Evaluations > Disseminating knowledge generated by evaluations. Some examples from the EU. > Briefs

Briefs

Here you will find examples of evaluation reports that were summarised in briefs, among which is the multi-country study “La voix des jeunes au Sahel”, with a global brief and six country-specific editions. Full evaluation reports can be provided upon request.

Please access the briefs by clicking on the thumbnails below.

 

Evaluation Brief of European Union’s Cooperation with Myanmar (EN) Evaluation of the EU State Building Contracts (2012-2018) (EN) Evaluation of the EU CPPB Support (2013-2017) (EN)

La voix des jeunes au Sahel. Total. (FR) La voix des jeunes au Sahel. Tchad. (FR) La voix des jeunes au Sahel. Burkina Faso. (FR)



La voix des jeunes au Sahel. Mali. (FR) La voix des jeunes au Sahel. Mauritanie. (FR) La voix des jeunes au Sahel. Niger. (FR)



Evaluation of the TCP on Renewable Natural Resources Bhutan. (EN) Evaluation of the Capacity Building
Program for Primary and Secondary
Education in Iraq (EN)
 


 

 

Evaluation methodological approach > EN: Methodological bases and approach > Disseminating Evaluations > Disseminating knowledge generated by evaluations. Some examples from the EU. > Infographics

Infographics

Here you will find examples of evaluation reports that were summarised in an infographic. Full evaluation reports can be provided upon request.

Please access the infographics by clicking on the thumbnails below.

 

Evaluation of the EU's regional development cooperation with Latin America (2009-2017) (EN)    

 

 
Evaluation methodological approach > EN: Methodological bases and approach > Disseminating Evaluations > Disseminating knowledge generated by evaluations. Some examples from the EU. > Reader-friendly reports

Reader-friendly reports

Here you will find examples of evaluation reports that were edited in a communicative and reader-friendly manner. Full evaluation reports can be provided upon request.

Please access the reports by clicking on the thumbnails below.

 

Ex-post evaluation of the CBTPSD project in Malawi. (EN) Evaluation of the EU SRSP's PEACE Programme. (EN) Analysis of the PARAQ Programme. (FR)
 

 

Evaluation methodological approach > EN: Methodological bases and approach > Disseminating Evaluations > Disseminating knowledge generated by evaluations. Some examples from the EU. > Video

Video

Here you will find examples of EU evaluations of which the findings have been summarised in a short, communicative video. Full evaluation reports can be provided upon request.

Please access the videos by clicking on the thumbnails below.

 

An infrastructure project in Ivory Coast, 2019 (FR)    

 

 
Evaluation methodological approach > EN: Methodological bases and approach > Disseminating Evaluations > How-to Guides on Evaluation Dissemination

How-to Guides on Evaluation Dissemination

Through analysing a large number of evaluation dissemination products produced recently and information collated from interviews with relevant institutions, this study produced a series of five “How-to guidelines” on the most frequently used dissemination products (Videos, Infographics, Evaluation Briefs, Podcasts and Blogs) to improve and support evaluation dissemination practice. These How-to-Guides can be accessed by clicking on the following:

 




     




 

Evaluation methodological approach > EN: Methodological bases and approach > Disseminating Evaluations > How-to Guides on Evaluation Dissemination > How-to Guide on Evaluation Dissemination Seminars

How-to Guide on Evaluation Dissemination Seminars

A dissemination seminar is a way to share evaluation findings during an online or face-to-face meeting with key stakeholders or a wider audience. It is a good way to get feedback before the final report is written and ensure that findings are supported by relevant parties.

These guidelines support your work with practical tips on organising a seminar to communicate the learning from your evaluation.

You may access the document by clicking on the image below.


 

Evaluation methodological approach > EN: Methodological bases and approach > Disseminating Evaluations > How-to Guides on Evaluation Dissemination > How-to guide on producing blogs for evaluation dissemination

How-to guide on producing blogs for evaluation dissemination

A blog provides an accessible and user-friendly format to communicate your evaluation results. You can share findings from the evaluation as a whole, or specific aspects such as key learning, recommendations or experience of using different methods and tools. All of these may incite your audience to read the full report.

The DEVCO/ESS study to understand the different ways in which evaluation results are communicated reveals that 7 out of the 18 institutions consulted use blogs to disseminate their evaluations. Blogs do not require particular preparation in terms of budgeting and planning, but there are some writing principles to respect.

These guidelines provide you with practical tips, examples, and references to additional resources to get you started on evaluation blogs. Click on the document below and let us know how you get on with evaluating your evaluation results through blog posts. 

You may access the document by clicking on the image below.


Evaluation methodological approach > EN: Methodological bases and approach > Disseminating Evaluations > How-to Guides on Evaluation Dissemination > How-to guide on producing briefs for evaluation dissemination

How-to guide on producing briefs for evaluation dissemination

Evaluation brief is a way to share key findings on a particular thematic area through an attractive 2-4 page summary. The DEVCO/ESS study to understand the different ways in which evaluation results are communicated reveals that 9 out of the 18 institutions consulted use briefs to disseminate their evaluations. They are relatively easy to make and do not require important budget or preparation. Critical, though, is to make sure you provide enough context and adapt the language for your audience to understand the summary.

These guidelines support your work with practical tips, examples, and references to additional resources to communicate the learning from your evaluation.

You may access the document by clicking on the image below.


Evaluation methodological approach > EN: Methodological bases and approach > Disseminating Evaluations > How-to Guides on Evaluation Dissemination > How-to guide on producing infographics for evaluation dissemination

How-to guide on producing infographics for evaluation dissemination

Infographics provide an increasingly popular way of sharing your evaluation findings. The DEVCO/ESS study to understand the different ways in which evaluation results are communicated reveals that 11 out of the 18 institutions consulted use infographics to disseminate their evaluations.

Infographics are relatively low cost and do not require particular planning at the start of the evaluation process. What they do require is the definition of clear key messages that convey a story.

These guidelines provide you with practical tips, examples, and references to additional resources to communicate the
learning from your evaluation.

You may access the document by clicking on the image below.


Evaluation methodological approach > EN: Methodological bases and approach > Disseminating Evaluations > How-to Guides on Evaluation Dissemination > How-to guide on producing podcasts for evaluation dissemination

How-to guide on producing podcasts for evaluation dissemination

Podcasts are an increasingly popular way to communicate evaluation results. The DEVCO/ESS study to understand the different ways in which evaluation results are communicated reveals that 6 out of the 18 institutions consulted use podcasts to disseminate their evaluations. Podcasts can be engaging; they are easy to share and provide convenient access to knowledge.

These guidelines provide you with practical tips, examples, and references to additional resources to get you started on evaluation podcasts. Have a look and let us know how you get on with promoting your evaluation results through the power Podcasts. 

You may access the document by clicking on the image below.


Evaluation methodological approach > EN: Methodological bases and approach > Disseminating Evaluations > How-to Guides on Evaluation Dissemination > How-to guide on producing video for evaluation dissemination

How-to guide on producing video for evaluation dissemination

Evaluation Video for Dissemination

Video is an engaging way of sharing your evaluation findings. The DEVCO/ESS study to
understand the different ways in which evaluation results are communicated reveals
that 12 out of the 17 institutions consulted use video to disseminate their evaluations. It
requires preparation in terms of budgeting and planning, as well as in terms of reflecting on
the story you want to tell. Whose story will it be? Is it an account of programme success
or an evaluative journey? These guidelines provide you with practical tips, examples, and
references to additional resources to communicate the learning from your evaluation.

Evaluation methodological approach > EN: Methodological bases and approach > Disseminating Evaluations > Evaluation Dissemination - Dynamic report

Evaluation Dissemination - Dynamic report

Here you can download and read the dynamic interactive PDF report that has been produced to disseminate the findings of the study on evaluation dissemination at DG DEVCO, EU Delegations and non-EU organisations around the world such as the Asian Development BankAfrican Development BankInternational Fund for Agricultural DevelopmentWorld Food Programme, Global Environment FacilityGreen Climate FundUN WomenUNICEFUNFPAY Care International and Oxfam.

Please note that this Dynamic Interactive Report has been produced from the ESS study, ‘A Report on the current practices of evaluation dissemination within the European Commission DG INTPA, and other non-EU institutions’. This report can be accessed via this weblink.

Should you need any support in the management of your evaluations, including advice on ways to best disseminate their results, the ESS helpdesk remains at your full disposal and can be reached at helpdesk@evaluationsupport.eu. Please do not hesitate to contact them and share your examples of evaluation dissemination: we would be happy to publish them in this C4D space!

Evaluation methodological approach > EN: Methodological bases and approach > Methodological bases
Evaluation methodological approach > EN: Methodological bases and approach > Methodological bases > Subject

Subject

.

This section is structured as follows:

.

EVALUATION SCOPE

.

What is this?

The scope of the evaluation is everything that is judged. It is defined in terms of various dimensions such as the territory concerned, the period under consideration and the regulatory framework, for example:

  • All the funds allocated by the European Union to Latin America since 1998.
  • Local development actions planned in Albania over the past four years.

The central scope of the evaluation matches the very contour of the intervention that is under evaluation. A second broader perimeter specifies the related actions and the elements from the context to be taken into account, especially for the analysis of external coherence/complementarity. This is the extended scope, as opposed to the central scope. 

The central scope is specified in the terms of reference and the extended scope in the inception report.

Why specify the evaluation scope?

  • To identify the commissioning body's expectations better and define clearer priorities for the evaluation
  • For the evaluation team to focus on priorities and not to waste its resources on areas of secondary interest.

How to delimit the central scope

- in terms of a territorial perimeter

  • the world (global evaluation)
  • a region
  • a country
  • one or more areas within a country.

- in terms of the period under consideration

  • the entire period during which the evaluated intervention was implemented (case of an ex post evaluation)
  • or only that which was implemented during the years [n-4 to n-2] so that the evaluation team can observe real effects in the field
  • or by including the preceding and/or next intervention.

- in terms of the regulatory framework

  • a project
  • a support programme to sector policy
  • a country strategy
  • a regional co-operation agreement
  • an instrument.

- in terms of the sector

Only where relevant:

  • an entire sector or field of intervention (education, agriculture, etc.)
  • or only a sub-sector
  • or a set of sectors.

How to delimit the extended scope

- in terms of related policies

  • The interventions of national authorities or other donors covering the same territories
  • … or targeting the same groups
  • … or addressing the same problems.

- in terms of co-funding

  • interventions co-funded with national authorities
  • interventions co-funded with other donors.

Recommendations

  • Limit the scope to allow more in-depth analysis of effects and a better understanding of the phenomena.
  • In the case of co-funded, multisectoral interventions, select a portion of the institutional and sectoral scope for a high-quality evaluation.

A warning

Be careful not to confuse a sample (of individuals, firms, etc.) selected for survey purposes, with the evaluation scope. The sample is used to collect data and to infer results extrapolated for the entire survey group. In this case, the evaluation scope is the surveyed group. 

Don't confuse scope and question, Examples:

  • scope = the entire country strategy; question = the effect on gender equality (question)
  • scope = multi-donor programme to support the national education policy; question = sustainability of the impact on the management of the education system.

A sector (e.g. education) can constitute a delineation of the scope, but the same does not apply to the cross-cutting issues (e.g. equal opportunities) that resemble questions. For example, a programme to support the national education policy (scope) will be evaluated from the point of view of gender equality (question).

.

EVALUATION OF PROJECTS, PROGRAMMES AND STRATEGIES

.

What does this mean?

The evaluation may concern a simple intervention such as a project that produces effects directly in the field, or a more complex intervention that produces its effects through other interventions at a lower level. The questions will obviously differ, depending on what is evaluated.

Why make the distinction?

  • To adjust the questions and the process to the nature of the evaluated intervention.
  • To clearly understand that a complex intervention is not evaluated with the same methods as a simple intervention.

What distinguishes the three types of evaluation?

- Evaluation of a project

A project is an indivisible operation, delimited in terms of schedule and budget, and usually placed under the responsibility of a single operator. The simplest projects to evaluate are homogeneous, that is, characterised by:

  • One main activity
  • Implemented in a single context
  • Intended for a single group
  • A single direct result expected for that group
  • A single expected impact at a more global level.

The evaluation of a simple project is facilitated by the fact of focusing on a single cause (the main activity) and a single effect (expected result for the targeted group). Owing to this simplicity, techniques such as questionnaires or comparison groups can be used. It is also possible to ask many different questions and to answer them satisfactorily. 

A project evaluation often focuses on questions of interest to managers, operators and the targeted group.

- Evaluation of a programme

A programme is a set of simple, homogeneous interventions grouped together to attain global objectives. It is delimited in terms of schedule and budget and usually placed under the responsibility of a monitoring committee. 

A homogeneous programme is made up of parallel and similar projects. Evaluation of a homogeneous programme can consist of syntheses of project evaluations or extrapolation from a sample of projects. 

So-called "integrated" programmes encompass heterogeneous but complementary interventions with a common objective. Such programmes are complex objects in so far as they are designed with:

  • Several activities (subsidies, direct investments, technical assistance, etc.)
  • Implemented in several contexts (different geographical areas, different sectors, etc.)
  • Targeted at several groups
  • Expected to produce multiple results for the different groups
  • But expected to generate one common impact at a more global level.

The evaluation of a complex, heterogeneous programme is complicated by the fact that multiple causes (activities and instruments) and effects need to be studied, not to mention the synergy effects between the different components of the programme, which in fact justifies the programme's very existence. 
Due to this complexity, some traditional evaluation techniques such as questionnaires or comparison groups cannot be used, although approaches based on case studies remain applicable. 
To avoid adding to the complexity and to preserve the quality of answers, it is necessary to limit the number of questions processed. 
Evaluation of a complex programme is neither the sum nor the synthesis of evaluations of components of the programme. It focuses on questions that are relevant to the programme managers but are seldom if ever addressed at lower levels (effects of synergy, relevance of the sharing of resources between components, contribution to meeting the goal).

- Evaluation of a strategy

A strategy covers a wide set of simple or complex, weakly inter-related interventions. It is defined by priorities rather than as a set of clearly delimited interventions. 

Unlike a programme, a strategy generally has several objectives with differing degrees of priority. 

Evaluating a strategy involves the same difficulties as those encountered in the evaluation of a complex programme. The same approaches are used to deal with these difficulties:

  • Focus the evaluation on a few key questions
  • Ask questions that specifically concern the strategy and not only components of the strategy.
  • Answer these questions with more than simply a combination or synthesis of evaluations of the components.

.

SECTORS, THEMES AND CROSS-CUTTING ISSUES

.

What does this mean?

Public interventions are usually classified by sector or theme. A particular intervention or set of interventions can be evaluated from the point of view of a cross-cutting issue. This is also referred to as a thematic evaluation. These terms need to be clarified.

Examples

How to make the distinction?

- Sectors

Interventions are classified in a particular sector according to the nature of the activities and outputs. For instance:

  • Support to training primary school teachers will be classified in the Education sector
  • Advice provided to groups of farmers will be classified in the Agriculture sector

- Cross-cutting issues

A cross-cutting (or horizontal) issue is defined in relation to impacts and not outputs, as shown by the two examples in the table below:

Output Teachers trained Advice provided to groups of farmers
Sector Education Agriculture
Impact Girls benefit from a better access to education New breeding practices that limit desertification
Cross-cutting issue Gender Environment

- Themes

The term "theme" is sometimes used in the sense of a sector and sometimes in the sense of a cross-cutting issue. It is therefore advisable to redefine it every time it is used.

Recommendations

  • Take cross-cutting issues into consideration in the preparation of evaluation questions. Questions on cross-cutting issue s are particularly relevant in the evaluation of a complex intervention such as a strategy or programme.

Warning:

The same term can denote a sector or a cross-cutting issue , as shown in the table below:

  Human rights = sector
= secteur
Human rights = cross-cutting issue
= aspect transversal
Output Staff of the law courts trained in human rights Creation of access to Internet in poor rural areas
Sector Human rights NICT
Impact   Information on human rights violations circulates more easily
Cross-cutting issue   Human rights
Evaluation methodological approach > EN: Methodological bases and approach > Methodological bases > Timing of the evaluation and intervention cycle

Timing of the evaluation and intervention cycle

.

This section is structured as follows:

.


TIMING OF THE EVALUATION

.

What does this mean?

An evaluation can be performed before, during or after the evaluated intervention. Depending on the timing, its purpose and use will differ.

What is the point?

  • To optimise the resources allocated to the evaluation by launching it at the time it is likely to have the most added value.
  • To meet the needs of the main users of the evaluation at the most appropriate time.
  • To ensure that a critical mass of results and impacts are already materialised in the field and they are ready for data collection.
  • To avoid conflict with the concomitant exercises of review or audit.

Different times of an evaluation

Ex ante

An ex ante evaluation is performed before adopting or implementing the intervention. It gives support to the intervention design and contributes to ensuring the design quality. It is concerned with the following points:

  • Need to satisfy in the short or in the long run.
  • Objectives to be met.
  • Expected results and necessary indicators for their evaluation.
  • Added value of the community intervention.
  • Risks linked to the proposals.
  • Open alternative options.
  • Lessons from similar experiences already undergone.

It aims at having a direct influence on the decisions upstream from the implementation, to the extent that it transposes lessons from past experiences into the framework of the new intervention.

Mid-term or final

An evaluation during or at the end of the implementation is intended to draw lessons from the first years of the intervention implementation and to adjust the contents of the ongoing intervention in relation to realities in the field and/or contextual developments. It often includes a report on outputs and an analysis of the first results and impacts achieved. It aims at improving the intervention under way and its conclusions may be supported by observations in the field.

Ex post

The ex post evaluation is performed right after or a long time after completion of implementation. It is mainly concerned with checking achieved impacts, identifying and judging unexpected impacts and assessing the sustainability of the intervention's benefits. 

It enables to detect the real changes in the field and, if the changes occur soon enough, they can be analysed to estimate those that are attributable to the intervention. 

The ex post evaluation often aims to report to the institutions that have allocated the resources. Likewise, it helps to transfer acquired experiences to other countries or sectors. 

Recommendations

  • Mid-term evaluation offers a good compromise between utility and reliability, especially if it takes place in the second half of the cycle.
  • If the evaluation is to draw on the experience of previous programmes in order to improve future ones, it is recommended to establish a multiannual evaluation plan covering several programming cycles.

.


EVALUATION AND THE INTERVENTION CYCLE

.

What does this mean?

The evaluation can be scheduled:

  • For the beginning of the cycle, in the design phase;
  • During the cycle (mid-term or at the end of the implementation phase);
  • After the end of the cycle.

Is there a best time to evaluate or should three evaluations be performed for each cycle?

What is the purpose?

Many interventions are characterised by successive cycles and show a relative continuity between cycles. 

A new cycle can start before the intervention of the previous cycle has yielded all its effects. 

In light of this, several cycles need to be considered when choosing the timing of the evaluation, in order to:

  • Draw conclusions from an observation of the effects of the intervention in cycle "n-1".
  • Feed into reflection on the intervention in cycle "n-1".
  • Observe the implementation of cycle "n" and rapidly provide feedback.

How to take the cycles into account?

For defining the purposes of the evaluation:

  • If the evaluation takes place at the beginning of the cycle, it verifies the relevance and coherence of the objectives of intervention "n" by analysing needs and also the effects of previous interventions "n-1" and "n-2".
  • If the evaluation takes place in the middle of the cycle, it aims to improve intervention "n" by taking into account its first observable effects, and also by analysing the effects of intervention "n-1".
  • If the evaluation takes place at the end of the cycle, it draws conclusions from the analysis of the first observable effects and also from the effects of interventions "n-1", with a view to preparing the strategy and objectives of the next intervention "n+1".

For choosing evaluation questions

  • Retrospective questions relating to the impacts and sustainability of preceding interventions.
  • Concomitant questions relating to the direct results and relevance of the intervention under way.
  • Prospective questions relating to the strategy of the interventions in the next cycle.

Recommendations

Where possible, ensure that the evaluation ends about a year before the end of the cycle. At that stage it will still be possible and useful to adjust the implementation because it always lasts for a year or two after the end of the cycle. It will also be possible to assist in planning the intervention strategy of the following cycle. Finally, it is not too late to perform a sound analysis of the impacts of the previous cycle's intervention.

Evaluation methodological approach > EN: Methodological bases and approach > Methodological bases > Utilisation

Utilisation

.

This section is structured as follows:

.

USERS OF AN EVALUATION

.

Who do we mean?

Evaluation is intended for a variety of users:

  • policy-makers and intervention's designers
  • managers, partners and operators involved in the implementation
  • institutions that granted funds and to which the managers of the intervention are accountable
  • public authorities that conduct related or similar interventions
  • actors in civil society
  • experts.

What is the purpose?

To optimise the usefulness of the evaluation for the various partners, and especially:

  • To ensure that the evaluation meets the expectations of the targeted users, in a way and at a time that fulfills their needs
  • To ensure that the evaluation has the required credibility vis-à-vis the targeted users.

Policy-makers and designers

Policy-makers and designers use the evaluation to prepare the launching of new interventions, the reform of existing interventions, the choice of strategic orientations, and decisions on allocation of budgetary, human and other resources, etc. 

They need information that:

  • is directly exploitable in the decision-making process,
  • arrives on time
  • answers their questions, clearly, concisely and reliably.

They are interested in strategic issues, external coherence and global impacts, which constitute the ultimate goal of the intervention.

Managers, partners and operators

The managers are responsible for the implementation and monitoring of the intervention, from headquarters to the field. The actors closest to the public are the operators. Field level operators may either belong to the EC or to partner organisations sharing the responsibility of implementation. 

They use evaluation findings about the results of their action as a feedback. 

They need information that arrives as early as possible to remedy problems or validate changes. They are able to interpret complex and technical messages. 

They are interested in the direct results of the intervention, in the needs and behaviour of the targeted group, and in interactions between partners.

Other actors

The institutions that funded an intervention expect accountability. This applies to Parliament or the Council of Ministers, but also to all the co-funders. The taxpayers and citizens are also addressees of an evaluation. 

The public authorities that conduct related or similar interventions are potential users of the evaluation, especially in the form of transfer of lessons learned. The same applies to the expert networks concerned by the intervention. 

Finally, an evaluation is likely to be used by the actors in civil society, especially those representing the interests of the beneficiary groups.

Recommendations

  • From the launch phase, draw up an inventory of the potential users and classify them in the above categories.
  • Question key informants to understand the users' expectations.
  • Choose the expectations to focus on, especially in relation to the challenges and responsibilities of the institution that initiated the evaluation.
  • Draw up a communication and dissemination plan suited to the main users targeted.
  • Take into account the different levels of information in relation to the users: strategic for the decision-makers, technical and operational for the managers, general for outside actors.

.

TYPES OF USE

.

What does this mean?

An evaluation can be used:

  • As an aid in decision-making.
  • As an aid in making judgements and informing public opinion.
  • To promote knowledge and understanding.

These three types of use are not mutually exclusive. It is necessary to understand in order to judge, and to judge in order to decide.

What is the purpose?

To optimise the usefulness of an evaluation, that is:

  • To choose and draft the questions in relation to the expected use.
  • To adjust the schedule and dissemination in relation to the expected use.

Assisting decision making

The evaluation may be undertaken for the benefit of those who have to decide or to negotiate an adjustment or reform of the evaluated intervention. In that case it is used to adjust the implementation, to design the next intervention cycle or to redefine political orientations. 

To facilitate this type of use, also called feedback, the evaluation questions must be asked in relation to the decision-makers' expectations and to their planned decision-making agenda at the time the report is submitted. 

The evaluation can aid decision-making in two different ways:

  • By formulating conclusions independently, and then proposing recommendations. This type of evaluation is referred to as "summative".
  • By favouring the involvement of the decision-makers concerned, or at least their close collaborators, with a view to creating a phenomenon of take-up or direct feedback. This type of evaluation is referred to as "formative".

Evaluations may assist decision-making in different ways, depending on the context of the decision:

  • Recommendations may be made to the managers and/or designers of the intervention. Evaluations that favour this type of use are referred to as "managerial".
  • Recommendations may be made to all the partners and co-funders of the intervention. Evaluations put to this type of use are referred to as "partnership".
  • Finally, an evaluation may be conceived as an aid to negotiation and problem-solving between a wider range of stakeholders, including interest groups and actors in civil society. Evaluations used for this purpose are referred to as "pluralistic".

Assisting the formulation of judgements

The evaluation may help users to shape their opinion on the merits of the intervention. 

The formulation of an overall assessment is particularly useful for accountability purposes. In this case, the evaluation examines the merits of the intervention in relation to the different points of view (summative evaluation). It answers questions that are important for the funding institutions. The report is accessible to the general public. The independence of the evaluation and the transparency of the judgement are highlighted. 

In this instance, particular attention is paid to the definition of judgement criteria (also called "reasoned assessment criteria"). Yet the judgement itself is definite only when the final report is submitted and its conclusions are discussed. Using the evaluation for accountability purposes therefore means having to wait for the end of the process.

Knowing and understanding

Apart from assisting in making decisions and formulating judgements, which are the two main forms of use, the evaluation may also enable users to learn from the intervention, to better understand what works and what does not, and to accumulate knowledge.. Indirectly, it contributes to transferring knowledge thereby acquired, to the benefit of professional networks that may not have a direct link with the evaluated intervention. 

Unlike feedback, which directly concerns those responsible for the evaluated intervention, the transfer of lessons is an indirect process that takes place through networks of experts both within and outside the European Commission. 

Capitalising on knowledge often starts during the evaluation process, through the experts who belong to the evaluation team or reference group. However, the transfer of lessons learnt may only occur after the final report has been delivered. A key step in this perspective is presentation of the evaluation in specialised networks, in the form of seminars or technical articles.

Recommendations

From the outset, the evaluation manager should prioritise one or more types of use, and then optimise the evaluation process in order to make it as "user friendly" as possible, for instance in adjusting:

  • Membership of the reference group.
  • Evaluation questions.
  • The dissemination strategy.

.

EVALUATION AND DECISION MAKING

.

What is this about?

Evaluation provides feedback and thus facilitates decision-making, for instance in adjusting the implementation of the intervention, designing the next cycle, or helping to redefine political priorities. In this context the formulation and follow-up of recommendations are key steps in the process.

What is the purpose?

Facilitating future decision-making is generally the main type of use of evaluation. Decision-makers' needs have, therefore, to be taken into account throughout the evaluation process to increase the chances of the evaluation being useful to them. 

What is the link between evaluation and decision-making?

Some evaluations are designed primarily to provide information for management decisions or for reform of the evaluated intervention. They are intended for operational actors in the field, management services, and the authorities responsible for the intervention or their partners. In this perspective, mid-term evaluation is to be preferred and careful attention needs to be paid to the formulation and follow-up of recommendations. These evaluations are referred to as formative. 

Other evaluations are designed primarily to learn lessons from the experience and to serve decision-making in other contexts. In this perspective, ex post evaluation is the most appropriate and careful attention must be paid to the formulation and transferability of the lessons learned. These evaluations are referred to as summative. 

Yet one has to be realistic: decision-makers will not necessarily follow the recommendations and lessons. The decision-making process almost always involves many actors and multiple factors, evaluation being only one of them.

Advice for performing a decision-making oriented evaluation

  • Hold targeted interviews, from the outset, in order to establish exactly what decision-makers expect and to anticipate the decision-making agenda.
  • Target the questions and determine the evaluation schedule in relation to the decision-making process.
  • Involve individuals in the reference group whose point of view is close to decision-makers'.
  • Organise quality assessment, the drafting of documents and their dissemination with a view to decision-makers' needs.
  • Emphasise the quality of recommendations and verify their operational feasibility from the decision-makers' point of view.
  • Stick to the schedule to allow timely feedback in the decision-making process.
  • Follow-up recommendations by asking the decision-makers concerned to respond rapidly to them and after one year to report on decisions made accordingly.

.

EVALUATION AND KNOWLEDGE TRANSFER

.

What is this about?

Evaluation is a learning-intensive exercise in so far as lessons learned from experience can be capitalised on, and knowledge acquired can be transferred and reused by actors who have no direct link with the evaluated intervention. In this context the identification of good practices and transferable lessons is a key step.

What is the purpose of knowledge transfer?

  • Without an effort to capitalise on and manage knowledge, organisations tend to forget the lessons learned and to repeat costly efforts.
  • Evaluation reveals and validates certain knowledge of the actors involved in the interventions, which may be of use to people working in other countries, institutions or sectors.

Draft the report to promote the accumulation of knowledge

The only pages of a voluminous evaluation report that are oriented towards knowledge transfer are those concerning good practices and lessons learned. It is therefore important to identify them and to draft them with that in mind. 

Identifying a good practice means judging that it produced the expected effects in a particularly effective, efficient or sustainable way. 

Identifying a transferable lesson means judging that a given practice generally succeeds (or fails) to produce the expected effects in a given context. 

The summary of the evaluation report highlights the main lessons learned with a view to facilitating their accumulation and transfer. For each lesson, references to the body of the report serve to specify:

  • The exact content of the lesson learned.
  • The soundness of the conclusions on which it is based.
  • The context in which knowledge was learned and the factors of context that make it transferable or not.

Mobilise available knowledge

Certain public institutions or professional networks accumulate knowledge in Intranet or Internet databases. Knowledge may also be accumulated informally through expert networks that build on lessons learned in a particular sector or on a particular topic. In the latter case, the members of the network try to validate the knowledge before capitalising on it, through meta-evaluations or expert panels. 

Not only does evaluation contribute towards the accumulation of knowledge, it also facilitates the circulation of the acquired knowledge. The evaluation team mobilises available expertise and documentation to provide an initial partial answer to the questions asked. If the transferable lessons have been learned through other evaluations and properly capitalised, the evaluation team identifies and reuses them. 

.

DISSEMINATION OF THE EVALUATION

.

What does this mean?

Dissemination concerns the final evaluation report, as well as all other means of publicising the conclusions, the lessons learned and the recommendations. Dissemination activities target the services of the Commission, European Institutions, external partners, networks of experts, the media and the wider public.

What is the purpose?

The dissemination process promotes the use of the evaluation if it is done well. It serves to:

  • Transmit the key messages of the evaluation to decision-makers, designers, managers, operators and partners concerned by the evaluated intervention (feedback).
  • Report to the authorities and institutions concerned (accountability).
  • Further knowledge within professional networks.
  • Influence opinions throughout society at large.

Which measures need to be taken?

  • Plan dissemination when drafting the terms of reference, especially by specifying how the report will be published and what the evaluation team's role will be in that phase.
  • Throughout the process, keep in mind the quality of the evaluation and its products (report, summary, annexes), and formally assess the quality at the end of the process.
  • In the last evaluation reference group meeting, identify the main messages delivered by the evaluation and the targeted audiences.
  • After approval of the report, finalise the communication plan by choosing the messages to be highlighted and the most suitable information channels for each audience.
  • Ensure the necessary cooperation for the implementation of the dissemination plan and divide the work and responsibilities between the evaluation manager, the team (check that this is part of its mission and is specifically mentioned in the terms of reference) and the members of the reference group.

Which channels for dissemination?

The evaluation report is disseminated on the Internet and is thus accessible to all audiences. More active dissemination is also undertaken for specific audiences:

  • The report and/or its summary are sent to the services concerned and to the partners.
  • A one-page summary is written specifically for the hierarchy of the service that managed the evaluation. It highlights the main conclusions and recommendations.
  • A summary is also published on the relevant Intranet sites, with a link to the report.
  • A two to three pages abstract is sent to the OECD for publication on its website. This summary is intended for the international development aid community. It highlights the lessons learned if they are transferable.
  • One or more articles may be written for the general public or specialised networks.
  • Finally, the report may be presented in meetings, workshops or seminars.

Recommendations

  • Draw up the rules for quoting names in the report in relation to the intended dissemination.
  • Draw up a budget for the communication plan at the launching stage of the evaluation (mention whether the external evaluation team has to include it in its proposal).

.

DOCUMENTS PRESENTING THE EVALUATION

.

What does this mean?

Since full-length evaluation reports are available on the Internet, it is advisable to disseminate one or more shorter documents that are suited to the different audiences and present the evaluation in an attractive and accessible way.

What is the purpose?

These documents are intended to:

  • Inform decision-makers of the main conclusions and recommendations.
  • Inform the wider public, experts and the media of the existence and content of the evaluation report.
  • Facilitate knowledge transfer, particularly best practice and lessons learned.
  • Promote accountability and democratic use of evaluation.
  • Encourage people to consult the report on the Internet.

Formats and uses of the different documents

The executive summary is one of the starting points for drafting presentations. It is usually three pages long and no particular audience is targeted in the content. Beyond the executive summary, various documents are or may be produced:

  • A one-page note focused on conclusions and recommendations at a strategic level and intended for policy-makers.
  • A two-page abstract published by the OECD (EvInfo) to inform the development aid community. As far as possible, this abstract mentions transferable lessons learned.
  • A paragraph presenting the evaluation in synthesis documents.
  • A short article intended for a particular audience.

Advice for drafting a presentation document

- Choose the orientation of the document

  • Define the target: is the document for the public at large or for specialists and, if the latter, which ones? 
  • Select a few messages to highlight in the title and sub-titles, in relation to the targeted audience. 
  • Adjust the length of the document and the style to suit the targeted audience.

- Describe the evaluated intervention

  • Who initiated and financed the intervention? When?
  • Budget and typical outputs.
  • Expected effects and rationale.

- Describe the evaluation

  • Who initiated the evaluation? Why?
  • Evaluation scope.
  • Main evaluation questions asked.

- Main messages

  • Findings and new knowledge acquired.
  • Conclusions.
  • Transferable lessons and replicable good practices.
  • Recommendations.

- Method

Describe the methodological design in a non-technical way and make a statement about the soundness of the main messages. 

If one aspect of the method is of particular interest to the targeted audience, add a more technical boxed-in section.

Recommendations

  • Avoid the technical jargon of evaluation, the specialised vocabulary of the sector of intervention, and administrative acronyms, especially if the document is intended for the general public.
  • Focus on a few key messages and propose hyperlinks towards the executive summary and the report available on the Commission's website.

.

PRESENTING AN EVALUATION IN A SEMINAR

.

What is the purpose?

In an environment with an over-abundance of written information, an effective way of reaching potential users consists in presenting the evaluation results in meetings or seminars. 

An oral presentation of the evaluation helps to:

  • Facilitate knowledge transfer, especially best practice and lessons learned in professional networks
  • Inform people who will relay messages to decision-makers or to a broader audience
  • Draw the attention of the institutions to which one is accountable and reinforce the democratic use of evaluations
  • Discuss the conclusions.

Practical advice

A short presentation (10 to 20 minutes) is enough for the main points but more time needs to be left for questions (20 to 40 minutes). 

The presentation covers the following points:

- The evaluated intervention

  • Who conducted the intervention, when, with which resources? What were the outputs, objectives and rationale?
  • Main lines of the intervention logic

- The evaluation

  • Who decided on the evaluation and why?
  • Who carried out the evaluation and who is responsible for the conclusions?
  • Which part of the intervention was evaluated and which questions were asked?

- Messages resulting from the evaluation

A few particularly important messages from the point of view of the people participating in the meeting or seminar (key data, findings, conclusions, transferable lessons and/or recommendations).

- Strengths and weaknesses of the messages

Explanation of the methodology employed and reasons for which a particular message is sound (valid) or fragile. Recommendations for using messages if they are fragile.

Recommendations

  • Adapt the level of language to the audience, e.g. without jargon for a mixed audience, more technical for an audience of sector experts.
  • Indicate how to access the final report and the summary, especially via Internet.
Evaluation methodological approach > EN: Methodological bases and approach > Methodological bases > Roles

Roles

.

This section is structured as follows:

.

This Evaluation methodological approach is generally intended to Evaluation managers and to external evaluation teams. This is why we have visually distingueshed their respective tasks, as follows : 
1420993037_em.png The evaluation manager
1421077476_1418988054_teams.png The external evaluation team
 
These icons are used in main sections of guidelines for Project and programme evaluations and Strategic, geographic and other complex evaluations.  
 

.

THE DISTRIBUTION OF ROLES

.

What does this mean?

The three key players are the evaluation manager, the reference group and the external evaluation team. The distribution of roles is specific to each phase and each stage of the evaluation.

All the actors in the evaluation process have to know what their role and responsibilities are, for example: Who is responsible for the recommendations? Who has to assess the quality of the report?

What is the purpose?

  • To mobilise the different actors, depending on their specific competencies and resources.
  • To avoid deadlocks due to possible confusion regarding roles.
  • To preserve the impartiality of the evaluation and avoid biased judgements.

Who does what?

The three key players in an evaluation are the evaluation manager, the reference group and the external evaluation team. Their roles differ, depending on the following phases and steps:

- Decision to evaluate

The decision is taken by the authorities responsible for the intervention and is reported in a document such as:

  • the annual evaluation programme
  • the document scheduling the intervention.

The decision refers to an evaluation framework that defines the rules of the game. If necessary, it completes this framework.

- Preparation (phase 0)

Constitution of the reference group:

  • by the evaluation manager.

Drafting of the Terms of Reference:

  • by the evaluation manager.
  • after consulting the reference group.

Selection or approval of the external evaluation team:

  • by the evaluation manager,
  • after consulting the reference group.

- Desk (1)

Clarification of the intervention logic:

  • by the external evaluation team,
  • after interaction with the evaluation manager and the reference group.

Choice and drafting of evaluation questions:

  • by the evaluation team,
  • validation by the reference group,
  • arbitration by the evaluation manager if necessary.

Definition of the judgement criteria (also called reasoned assessment):

  • by the external evaluation team,
  • approval by the reference group,
  • arbitration by the evaluation manager if necessary.

Choice of indicators:

  • by the external evaluation team,
  • after consulting the reference group.

Design of the method (data sources, tools and work programme):

  • by the external evaluation team,
  • after consulting the reference group.

Collecting available data at the head office of the Commission and/or the Delegation (s):

  • by the external evaluation team,
  • if necessary, with the aid of members of the reference group for access to informants and in interaction with the evaluation manager.

- Field (2)

Data collection:

  • by the external evaluation team,
  • if necessary, with the aid of members of the reference group for access to informants, and in interaction with the evaluation manager in cases of serious problems.

- Synthesis (3)

Report, including the conclusions and recommendations:

  • by the evaluation team,
  • after interaction with the reference group.

Verification of the quality of the report:

  • by the evaluation manager, possibly with an external support,
  • after consulting members of the reference group.

- Dissemination and follow up of recommendations (4)

Internal and external dissemination:

  • by the evaluation manager

Use of the conclusions and recommendations:

  • by the Commission's services concerned
  • with follow up by the evaluation manager.

.

THE EVALUATION MANAGER

.

Who is this?

The evaluation manager is a member of the service in charge of the evaluation. He/she manages the entire evaluation process, from A to Z on behalf of the commissioning body.

Why appoint an evaluation manager?

  • To ensure consistency throughout the evaluation process, from the terms of reference to the dissemination of the report and the follow-up of recommendations.
  • To be the contact person for administrative issues and to coordinate the activity of the different actors (reference group and evaluation team).
  • To organise, supervise and assess the quality of the different phases of the work.
  • To ensure the smooth running of the evaluation.

What does he / she do?

  • He / she forms the reference group after consulting the heads of the services concerned.
  • He / she drafts the terms of reference after consulting the reference group members.
  • He / she engages the external evaluation team after consulting the reference group.
  • He / she helps the evaluation team to get access to information and informants.
  • He / she organises the discussion and approval of conclusions and recommendations in interaction with the reference group and the evaluation team.
  • He / she performs a methodological quality assessment of the reports, double checked by a second person.
  • He / she disseminates the evaluation to the different actors concerned.
  • He / she monitors whether the recommendations are taken into account in interaction with the authorities concerned.

Recommendations

  • Appoint a deputy to help the evaluation manager on a daily basis and to stand in when he/she is absent.
  • Appoint an evaluation manager who knows both his/her institution and the evaluation methods.
  • Ensure that the evaluation manager is not too close to the managers of the evaluated intervention.

.

THE REFERENCE GROUP

.

What is this?

A small group gathering the services concerned by the intervention and possibly other stakeholders. One or more experts may also be invited to participate.

The reference group is an interface between the evaluation manager and the external evaluation team. Its members help the external evaluation team to identify data sources and to access them. They validate the evaluation questions, and discuss the conclusions and recommendations.

The reference group allows the variety of points of view on the evaluated intervention to be expressed.

What is its role?

Reference group members:

  • Discuss and comment on the terms of reference drawn up by the evaluation manager.
  • Aggregate and summarise the views of the Commission services and act as an interface between the evaluation team and the services, thereby supplementing bilateral contacts.
  • Ensure that the evaluation team has access to and consults all information sources and documentation on activities undertaken.
  • Validate the evaluation questions.
  • Discuss and comment on notes and reports produced by the evaluation team. Comments by individual members of the reference group are compiled by the evaluation manager and subsequently transmitted to the evaluation team.
  • Assist in feedback of the findings, conclusions and recommendations from the evaluation.

Whom to invite and how?

Different types of actor can be invited to participate in the reference group. If the evaluation is managed at headquarters, membership includes:

  • Services of the Commission concerned by the intervention, including the Delegation concerned in case of a country/region level evaluation.
  • Specialists on a sector or a cross-cutting issue within the Commission.
  • Embassy of the partner country in the case of a country level evaluation.

If the evaluation is managed in the partner country, membership may extend to:

  • Government services.
  • Selected development partners.
  • Selected civil society organisations.
  • Experts

Experience has shown that it is preferable for the group to be limited in size (10 to 12 members) if it is to function effectively.

After identifying the services, institutions or organisations to invite to the reference group, the head of the service managing the evaluation sends them an invitation to:

  • Officially announce the launching of the evaluation
  • Request that they appoint one person as a member of the reference group.

When should it be asked to meet and how should it be involved?

The reference group is generally involved in the following way:

Stage Meeting Role
Terms of reference   Comments
Engagement of the external evaluation team   Comments
Evaluation questions 1 Validation
Inception report   Comments
First phase report (desk) 2 Discussion, comments
Field phase debriefing 3 Discussion
Version 1 of the final report, conclusions and recommendations 4 Discussion, comments

Opening the evaluation process

In addition to inputs made by the reference group, the evaluation may benefit from a wider variety of views through other channels like:

  • Consultation of key stakeholders by the evaluation team in the first stage of the process
  • Gathering of participatory workshops, focus groups or expert panel(s) during the evaluation process
  • Discussion of the draft evaluation report in a widely attended seminar in the partner country (when relevant).

Experts invited to the reference group

What does this mean?

One or more members of the reference group may be invited as a resource person, for their expertise.

Why invite them?

  • To orientate the evaluation work towards the acquisition of new knowledge and to learn from others' experience.
  • To describe the point of view of certain stakeholders, where relevant.
  • To contribute towards quality assessment of work in their field of expertise.
  • To facilitate knowledge transfer.

Who should be invited?

  • The experts invited may or may not belong to the services of the Commission.
  • It is preferable for them to be familiar with the entire evaluation field.
  • As part of the global network they can mobilise knowledge about the entire development aid community.

Charging a reference group meeting

What to do?

  • Prepare a precise agenda and ensure that you adhere to it: (see: inception meeting, first phase meeting, debriefing meeting, final discussion meeting)
  • Ensure that the discussions are balanced, impartial and constructive.
  • Draft clear and synthetic minutes that highlight the points debated and report on points of agreement (this task is preferably accomplished by the deputy manager).

What to do in case of disagreement?

In case of lasting disagreement on the evaluation method, the service in charge of the evaluation arbitrates. If necessary, the decision is taken by the head of the Unit.

.

THE EXTERNAL EVALUATION TEAM

.

What does this mean?

The external evaluation team is responsible for data collection and analyses and for the evaluation report, including the formulation of judgements (also called reasoned assessment) and the drafting of conclusions and recommendations. It interacts with the reference group and the evaluation manager. It provides evaluation services to the commissioning body under contract.

The members of the external evaluation team are independent of the organisations that participated in the design and implementation of the evaluated intervention. They belong to national or international research organisations or consultancy firms, or are experts working independently. Where possible, some members of the team are from the country or countries targeted by the evaluated intervention.

Why contract an external evaluation team?

  • To ensure the independence, impartiality and credibility of the evaluation process.
  • To provide expertise on evaluation methods and techniques.
  • To provide expertise in the sector(s) concerned.

How to select the team?

The external evaluation team is selected on the basis of proposals via different procedures, e.g. pluri-annual evaluation service contract or specific call for tenders.

The evaluation manager engages the team after consulting members of the reference group.

The following criteria are used as guidelines:

  • Relevant experience in evaluation.
  • Knowledge of the evaluated development aid modalities.
  • Knowledge of the evaluation scope (e.g. sector, country).
  • Presence of local experts and consultants.
  • Team leader's managerial skills (organisation, coordination, dialogue with the client, etc.).
  • Team's independence as regards the evaluated intervention and absence of conflicting interests.
  • Evaluation skills and particularly the ability to formulate judgements, to draw up conclusions and recommendations and to draft syntheses.
  • Mastery of data collection and analysis tools.
  • Diversity of team members' profiles and expertise.
  • Price.

The evaluation manager sets the relative weight of criteria and specifies it in the terms of reference.

Recommendations

  • Involve local consultants, as far as possible, to promote the development of local capacity (development of national evaluation expertise) and benefit from their close knowledge of the field.

Evaluation capacity

What does this mean?

Having evaluation capacity implies sufficient knowledge and experience to:

  • Guarantee impartiality in the choice of questions and the formulation of conclusions
  • Clarify the intervention logic
  • Design and apply a data collection and analysis method suited to the questions and context
  • Guarantee the reliability of data and the validity of analyses
  • Treat value judgements in a structured, transparent and impartial way
  • Distinguish the findings, conclusions, lessons and recommendations
  • Present evaluation work synthetically
  • Optimise the usefulness of the evaluation for different audiences.

Evaluation capacity is required both:

  • in the service managing the evaluation
  • and in the external evaluation team.

How to assess professional experience?

An evaluation professional meets most of the following criteria:

  • Varied evaluation experience (regions, themes, sectors, tools)
  • Training and experience in evaluation methods and tools
  • Active participation in one or more professional evaluation networks
  • Practical experience.

.

THE STAKEHOLDERS

.

What does this mean?

The stakeholders are individuals, groups or organisations that have responsibilities and/or direct or indirect interests in the evaluated intervention. They may be affected by the intervention or not.

Why identify them?

  • To take their points of view into consideration
  • To benefit from their expertise and knowledge
  • To encourage them to use the evaluation.

How to take them into account?

  • Identify the different stakeholders from the outset, on the basis of available documentation and interviews.
  • Identify their expectations and take them into account when drafting the evaluation questions.
  • Analyse each one's possible contributions (information, expertise, contribution to the interpretation of data).
  • Choose an option to benefit from their contribution, for example: individual interviews, participatory workshops, focus groups, expert panel(s), integration in the reference group, etc.
  • Choose the modalities of dissemination of the evaluation results to ensure that they reach the stakeholders.

Recommendations

  • Talk to the stakeholders to know what their needs and priorities are.
  • Involve the stakeholders in the interpretation of information to avoid mistakes and reach sounder conclusions.
Evaluation methodological approach > EN: Methodological bases and approach > Methodological bases > Specificity of the evaluation compared to monitoring and auditing

Specificity of the evaluation compared to monitoring and auditing

.

This section is structured as follows:

.


What does this mean?

.

Monitoring, auditing and evaluation are all exercises in sound public management. They are similar and complementary but cover different issues.

All three contain value judgements but the judgement references are not the same. 

.


What is the purpose?

.

  • To avoid misunderstandings and deceptions. If the external evaluation team deals with questions that fall under auditing or monitoring, it may not answer them correctly because it lacks the appropriate resources.
  • To reformulate a question formulated in a spirit of auditing or monitoring so that it becomes an evaluation question.
  • So that informants adopt an attitude of cooperation with the evaluation, knowing that the relationship with an evaluator is not the same as one with an auditor.
  • To promote synergy between the three exercises.

.


What distinguishes the three exercises?

.

Subject of study

  • An audit (in the traditional sense) verifies the legality and regularity of the implementation of inputs.
  • Monitoring verifies the sound management of interventions and informs on the progress and use of inputs, and on the progress of outputs and results. Monitoring produces systematic information with a short periodicity, unlike auditing and evaluation.
  • Evaluation judges the design and implementation of the intervention from the standpoint of results and impacts mainly, whether they were expected or not.

Judgement criteria

  • Auditing judges on the basis of criteria that are known and clarified in advance (budgets, regulations, management standards).
  • Monitoring judges progress in the use of inputs and the delivery of outputs, in relation to operational objectives. The quality of outputs is judged along professional standards.
  • Evaluation judges the consequences of the intervention in relation to social needs, economic challenges, environmental problems, etc. It refers to objectives but often has to clarify them. The judgement criteria that it uses are seldom established in advance.

Time scale

  • Auditing is carried out periodically, according to pre-established rules, or ad hoc if a risk analysis justifies it.
  • Monitoring is a continuous process of collection, with frequent reports (one to four per year).
  • Evaluation is an in-depth exercise that mobilises complex tools for collecting and analysing data, and that cannot be carried out annually.

Use

  • Audit is intended to detect and prevent irregularities.
  • Monitoring gives warnings and allows rapid feedback in case of problem. It is also used for reporting on performance throughout the management chain, from the field operators to the top managers.
  • Evaluation furthers understanding on the effects of interventions. It makes recommendations for reform, suggests strategic and political options, and puts forward proposals to adjust the implementation.

The required professional qualities

Each exercise requires an appropriate professional capacity and distinct core competencies:

  • The auditor has legal and accounting competencies. His/her action is authoritative.
  • The monitoring manager has sound professional knowledge in the field of the intervention and management skills. His/her action is of a supervisory nature.
  • The evaluation team has investigative skills and sector-specific expertise. It requires the cooperation of the actors concerned by the intervention to collect relevant data, interpret it correctly and formulate a legitimate judgement.

.


How to ensure complementarity between the three exercises?

.

A good monitoring system considerably facilitates the evaluation. When an evaluation deals with intended effects, the first thing to do is to check whether these effects have been monitored, which indicators have been used, and whether baseline data have been recorded. In turn, evaluation reports often include suggestions for improving the monitoring system.

Auditing and monitoring may detect apparent success or failure and recommend that the evaluation deepens the causes and effects analysis, clarifies the judgment criteria, or validates a good practice.

An evaluation can show that the right application of a legal or professional standard is a factor of either effectiveness or ineffectiveness, and therefore provides feedback into auditing or monitoring practices.

.


Recommendations and clarification

.

  • If illegal practices or bad management are assumed, then it is better to opt for an audit, to stabilise the situation and to postpone the evaluation.
  • If monitoring information is absent or unreliable, it complicates evaluation but does not preclude it. The insufficiency of monitoring should never be used as a reason to refuse to deal with an evaluation question.
  • The auditing practice described here is the traditional practice of regulatory audit. Sometimes professional auditors may be requested to judge results and impacts and to answer questions of an evaluative nature. This practice, often called performance auditing, can become equivalent to evaluation. In this case the auditors adapt their attitudes, judgement criteria and professional competencies. The absence of a clear boundary between performance auditing and evaluation is often a cause of confusion.
  • The monitoring practice described here is the traditional one. Traditional monitoring systems are increasingly integrated into performance assessment frameworks. Performance monitoring involves the setting of performance targets in terms of outputs and results, the systematic measurement of progress towards targets, periodic performance-oriented discussions, and incentives for good performance.
  • The monitoring practice described here is also different from the EC's Result Oriented Monitoring (ROM). The ROM system rates all external aid projects and programmes with a view to the achievement of intended results and impacts. However, the assessment is not continuous and does not rely on systematically quantified indicators.
Evaluation methodological approach > EN: Methodological bases and approach > Evaluation methods
Evaluation methodological approach > EN: Methodological bases and approach > Evaluation methods > Intervention strategy

Intervention strategy

.

This section is structured as follows:

.

RATIONALE OF THE INTERVENTION

.

What is this?

The rationale of an intervention is to address the needs, problems or issues that are considered to be priorities in a given context, and that cannot be addressed more effectively in another way.

It is in the programming stage that the rationale of an intervention must be justified. At the evaluation stage it is enough to note the main points or to redefine them if the programming documents lack precision.

The evaluation identifies the reasons for which the priorities have been chosen, for example: priorities of the policy in which the intervention takes place, urgency of the needs to be satisfied, and comparative advantages compared to alternative options.

What is the purpose?

  • To help in defining the criteria of the relevance family.
  • To present the intervention in the evaluation report.

How to clarify it?

.

- Reviewing the problems and responses

The evaluation report succinctly sets out the following:

  • Context of the intervention when it was initiated
  • Main problems diagnosed (needs, issues)
  • Why the institution responsible for the intervention and its partners (where relevant) are in the best position to solve the problems diagnosed.
  • Why certain strategic options have been chosen rather than others.

The evaluation team searches for this information in the official documents that instituted the intervention and in the preparatory design work. It completes its understanding with interviews with the key informants. 

Certain evaluation questions may concern problems addressed by the intervention, for example:

  • Are there alternative options for solving the problems identified?
  • Has the nature of the problems that justified the intervention changed?
  • What is the precise extent of the needs justifying the intervention?

- Reviewing the rationale

Where relevant, the evaluation report highlights the following:

  • Justification of the fact that the needs, problems and issues cannot be solved ad hoc by private initiatives.
  • Justification of the fact that they cannot be solved more effectively by other public initiatives.

The evaluation team looks for this information in the official documents that instituted the intervention. It completes its understanding of the rationale by interviews with key informants. 

Certain evaluation questions may concern the rationale of the intervention, for example:

  • Could the institution that initiated the intervention have chosen other partners to address these problems?
  • Would the intervention have been more effective had it been run by other public institutions?

Recommendations

  • The official documents often focus on the strategic options that were finally selected. If we want to know what the alternatives were, it may be useful to hold interviews with key informants, whilst remaining wary of possible biases.

.

INTERVENTION LOGIC

.

What is this?

All the activities and expected effects (outputs, results and impacts) of an intervention, as well as the assumptions that explain how the activities will lead to the effects in the context of the intervention.

The intervention logic may be "faithful" to the programming documents and to the documents establishing the policy to which the intervention is related. In this case, the expected effects are inferred from the stated objectives in the official documents.

When the intervention logic is reconstructed during the evaluation, implicitly expected effects that were not mentioned in the initial documents may be taken into account. The fact that this is no longer a "faithful logic" must then be mentioned. The "faithful" approach is relevant when the objectives are expressed precisely and in a verifiable way. The other option is preferable if objectives are too vague or ambiguous.

The intervention logic often evolves over time. In such cases, it is justified to reconstruct it several times, for successive periods.

The intervention logic is a useful simplification of reality, but one has to bear in mind the complexity of the real world. In addition to the reconstruction of the intervention logic, it is useful to identify the main external factors that condition or limit the implementation and effects. One also has to remember that real causal explanations are often more complex than initial assumptions.

What is the purpose?

  • To help to clarify the objectives and to translate them into a hierarchy of expected effects so that they can be evaluated.
  • To suggest evaluation questions about these effects.
  • To help to assess the internal coherence of the intervention.

How can it be reconstructed?

- Collect and analyse reference documents:

  • Collect the official documents establishing the intervention and allocating resources.
  • Identify the main activities.
  • Identify the objectives.
  • Translate the objectives into expected results and impacts.
  • Connect the activities to the expected impacts by reconstructing the logical cause-and-effect relations (if… then…).
  • Check that cause-and-effect relation are logical, i.e. considered as plausible in the light of available knowledge.
  • Discuss the reconstructed logic with a few key informants (designers and managers) and with the experts of the policy domain/country concerned.
  • Present and discuss the intervention logic in a reference group meeting.

The most common presentations

.

- Logical framework

This technique consists in producing a presentation in the form of a matrix that specifies the objectives (first column), how they should be verified (second column: indicators, and third column: sources), and the main assumptions on the external factors (fourth column).
The lines of the logical framework have a simplified presentation of the intervention logic, on only four rows: activities, results, purpose (i.e. direct benefit for the targeted group) and goal.
This representation is adequate for a simple intervention, like a project or well-targeted programme. However, it cannot fully grasp the complexity of a heterogeneous intervention such as an integrated rural development programme, a country strategy or support for a sector policy.
Due to its simplified nature, the logical framework allows for specifying the indicators that should be used at each row, as well as most external factors.

- Objectives diagram

This technique consists in the identification of officially stated objectives and a graphical presentation of the logical relations between objectives, from the most operational to the most global. The intervention logic is represented in the form of boxes and arrows.
A particular form of representation is the objectives tree. It is applied in the case where each objective on a lower rank contributes towards a single objective on a higher rank.
Unlike the logical framework, the diagram can have as long chains of objectives as necessary. It can be used to highlight synergies and complex relations between objectives (except in the case of the objectives tree).
This representation is appropriate for complex interventions such as integrated programmes, country strategies or sector policies.

- Diagram of expected effects

This technique is similar to the diagram of objectives since it also builds upon officially stated objectives. However, the objectives are translated into expected effects before being presented as a diagram. By translating objectives into expected effects, more concrete and easily verifiable concepts can be worked on.

Recommendations

  • Always re-examine the intervention logic. In the ideal world the intervention logic should be part of the intervention design and should be available at the time of the evaluation in the form of a framework or diagram. In the real world these points should be verified.
  • In the case of complex interventions (and especially country level evaluations), it is useful to produce several detailed diagrams and a synthesis diagram. A highly complex diagram cannot be used.
  • Beware of the risk of ex post rationalisation. To reduce this risk, clearly explain the reconstruction process and the options chosen.

.

OBJECTIVES

.

What are these?

Objectives are the initial and formal statements on what is to be achieved by the intervention. They may relate to outputs, direct results or impacts. 

Objectives may be expressed in a verifiable way, which specifies clearly the expected achievements. They may be expressed in more political or rhetoric way, making difficult to connect them to a precise achievement. 

The evaluation process may uncover implicit objectives, i.e. cause-and-effect assumptions that necessarily explain how an objective may be achieved.

What is the point?

Identifying and clarifying objectives is a necessary step towards:

  • Reconstructing the intervention logic.
  • Asking questions and defining evaluation criteria from the families of relevance and effectiveness.
  • Judging internal coherence.

How to clarify them?

- Examining the clarity of objectives

The evaluation identifies the objectives and, for each one, sets out the following:

  • statement of the objective
  • possible ambiguities in some of the terms and clarifications proposed
  • means proposed for checking whether the objective has been achieved
  • date at which the objective has to be achieved.

The evaluation team searches for the objectives in the official documents that instituted the intervention and in the documents framing it at a higher political or strategic level. 

When the statement of an objective contains ambiguities, the evaluation team proposes clarifications based on interviews with key informants.

- Examining the hierarchy of objectives

The evaluation ranks the objectives of the intervention:

  • global objectives correspond to the rationale of the intervention,
  • operational objectives correspond to the products and services delivered by the intervention (outputs),
  • one or more levels of intermediate objectives can be identified.

The reconstruction of the intervention logic helps to differentiate the levels of objectives as well as the effects, direct finding s and impacts corresponding to them. 

Some evaluation questions may concern the articulation of objectives, e.g.:

  • Is there a high level of internal coherence between the operational objectives and the more global objectives?

Recommendations

If the intervention is framed by numerous documents on a political or strategic level, or if these documents also contain many objectives, the evaluation team is faced with an over-abundance of objectives. Rather than presenting them all, the evaluation team underlines this fact and concentrates on the most important ones in the context of the evaluated intervention. 

There is a continuum between more or less explicit objectives:

  • an objective clearly stated in the official documents
  • an ambiguous objective clarified with the stakeholders' help
  • an intermediate effect that has to be obtained if an objective is to be met (implicit objective)
  • an effect expected by the stakeholders but not initially anticipated, even implicitly

All these elements warrant explanation in the report, but the status of each one must be specified.

.

RELATED POLICIES

.

What does this mean?

The idea is to identify the main related interventions and to situate the evaluated intervention in relation to them.

What is the purpose?

  • To help ask questions belonging to the coherence/complementarity and relevance families
  • If necessary, to examine the quality of the intervention design and especially the fact that the objectives are complementary to those of the other related policies.

How to proceed?

The evaluation report briefly sets out the following:

  • Main national and international institutions active in the same region and sector and/or targeted at the same group.
  • Main policies implemented by these institutions in the same region, in the same sector and/or targeted at the same group.

This analysis offers a brief insight of the main related policies, if possible with their starting and finishing dates, the tools used, the groups and objectives aimed for, and the resources allocated. 

Certain evaluation questions may concern related policies for a more in-depth examination, for example:

  • Is the intervention consistent with and complementary to the policies and priorities of the partner country and/or other Community policies?
  • Has the implementation been coordinated with the actions of the other sponsors and has the complementarity been improved?
  • To what extent is there value added when the intervention is conducted at EC rather than member-State level?
Evaluation methodological approach > EN: Methodological bases and approach > Evaluation methods > Evaluation questions
Evaluation methodological approach > EN: Methodological bases and approach > Evaluation methods > Evaluation questions > Focus the evaluation on key questions

Focus the evaluation on key questions

.

Structure de la section:

WHAT IS ABOUT THE PURPOSE?

.

There are technical limitations that make it impossible to answer multiple questions or, more precisely, to provide quality answers to an excessive number of questions. This guide recommends a maximum of ten questions.

How to choose the questions?

.

- Identify questions

A first version of the evaluation questions is proposed on the basis of:

  • The analysis of the intervention logic.
  • The analysis of the intervention rationale.
  • Issues that justified the decision to launch the evaluation.
  • Issues to be studied, as stated in the terms of reference.
  • Questions raised in the ex ante evaluation, where relevant.

In a second version, the list and wording of the evaluation questions also take into account:

  • Issues raised by key informants at the start of the evaluation.
  • Expectations of the members of the reference group.

- Assess the potential usefulness of answers

Assuming that a question will be properly answered, it is necessary to assess the potential usefulness of the answer, by considering the following points:

  • Who is to use the answer?
  • What is the expected use: knowledge, negotiation, decision-making, communication?
  • Will the answer arrive in time to be used?
  • Is the answer not already known?
  • Is there not another study (audit, review) underway, likely to provide the answer?

If the choice of questions has to be discussed in a meeting, it may be useful to classify them in three categories of potential utility: higher, medium, lower.

- Check that nothing important has been overlooked

Experience has shown that it is most harmful to the quality of the evaluation if the following type of question is left out:

  • Questions on efficiency and sustainability.
  • Questions concerning negative effects, especially if those effects concern underprivileged groups.
  • Questions concerning very long-term effects.

- Assess the feasibility of questions

The feasibility (evaluability) of a question should be examined, but always after its usefulness. For this purpose the following should be consulted:

  • The service managing the intervention.
  • One or more experts in the field.
  • One or more evaluation professionals.

If the choice of questions has to be discussed in a meeting, it may be useful to classify them in three categories:

  • Strong probability of obtaining a quality answer.
  • Average probability.
  • Weak probability.

If a question is potentially very useful but difficult to answer, check whether a similar question would not be easier and equally useful. For example, if a question concerns a relatively distant or global impact, its feasibility could probably be improved by focusing on the immediately preceding impact in the intervention logic.

- Discuss the choice of questions

The choice of questions is discussed at the inception meeting. 

The selection is more likely to be successful if potential users have been consulted and have agreed on the selected questions, and if no legitimate point of view has been censored.

.

REASONS FOR SELECTING A QUESTION

.

Because someone raised it

Someone who proposes a question tends to cooperate in answering it and in accepting the conclusions. 

It is therefore preferable to select questions clearly requested by the actors concerned, for example:

  • Authorities or services of the Commission, especially those who participate in the reference group.
  • Key informants consulted by the evaluation manager or the evaluation team.

An actor may ask a question primarily with the intention of influencing or even obstructing the action of another actor. The potential usefulness of this type of question has to be examined carefully.

Because it is useful

A question is particularly useful if:

  • The intervention or one of its aspects is innovative and several actors expect a validation.
  • A decision is going to be taken and the conclusions may arrive in time to help in taking that decision.
  • A public debate is planned and the conclusions may be ready in time to feed into the debate.

Because the answer is not known

A question is useless if:

  • Another evaluation, an audit or a study has already answered it.
  • It has already been asked in many other evaluations and has always had the same answer.

It may nevertheless be useful to ask the question again if the answer requires verification.

Assessing the overall intervention through a limited number of questions

Focusing on questions does not prevent one from drawing conclusions on the intervention as a whole. On the contrary, it makes it possible to formulate an overall assessment which builds upon professional data collection and analysis, and avoids the risk of being superficial and impressionistic. 

This can be explained with the analogy of oil exploration. One cannot discover oil by just looking at the surface of the earth. Oil seekers need to explore the underground. The same applies to an intervention being evaluated. The surface of things is visible through reporting, monitoring information, and change in indicators, but what needs to be discovered remains invisible, e.g. the EC's contribution to changes, sustainability, etc. 

Evaluation questions can be compared with the oil seekers' exploratory wells. Each evaluation question provides a narrow but in-depth view into what is usually invisible. By synthesising what has been learnt by answering the questions, it becomes possible to provide an overall assessment of the intervention. The process can be compared to that of mapping oil fields after an exploratory drilling campaign.

QUESTIONS AND COMPLEXITY OF EVALUATION

Why work with a limited number of questions?

Focusing an evaluation on a few key questions is all the more necessary when the intervention concerned is multidimensional and when the evaluation itself is multidimensional. In that case, if one wanted to evaluate all the dimensions of the aid and all the dimensions of the evaluation, the work would be extremely costly or very superficial. It is therefore necessary to make choices.

Multidimensional interventions

An intervention is multidimensional if it concerns several sectors, applies several instruments, and targets several objectives, population groups and/or regions.

.

Example of multidimensional intervention (EC aid at country level)
Sectors Education, transport, water, agriculture, health, trade.
Instruments Global financial support, sectoral financial support, projects, etc.
Objectives Reduction of poverty, ……, fair access to primary schooling, ….., development of public management capacities, …. etc.
Target groups Pupils, firms, farmers, communities, women, etc.
Regions The entire country, poor regions, underprivileged urban areas, etc.

.

Multidimensional evaluations

An evaluation is multidimensional if it refers to several families of evaluation criteria and covers several cross-cutting issues and/or neighbouring policies.

.

Example of multidimensional evaluation (EC aid at country level
Criteria Relevance, effectiveness, efficiency, sustainability, impact, Community value added, coherence/complementarity.
Cross-cutting issues Gender, environment, good governance, etc.
Related policies Other EC policies (refugees, trade, agriculture, fishing, etc.)
Policies of the partner country and other sponsors in the sectors concerned.
Evaluation methodological approach > EN: Methodological bases and approach > Evaluation methods > Evaluation questions > Preparing an evaluation question

Preparing an evaluation question

.

This section is structured as follows :

.

Why is it important?

.

The questions serve to concentrate the work on a limited number of points in order to ensure that the conclusions are useful and of a high quality. They therefore have to be carefully prepared and worded with precision.

Ensure that the answer to the question will be useful

As far as possible, the evaluation questions are proposed together with a comments on the following points:

  • Which users will be interested in the answer to the question?
  • How will they use it?
  • Considering the time needed to finalise the evaluation, will the answer to the question arrive in time to meet users' expectations?

If there is uncertainty on the usefulness of the question, it is better to exclude it and to concentrate the evaluation on other more useful questions.

.

SPECIFY THE NATURE OF THE EXPECTED USE

.

Question for knowing and/or understanding

A question for knowing, understanding and/or estimating the effects of the intervention, for example:

  • Has the intervention contributed towards generating effect X and, if so, in what way?
  • To what extent has the intervention helped to generate effect X and in what way?
  • Which groups have benefited the most from the intervention and in what way?
  • What types of know-how have the former trainees acquired?

Questions of this nature reveal new aspects of the intervention. They help to understand the effects and impact mechanisms, and raise the level of knowledge.

Question for judging

A question for formulating or helping to formulate a judgement on the evaluated intervention, for example:

  • Has the intervention contributed towards generating effect X satisfactorily compared to the objectives? In this example the question implies an 'effectiveness' judgement.
  • To what extent has the intervention contributed to generate effect X at a reasonable cost?
  • To what extent have the actions funded by the Commission improved gender equality?
  • Were the causes of shortcomings in municipal management sufficiently analysed and clarified ?

Questions of this nature allow users to judge the merits of the intervention and to recognise good and unsatisfactory practices. They use the evaluation to communicate on the intervention, positively or negatively.

Question for deciding

A question for showing how the intervention can be improved, for example:

  • Has a particular way of implementing the intervention contributed better towards generating effect X as expected?
  • Has the use of a particular implementation modality contributed to generate effect X more sustainably ?
  • To what extent did the distribution and coordination of work among the partners improve the sustainability of progress regarding food safety?
  • Was the choice of a strategy based on occupational training rather than schooling relevant to obtain the creation of new tourist activities?
  • To what extent did the choice of infrastructure projects contribute towards improving the accessibility of the most disadvantaged areas?

Questions of this nature lead to recommendations based on lessons from experience. The answers serve to prepare reforms or adjustments.

Choosing one of three options

The three types of question are not exclusive. On the contrary, there is a progression in the nature of the questions:

  • in order to judge one first has to know
  • in order to decide one first has to know and then to judge.

If all the questions of the same evaluation have no purpose other than furthering knowledge and understanding, the exercise is more a study or a piece of research than an evaluation.

The nature of use has different levels: to decide, one has to have judged, and to judge one has to have understood. It is therefore enough to draft the question in relation to the highest level of use (question for decision-making or judging). Uses on a lower level can be considered as sub-questions. For example the question might be:

  • Has the intervention contributed towards generating effect X at a satisfactory level compared to the objectives?

and a sub-question would be:

  • Has the intervention contributed towards generating effect X and, if so, in what way?

Recommendations

  • Prepare several questions of a different nature within the same evaluation.
  • Avoid having a list of questions that concern knowledge only, for this would constitute research or a study rather than an evaluation.

.

ENSURE THAT THE QUESTION CONCERNS EVALUATION

.

Before drafting a question, ensure that it does not concern audit or monitoring

  • If the question concerns only compliance with rules and procedures, it is an audit rather than an evaluation question.
  • If the question covers only the progress of outputs, it is a monitoring question.

If a question concerns audit or monitoring, there are two options:

  • Exclude the question and ask for it to be included in another exercise.
  • Amend the question so that it has an evaluative dimension.

Avoid auditing and monitoring questions

Evaluation, auditing and monitoring do not serve the same purpose. Consequently, the questions asked are not the same in each exercise. It is important to check whether a question is relevant to evaluation and, where necessary, to amend it so that it does not primarily concern auditing nor monitoring.

Amending an auditing question

  • How were the regulations applied during the implementation of the intervention?

This question is limited to the verification of the legality and regularity of the implementation of a project, which is a matter of auditing. In an evaluation it would be relevant rather to ask whether the application of regulations was a particular factor of effectiveness or ineffectiveness.

  • To what extent did the application of regulations favour or hinder Effect X?

Amending a monitoring question

  • Did the pace of outputs correspond to the original schedule?

This question concerns only the programme outputs, which is a matter of monitoring. In an evaluation it would be relevant to ask whether the quality of outputs is a particular factor of efficiency of inefficiency.

  • Was the pace of outputs sufficient in underprivileged areas for the poorest groups to be reached first?

.

SPECIFY THE SCOPE OF THE QUESTION

.

What is this about?

Generally, an evaluation question concerns both the effects of the intervention and the intervention itself:

  • either the entire evaluated intervention, from its design down to its implementation (the scope of the question is the same as the evaluation scope)
  • or an aspect of the intervention, for example, its design, its implementation, the use of an instrument, the application of a principle of implementation (the scope of the question is more limited than the scope of the evaluation).

What is the purpose?

  • to focus the question on aspects of particular interest to the designers and managers
  • to produce recommendations concerning a particular aspect of the intervention.

Questions on the intervention design

These questions may refer to the design of the intervention as a whole or to a particular step in the design process, such as:

  • The diagnosis of problems to solve and needs to satisfy
  • The analysis of other policies that overlap with the intervention
  • The choice of a relevant level for designing the intervention (country, sector, project)
  • The stakeholder analysis
  • The choice of relevant partners (local authorities, other donors)
  • The risk analysis
  • The impact assessment
  • The definition of objectives
  • The choice of instruments
  • The choice of field operators

    or to a particular modality such as:

  • Stakeholders' involvement of the design
  • Quality assessment of the design
  • Taking into consideration of lessons from past experience
  • Etc.

Have the procedures of dialogue with the actors favoured ownership of the strategy and increased the chances of sustainable impacts?

Questions on the implementation

These questions concern the entire implementation of the intervention or a particular aspect of the implementation process such as:

  • Allocation of resources to the field operators
  • Modalities of devolution of decision-making in the field
  • Tendering procedures
  • Criteria for selecting beneficiaries
  • Monitoring system
  • Quality management
  • Operational coordination with the authorities of the partner country
  • Operational coordination with the other donors
  • Etc.

To what extent do administrative funding and project management procedures facilitate or hinder the adaptation of aid to beneficiaries' needs?

To what extent have phasing out procedures favoured the sustainability of impacts as regards food safety?

Questions on the modalities of aid

These questions concern modalities such as:

  • Instruments of regional co-operation such as the Cotonou Agreement or the MEDA programme
  • Instruments of country-wide co-operation such as partnership or co-operation agreements
  • Modalities of funding such as general or sector budget support, projects, etc.

Has the funding modality opted for made it possible to obtain better effects in terms of food safety?

To what extent have interventions in the field of transport been conducted in the form of sector-specific programmes, and what difference does this make from the point of view of effects generated?

Questions on the intervention as a whole

Finally, certain questions are drafted broadly and concern the intervention as a whole, including its design and implementation, for example: "To what extent has the design and implementation of the intervention helped to produce effect X?".

If the question concerns the entire design and implementation, we choose to focus it on a precise, immediate effect. A question concerning the entire intervention and all its effects would probably be unevaluable.

.

INFERING A QUESTION FROM THE INTERVENTION LOGIC

.

What is this about?

The intervention logic specifies the expected effects. Most evaluation questions concern one (or several) effects, which have to be specified.

What is the purpose?

To focus the question on the effects that are considered to be the most important or the least known.

To find the right balance between:

  • the evaluation of long-term or global effects which are of particular interest to policy-makers but are difficult to evaluate, and
  • short-term or direct effects that are easier to evaluate but are of interest mainly to managers.

Effect, need or problem

Where possible and relevant, the question specifies the effect concerned. This is easy for effectiveness questions such as:

  • Has effect X been obtained? Is it likely to be obtained?

It is also preferable to specify the effect concerned in more complex cases, for example:

  • Will the effect be sustainable? Will it be achieved at a reasonable cost compared to …? (effect)
  • Was the objective of achieving effect X consistent with the needs of the population? (effect / need)
  • Was the objective of achieving effect X justified by the resolution of the main development problems in the sector? (effect / problem)

As shown above, many questions can be related to the intervention logic either directly (effectiveness) or indirectly (sustainability, efficiency, relevance). The only real exception is the question on unexpected effects.

Effect

  • To what extent has the intervention contributed towards improving the most underprivileged groups' access to basic services such as education, health and other social services?

This question concerns an expected effect: better access to basic services. It is inferred directly from the intervention logic.

Need

  • Do those actions that favour the contribution of fishing to food security give priority to the nutritional needs of the local communities dependent on fishing?

This question concerns a need that must be satisfied: better food for local communities dependent on fishing. It may be indirectly related to the intervention logic in so far as satisfaction of the need corresponds to an objective.

Problem

  • Was the objective of improving the local authorities' managerial capacities relevant in the context of decentralisation reforms implemented in the country?

This question concerns a problem. Is it realistic to want to improve the managerial capacity of local authorities in the current context of the country? The question is indirectly related to the intervention logic in that it concerns an objective.

More or less extensive effects

The question specifies whether it concerns:

  • a precise effect, for example: Have the former trainees acquired the required skills?
  • a set of logically related effects, for example: What have the effects been in terms of strengthening the institutions?
  • all the effects, for example: Has the sector-specific budget support made it possible to obtain effects that are at least as good as previously, with lower transaction costs?

Questions on sets of effects are of interest to policy-makers and strategic decision-makers but are generally more difficult to answer.

Operators and field level managers are more interested in questions on precise effects. These questions are also easier to answer.

If the question concerns all effects, it is focused on a specific aspect of intervention design or implementation. A question that covered the entire intervention and all its effects would probably not be evaluable.

All the effects

  • Did the procedures of dialogue with the actors favour actual ownership of the strategy and increase the chances of sustainable impacts?

The scope of this question is very broad: all the impacts of European Aid in a country.

Group of effects

  • To what extent have the actions funded by the Commission reinforced democratic processes and civil society?

This question concerns a group of logically related effects.

A particular effect

  • Can an improvement of know-how be observed among those professionals of tourism who were trained?

This question concerns a precise effect, for a clearly defined public.

Close or distant effects

The wording of the question indicates whether one is interested in:

  • A direct and immediate effect, for example: Have the former trainees acquired the required skills?
  • A more indirect or distant link in the chain of effects, for example: Have the former trainees spread their know-how throughout their community?
  • A very indirect or distant effect, for example: How has the training contributed towards local economic development?

Questions on the most distant impacts are of interest to policy-makers and strategic decision-makers but are generally more difficult to answer.

Operators and field level managers are more interested in questions on direct results, on the uptake by the targeted group, on their needs, or on the closest impacts. These questions are also easier to answer.

Reaching beneficiaries

  • Did eligible firms request the available aid in sufficient proportion?

This question concerns an initial result.

Result

  • Did the aid have a leverage effect on the investments of farmers who received it?

This question concerns a short-term result for direct beneficiaries.

Intermediate impact

  • To what extent did the actions co-financed with local authorities improve access to basic services in disadvantaged areas?

This question concerns an intermediate effect (improvement of access to services).

A more direct effect would have been, for example: greater priority given to the territories neglected by local authorities.

A more distant effect would have been, for example: reduction of regional disparities in access to basic services.

Global impact

  • To what extent have the modalities of implementation of EC interventions contributed towards furthering democracy and the rule of law and strengthening civil society?

This question concerns a set of distant effects.

A closer effect would have been, for example, local authorities' adoption of practices that involve actors in project design.

.

SPECIFYING THE EVALUATION CRITERION

.

What is this about?

The questions are classified in different families that correspond to different "viewpoints" on what is being evaluated. Seven of these viewpoints, also called evaluation criteria, are to be considered: relevance, effectiveness, efficiency, sustainability, impact, coherence/complementarity, and Community value added.

A question is drafted in relation to an evaluation criterion unless it is intended only for knowledge and understanding.

Questions of efficiency and sustainability are asked more rarely, partly because they are difficult to answer. Yet they are often more useful.

What is the purpose?

  • To ensure that essential viewpoints are not forgotten since certain evaluation criteria tend to be neglected (e.g. efficiency or sustainability) although they can lead to very useful conclusions.
  • To guide the drafting of evaluation questions.

Focus on one evaluation criterion for each question

Questions relating to several evaluation criteria are complex, for example: Has the intervention produced effect X satisfactorily compared to the objectives (effectiveness) and were the objectives in phase with the targeted beneficiaries' needs (relevance)?

With this type of question it is probably necessary to implement two different evaluation methods: one to judge whether the expected effect was obtained and the other to identify the target group's main needs and judge whether the intervention satisfies them. The two methods use different survey and analysis techniques. Applying them both would probably entail doubling the costs or halving the quality of the answer. In this example it would be better to draft two different questions, to decide which one is more useful, and then to concentrate the resources on that question.

Exception

In some cases it is not possible to consider an evaluation criterion independently from another one, for example: Is effect X likely to continue after the end of financial support (sustainability)? In this example, we cannot answer the question without first having asked a sub-question: Has effect X been obtained (effectiveness)?

In this case it is preferable to ask only one final question (sustainable effect). In preparing the answer, the evaluation team notes that a sub-question will be: Has effect X been obtained?

The families of evaluation criteria

This page proposes a typology of seven families of criteria. The first five correspond to the traditional practice of evaluation of development aid formalised by the OECD (DAC). The following two apply to all EC policies.

Relevance

The extent to which the objectives of the development intervention are consistent with beneficiaries' requirements, country needs, global priorities and partners' and donor's policies.

Note: Retrospectively, the question of relevance often becomes a question as to whether the objectives of an intervention or its design are still appropriate given changed circumstances.

Examples

  • To what extent does the concentration of aid on basic education correspond to the needs of the partner country?
  • To what extent are interventions in trade and regional integration in line with the needs and priorities of the partner country?
  • Do actions favouring the contribution of fishing to food safety prioritise the nutritional needs of local communities dependent on fishing?
  • Was the objective of improving the managerial capacities of the local authorities relevant in the context of decentralisation reforms implemented in the country?

Effectiveness

The extent to which the development intervention's objectives were achieved, or are expected to be achieved, taking into account their relative importance.

Note: Also used as an aggregate measure of (or judgment about) the merit or worth of an activity, i.e. the extent to which an intervention has attained, or is expected to attain, its major relevant objectives efficiently in a sustainable fashion and with a positive institutional development impact. Related term: efficacy.

Examples:

  • To what extent has the aid contributed to equal access to high-quality basic education?
  • To what extent did the support contribute towards fair access to basic education of a high standard?
  • Were the outputs achieved fast enough in disadvantaged areas to be able to reach the poorest groups first?
  • To what extent did the funding procedures and efforts at dialogue and co-ordination encourage/hinder the programme to reform the road sector?

Efficiency

A measure of how economically resources/inputs (funds, expertise, time, etc.) are converted to results.

Examples:

  • Has implementation in the form of sector-specific financial aid made it possible to obtain the same effects with lower transaction costs for the EC and the partner country?
  • Did implementation in the form of sector budget support produce the same effects as previously with lower transaction costs for the EC and the partner country?
  • Did the establishment of regional co-ordinators make it possible to limit costs per person trained?
  • Have the components of the programme that consumes the most resources produced the best effects in terms of accessibility of disadvantaged areas?

Sustainability

The continuation of benefits from a development intervention after major development assistance has been completed. The probability of continued long-term benefits. The resilience to risk of the net benefit flows over time.

Examples:

  • To what extent has the aid contributed towards durably remedying the backlog in road network maintenance?
  • To what extent has the support helped sustainability to remedy the area's backwardness as regards maintenance of the road network?

Impact

Positive and negative, primary and secondary long-term effects produced by a development intervention, directly or indirectly, intended or unintended.

Example:

  • From the point of view of the groups concerned, are environmental nuisances acceptable compared to the positive effects of the intervention?

Coherence/complementarity

This criterion may have several dimensions:

1) Coherence within the Commission's development programme

Example: Can it be said that the activities and outputs logically allow the objectives to be achieved? Are there contradictions between the different levels of objective? Are there duplications between the activities?

2) Coherence/complementarity with the partner country's policies and with other donors' interventions

Example: Can it be said that there is no overlap between the intervention considered and other interventions in the partner country and/or other donors' interventions, particularly Member States?

3) Coherence/complementarity with the other Community policies

Example: Is there convergence between the objectives of the intervention and those of the other Community policies (trade, agriculture, fishing, etc.)?

Community value added

The extent to which the development intervention adds benefits to what would have resulted from Member States' interventions only in the partner country.

Examples

  • To what extent has the sharing of roles between the EC and Member States contributed to optimise the impact of the support?

Use of the term criterion: A warning!

This document concerns evaluation criteria, that is, the main ways of judging the intervention.

In order to formulate fully transparent value judgements, the approach needs to be refined into evaluation questions, and then into judgement criteria.

The word "criterion" is also used with a third meaning in the framework of quality assurance. In that case it concerns the quality assessment criteria of the evaluation.

.

SHOULD QUESTIONS BE OPEN OR CLOSED?

.

What does this mean?

Generally questions requiring a 'yes or no' answer are avoided for two reasons:

  • the answers are often multi-dimensional and have to be qualified
  • decision-makers are averse to evaluations that dictate their choices.

Open questions requiring qualified answers

Examples:

  • To what extent has effect X been obtained more satisfactorily and at a lower cost since reform Z?
  • What was the beneficiaries' opinion on the proposed aid?
  • To what extent did the coordination enable the Commission to concentrate its strategy on those areas with the highest value added?
  • To what extent was result-oriented management successfully applied?

This type of wording is more appropriate if the question is intended to acquire knowledge or understanding, or to aid decision-making. In such cases the users expect qualified answers.

Closed questions requiring "yes or no" answers

Examples:

  • Did reform Z produce effect X with a better quality/cost ratio?
  • Were the beneficiaries sufficiently satisfied with the proposed aid?
  • Did the coordination result in the EC strategy being concentrated on those areas with the highest value added?
  • Did application of result-oriented management produce significant value added to achieve the expected effects?

This type of question is more appropriate for accountability purposes, in a context where the objectives were set with precision. A closed question can also be useful if the intention is to validate a procedure or innovative instrument or to confirm a good practice.

Evaluation methodological approach > EN: Methodological bases and approach > Evaluation methods > Evaluation questions > Examples

Examples

.

This section is structured as follows:

.

EVALUATION AT COUNTRY LEVEL

.

For an evaluation at country level the questions concern, for example:

  • The strategy (and possibly the preceding strategies), its design, its implementation and the relevance of its objectives, taking into account the evolution of the context
  • The focal sectors
  • The integration of cross-cutting issues
  • Coherence, complementarity and co-ordination

The questions below are examples. Many are drawn from recent country evaluations undertaken by the Evaluation Unit.

Questions on relevance

  • How appropriate is the EC strategy as regards the objective of reducing poverty?
  • Have the development problems encountered by the country and their causes been sufficiently analysed and clarified to justify the choice of strategic priorities?
  • Was the strategy sufficiently concentrated and ranked?
  • To what extent has the strategy incorporated lessons learned from the experiences of the EC and other sponsors, concerning factors that facilitate or impede the success of aid in the partner country?

Questions on effectiveness

- Questions on sector-specific effectiveness

This site proposes a method for choosing questions specific to a particular aid sector, based on the analysis of the intervention logic. The method generates the following standard type of question:

  • To what extent has European aid in the sector contributed towards [generating effect X (specific to the sector)]?

These are questions on effectiveness (attaining a specific objective or obtaining an expected impact). Each standard question can be reformulated in many ways:

...by specifying the scope for example:

  • To what extent has European aid in the sector [and more specifically that implemented since devolution] contributed towards generating effect X?

...or by specifying the effect concerned, for example:

  • " To what extent has European aid in the sector contributed towards generating effect X [for the poorest population groups]?

...or by changing the evaluation criterion, for example:

  • Has European aid in the sector helped to generate effect X [sustainably]? (sustainability)
  • Has European aid in the sector helped to generate effect X [with a limited cost compared to …]? (efficiency or cost-effectiveness)

- Generic questions on effectiveness

  • To what extent have the Commission's interventions in the different sectors contributed to political dialogue focused on meeting the needs of the poorest population groups?
  • To what extent have the Commission's interventions in the different sectors helped to improve the management of public funds?
  • To what extent have the Commission's interventions in the different sectors helped to mitigate the negative impacts of economic reforms?
  • To what extent have the Commission's interventions in the different sectors contributed towards fair access to basic services of a high quality?
  • To what extent have the modalities of implementation of the strategy made it possible to meet the needs of the poorest regions as a priority?

Questions on efficiency

Questions on efficiency (also called cost-effectiveness) may be specific to an aid sector and are formulated as follows:

  • " Has European aid in the sector contributed towards generating effect X [with limited costs compared to …]?

It is also useful to ask generic questions on efficiency:

  • To what extent have the Commission's interventions made it possible to produce the expected effects with the lowest transaction costs possible?
  • Is there evidence of greater value added by Community aid in the case of actions that have mobilised the most resources?
  • In what proportion have EC resources been used to achieve the various strategic objectives?

Questions on sustainability

Questions on sustainability are generally specific to an aid sector and are formulated in one of the following two ways:

  • To what extent has the Commission's intervention helped to generate [effect X] in such a way that it lasts after the end of the aid?
  • To what extent has the Commission's intervention helped to generate [effect X] in such a way that it lasts even if the aid is limited [to investments only].

Questions on coherence and complementarity

  • To what extent does aid in sector X (e.g. roads) contribute directly or indirectly towards attainment of the objectives in sector Y (e.g. trade)?
  • To what extent has the design of the strategy taken into account potential interactions and conflicts with other EC policies?
  • To what extent has coordination with bilateral and multilateral interests made it possible to identify those objectives that the EC can meet better than others?
  • To what extent has the Commission contributed towards coordination between the different development partners, and to what extent has this allowed progress on the various roads to poverty reduction?

Questions on cross-cutting issues

The cross-cutting issues mentioned most frequently in recent evaluations are:

  • Gender
  • Environment
  • Good governance (decentralisation, efficient administration that is accountable and oriented towards serving the people)
  • Human rights, the rule of law, democracy, development of the role of civil society
  • Aids

Each of the following questions may concern one or more cross-cutting issues:

  • To what extent has the EC included [cross-cutting issue X] in the implementation of its interventions?
  • To what extent has the EC contributed directly or indirectly towards the development [of cross-cutting issue X]?

Questions on instruments and implementation

  • To what extent was the choice of different instruments and their combination made to generate the expected effects in the context of the partner country?
  • To what extent have the monitoring mechanisms been focused on the results and helped to improve them.
  • To what extent do the administrative procedures of allocating funds and managing actions facilitate or impede the adaptation of aid to beneficiaries needs?
  • To what extent has the implementation [of the new tool "Y' (e.g. financial support for a sector)] significantly improved the chances of obtaining the expected impacts?
  • To what extent has the design and implementation of the interventions allowed good take-up by local partners?

Example of a recent evaluation

The following example presents nine questions chosen in the case of a recent country evaluation (with several important simplifications).

  • To what extent does the Commission's health strategy meet the basic needs of the population and especially of the poorest groups?
  • Is the Commission's strategy in the private sector suited to the needs of firms?
  • To what extent have the Commission's interventions concerning transport helped to improve national and regional trade?
  • To what extent have the Commission's interventions helped to ensure that investments are maintained in the transport sector?
  • To what extent have European interventions helped to reinforce the process of regional and global cooperation and economic integration?
  • To what extent have the Commission's interventions contributed towards the decentralisation process?
  • To what extent has macroeconomic support helped to improve the management of public finances?
  • To what extent does the EC ensure that there is coordination, coherence and complementarity between its interventions and the programmes of other sponsors, especially member States?
  • To what extent is the cross-cutting objective of good governance integrated into all the interventions?

.

EVALUATION AT REGIONAL LEVEL

.

For an evaluation at regional level the questions concern, for example:

  • The strategy (and possibly the preceding strategies), its design and implementation, and the relevance of its objectives, taking into consideration the evolution of its context
  • The instruments of cooperation set up at regional level
  • Support for regional integration.

The questions below are examples. Many are drawn from recent country evaluations undertaken by the Evaluation Unit.

Questions on relevance

  • To what extent is the Commission's strategy suited to the priorities of the region's integration process?
  • To what extent has the Commission identified the needs/problems that justify support at regional rather than national level?
  • To what extent is the Commission's strategy concentrated on the institutions/partners in the best position to deal with regional integration problems?
  • To what extent does the Commission's regional strategy take into account the specific needs of countries in the region?
  • To what extent is the Commission's strategy suited to the changing needs of the region and the countries comprising it?
  • To what extent does the strategy incorporate the experience gained in comparable regions?

Questions on effectiveness

This site proposes a method for choosing questions peculiar to a particular aid sector, based on the analysis of the intervention logic. The method generates the following standard type of question:

  • To what extent does European aid in the sector contribute towards [generating effect X (peculiar to the sector)]?

These are questions on effectiveness (achieving a specific objective or obtaining an expected effect). Each standard question can be formulated in many other ways:

...by specifying the scope for example:

  • To what extent has European aid in the sector [planned in the framework of the regional strategy underway] contributed towards generating effect X?

or by specifying the effect concerned, for example:

  • To what extent has European aid in the sector contributed towards generating effect X [in the least developed countries]?

...or by changing the evaluation criterion, for example:

  • Has European aid in the sector contributed towards generating effect X [sustainably]? (sustainability)
  • Has European aid in the sector contributed towards generating effect X [with limited costs compared to …]? (efficiency or cost-effectiveness)

It is also useful to ask questions on generic effectiveness, that is, cutting across an entire intervention logic, for example:

  • To what extent have the Commission's interventions contributed towards reinforcing the process of regional integration?
  • To what extent have the Commission's interventions contributed towards improving the capacity of regional aid institutions?

Questions on efficiency

Questions on efficiency (also called cost-effectiveness) may be specific to an aid sector and may be in the following form:

  • Has European aid in the sector contributed towards generating [effect X (specific to the sector) with limited costs compared to …]?

It is also useful to ask generic questions on efficiency, for example:

  • To what extent have Commission interventions helped to produce the expected effects with the lowest possible transaction costs?
  • Is there evidence of greater value added by Community aid in the case of actions that have mobilised the most resources?
  • In what proportions have EC resources been used to achieve the various objectives of its regional strategy?

Questions on sustainability

Questions on sustainability are generally specific to an aid sector and are formulated as follows:

  • To what extent has the Commission intervention helped to generate [effect X] in such a way that it will continue after the end of the aid?

Questions on coherence/complementarity

  • To what extent does aid in sector X (e.g. roads) contribute directly or indirectly towards attaining the objectives in sector Y (e.g. trade)?
  • To what extent does the strategy design take into account potential interactions and conflicts with the other EC policies?
  • To what extent is the Commission's strategy at regional level complementary to the strategies defined in each of the countries of the region?

Questions on cross-cutting issues

The cross-cutting issues most frequently mentioned in recent evaluations are:

  • Gender
  • Environment
  • Good governance (decentralisation, efficient administration that is accountable and oriented towards serving the people)
  • Human rights, the rule of law, developing the role of civil society
  • HIV/AIDS

Each of the following questions may concern one or more cross-cutting issues:

  • To what extent has the EC integrated [cross-cutting issue X] into the design and implementation of its interventions?
  • To what extent has the EC contributed directly or indirectly towards the development [of cross-cutting issue X]?

Questions on instruments and implementation

  • To what extent has the choice of different instruments and their combination been appropriate for generating the expected effects?
  • To what extent have the monitoring mechanisms been focused on the results and helped to improve them?
  • To what extent have administrative procedures of allocating funds and managing actions facilitated or impeded progress towards the strategic objectives?
  • To what extent have the design and implementation of interventions allowed good take-up by regional aid institutions?

Example of a recent evaluation

The following example presents the nine questions chosen in the case of a recent evaluation in the MERCOSUR region (with several important simplifications):

  • How consistent is the Commission's strategy or support with the priorities of MERCOSUR's integration process?
  • To what extent does the EC strategy take into account the specific needs of the member States of MERCOSUR and their evolution during the evaluated period?
  • To what extent has the Commission's support helped to enhance MERCOSUR's ability to negotiate an inter-regional partnership agreement effectively?
  • How have the economic and commercial projects helped to reinforce MERCOSUR's integration process?
  • To what extent have the Commission's cooperation procedures (including mechanisms for implementing and granting funding) affected potential results?
  • To what extent has the EC coordinated its strategy and ensured its coherence with its other policies that have an international dimension, especially the environment?
  • To what extent has the European strategy contributed towards the continuation of MERCOSUR?

.

SECTOR OR THEMATIC GLOBAL EVALUATION

.

For an sector or thematic global evaluation the questions concern, for example:

  • The effects peculiar to the sector, and their sustainability
  • The integration of cross-cutting issues
  • The effectiveness or efficiency of instruments peculiar to the sector.

The following questions are examples, many of which are drawn from recent country evaluations undertaken by the Evaluation Unit.

Questions on relevance

  • To what extent are the Commission's interventions adjusted to the specific needs of countries (or regions) [category X]?
  • To what extent does the design of the Commission's interventions make it possible to obtain an appropriate combination of activities and instruments for meeting the objective [X] ?
  • To what extent does the Commission consider the evaluated sector/theme as a priority, and to what extent are these choices justified?
  • To what extent has the Commission's strategy concerning the sector/theme incorporated the lessons learned from other sponsors' experiences?

Questions on effectiveness

This site proposes a method for choosing questions from the analysis of the intervention logic. The method is generally based on the analysis of one or more strategic documents that define a coherent set of objectives. Such documents may not exist in the case of thematic or sector evaluation on a global scale, but that does not preclude the more informal reconstruction of the intervention logic. It is then possible to formulate questions with the following standard form:

  • To what extent did European aid help to [generate effect X (specific to the sector/theme)]?

These are questions on effectiveness (attaining a specific objective or one of the expected effects). Each standard question can be reformulated in many ways:

...by specifying the scope for example:

  • To what extent has European aid [specifically devoted to the sector/theme under evaluation] contributed towards generating [effect X]?
  • To what extent has European aid [implemented according to a particular procedure / using a particular tool] contributed towards generating [effect X]?

...or by specifying the effect concerned, for example:

  • To what extent has European aid contributed towards generating [effect X for the poorest categories of the population]?

or by changing the evaluation criterion, for example:

  • To what extent has European aid contributed towards generating effect X [sustainably]? (sustainability)
  • To what extent has European aid contributed towards generating effect X [with limited costs, compared to …]? (efficiency or cost-effectiveness)

It is also interesting to ask generic effectiveness questions, that is, questions cutting across the entire intervention logic, for example:

  • In which cases has European aid contributed most to developing institutional capacities and a governance suited to needs in the sector?
  • To what extent have the Commission's interventions made countries able to identify their needs and to plan appropriate actions?
  • To what extent have the Commission's interventions allowed it to respond as a priority to the people's needs and especially to those of the most underprivileged groups?

Questions on efficiency

Questions on efficiency (or cost-effectiveness) can be specific to the sector/theme. They are formulated as follows:

  • Has European aid contributed towards generating [effect X] with a cost comparable to that of good practice observed in similar countries?
  • In which countries (and circumstances) has European aid contributed towards generating [effect X] with a particularly favourable cost-effectiveness ratio? What are the reasons for this success?
  • In what proportions have EC resources benefited the various groups/regions concerned?

Questions on sustainability

Questions on sustainability are generally formulated in one of the following two ways:

  • To what extent has the Commission's aid contributed towards generating [effect X] in such a way that it will last after the end of the aid?
  • To what extent has the Commission's intervention contributed towards generating [effect X] in such a way that it will last even if the aid is limited [to investment only]?

Questions on coherence/complementarity

  • To what extent do interventions in the sector contribute directly or indirectly to the objectives of another sector?
  • To what extent do interventions in the sector have mutually reinforcing effects (or contradictory effects) with other EC policies?
  • To what extent are the various EC policies mutually reinforcing (or contradictory) as regards generating [effect X]?
  • To what extent does European aid in the sector take into account potential interactions or conflicts with other European policies concerning the sector?
  • To what extent has coordination with bilateral and multilateral interests allowed more productive dialogue as regards generating [effect X]?
  • To what extent has the Commission contributed towards coordination between the different development partners, and to what extent has this allowed progress on the various paths towards achieving [effect X]?

Questions on cross-cutting issues

The cross-cutting issues mentioned most frequently in recent evaluations are:

  • Gender
  • Environment
  • Good governance (decentralisation, efficient administration that is accountable and oriented towards serving the people)
  • Human rights, the rule of law, democracy, development of the role of civil society
  • HIV/Aids

Each of the following questions can concern one or more cross-cutting issues:

  • To what extent has the EC integrated [cross-cutting issue X] into the design and implementation of its interventions?
  • To what extent has the EC contributed directly or indirectly towards the development [of cross-cutting issue X]?
  • In which countries/circumstances have EC interventions contributed most successfully towards the development [of cross-cutting issue X]?

Questions on instruments and implementation

  • To what extent have interventions in the sector implemented [tool X] and with what effect?
  • To what extent do administrative procedures of allocating funds and managing actions facilitate or impede the achievement of [effect X]?

Example of a recent evaluation

The following example presents the nine questions chosen in the global evaluation of the 'Commerce' sector:

  • To what extent are the Commission's programmes affected by the specific needs of the country/region for the negotiation of future agreements?
  • To what extent are the Commission's programmes affected by the specific needs of the country/region for the benefit of international trade?
  • To what extent do diagnostic and analytical practices in the identification and programming stages of the project cycle make a combination of activities, contributing to a balanced achievement of the objectives, more likely?
  • To what extent do the modalities of implementation (including available instruments and resources) determine the achievement of objectives?
  • To what extent have Commission interventions enabled the countries to strengthen their institutional framework for regional commercial agreements?
  • To what extent have the Commission's programmes made the countries able to identify and articulate their needs and to improve their position in international trade negotiations?
  • In those countries and regions in which the Commission has financed the largest number of interventions for trade, to what extent have agents in the productive sector effectively used the outputs of those interventions?
  • In those countries and regions in which the Commission has financed the largest number of programmes for trade, to what extent have those interventions affected the volume, value and direction of trade?
  • To what extent have EC initiatives helped to counter-balance possible negative effects of trade on human rights, basic social rights and environmental issues?
Evaluation methodological approach > EN: Methodological bases and approach > Evaluation methods > Judgement references

Judgement references

.

This section is structured as follows:

.

JUDGEMENT CRITERION

.

What does this mean?

A judgement criterion specifies an aspect of the evaluated intervention that will allow its merits or success to be assessed. Whilst "judgement criterion" is the appropriate word, an acceptable alternative is "reasoned assessment criterion". The criterion is used to answer an evaluation question. One or more judgement criteria are derived from each question.

What is the purpose?

  • To avoid subjectivity and to formulate judgements on accepted terms.
  • To improve the transparency of the evaluation by making the judgment explicit.
  • To structure the answers to the questions asked, since the judgement criteria will determine the indicators and, more generally, the nature of the data collected and the type of analysis.

How can a judgement criterion be clarified on the basis of a question?

All the evaluation questions relate to one or more judgement criteria, unless they are designed only to further knowledge or understanding about the intervention or its effects. The following is an example of a question:

Question
To what extent has EC aid improved the capacity of the primary educational system to enrol pupils from underprivileged groups without discrimination?

Like most evaluative questions, it has two parts:

  • What is being judged: "EC aid".
  • The way of judging: Has it "… improved the capacity of the primary educational system to enrol pupils from underprivileged groups without discrimination".
  • " This question belongs to the effectiveness family

The judgement criteria develop and specify the second part of the question, for example:

Judgement criteria
Capacity of the primary school system to enrol pupils from ethnic minority X satisfactorily.

Capacity of the primary school system to enrol pupils from disadvantaged urban areas satisfactorily.

The judgement criteria derive from the question, for instance in the case of the first criterion:

  • It concerns the way of judging and not what is judged. This is why the beginning of the question concerning "EC aid" has been removed.
  • It specifies the type of success to be evaluated, that is, an improvement in the "capacity of the primary school system to enrol pupils from underprivileged groups without discrimination", and specifically "pupils from ethnic minority X".
  • It emphasises the judgement and not the causality analysis. That is why the terms "To what extent … has it improved" have been removed.

To be used in practice, each judgement criterion has to be accompanied by a target level and one or more indicator(s).

Recommendations

  • Always define the judgement criterion before selecting an existing indicator or designing a new indicator. This is essential in order to clarify the concepts. By focusing too soon on indicators, one is likely to get trapped into existing information, even if it is inadequate for answering the question asked.
  • Have the definition of the judgement criteria discussed by the reference group so that the diversity of points of view relevant to the intervention can be taken into account.
  • There may be disagreement on the judgement criteria, for instance the same effect may have a dimension that is judged positively by certain members of the reference group and another dimension judged negatively by others. In this case there are two options: (1) choose only one judgement criterion but be careful to avoid biasing the evaluation; or (2) choose several criteria, although this will increase and complicate the data collection and analysis work.
  • To optimise the collection and analysis of data, it is best to define a limited number of judgement criteria for each question. This recommendation also takes into account users' capacity to absorb information.
  • Where relevant, explain any gaps between the criteria used to formulate the judgement at the end of the evaluation process and those identified in the first phase (desk) of the evaluation.

Be careful not to confuse concepts

On this site, the word criterion is used for three different concepts:

  • Judgement criteria presented on this page
  • The evaluation criteria: relevance, effectiveness, efficiency, sustainability, impact, Community value added, and coherence/complementarity.
  • The quality assessment criteria of evaluation reports.

According to the EC, the value added of an evaluation is the formulation of value judgements on the basis of evidence and explicit judgement criteria. When dealing with organisations which are not familiar with evaluation, it may be wise not to use the word "judgement", which may induce resistance. An acceptable alternative is "assessment", or preferably "reasoned assessment".

.

TARGET

.

What does this mean?

The concept of a 'target' is widely used in the context of public management for setting a verifiable objective or a level of performance to be achieved. In an evaluation context it is used in a much wider sense since the evaluated intervention may have to be judged against targets that were not set in advance but that are specifically identified, such as a benchmark, a success threshold or a comparable good practice.

What is the purpose?

  • To avoid subjectivity and formulate a judgement on accepted and recognised terms.

How can they be determined?

- By reference to an objective defined in a verifiable way

The target may appear in one of the intervention objectives, that is, as long as they have been established in a verifiable way. In this particular case, the same indicator helps to define the objective, to make the judgment criterion operational and to determine the target.

  • Example: the number of qualified and experienced teachers per 1,000 children of primary-school age is at least 20.

- In relation to comparable good practices outside the intervention

In this case, the target is established at the outset of the evaluation. It is not related to an objective or a performance framework existing prior to the evaluation.

  • Example: the access to primary education with qualified and experienced teachers is at least as satisfactory as in the case of X (recognised good practice at regional level).

The procedure is as follows:

  • Identify a comparable practice recognised for its quality (similar EC intervention in another country; intervention by another donor, intervention in another sector though using the same instruments).
  • Obtain information on the practice for comparison (this is easier if it has already been evaluated).
  • Ensure that the contextual conditions are close enough that they allow for comparison.
  • Proceed to carry out the comparison (essentially qualitative).
  • Discuss and validate the comparison with the reference group.

- Compared to best practices identified within the intervention

The target can be found within the evaluated intervention itself during the synthesis phase, provided that specific practices can be considered as good as regards the judgement criteria under consideration.

In this case, the good practices will serve as benchmarks to judge the others. Of course, it is advisable to check that the contextual conditions are close enough so as to allow for comparison.

  • Example: In areas where ethnic minority X concentrates, the number of qualified and experienced teachers per 1,000 children of primary-school age is close to the best performing areas in the country.

When should they be determined?

- Earlier or later in the evaluation process

If the target is derived from a verifiable objective or a performance framework, then it can be determined at the very first stage of the evaluation process.

If the target is derived from an outside benchmark, then it should be identified during the early stages of the evaluation. However, the process may involve the gathering of secondary data with a view to specifying the benchmark, as well as a careful examination of comparability. This means that the target will not be completely defined in the first phase of the evaluation.

If the target is to be derived from the best practices discovered within the intervention by the evaluation team, it will be determined in the synthesis phase.

- After choosing the judgement criterion

Determining the target takes place in a three-step process:

  • Choice and finalisation of the evaluation question.
  • Choice of the judgment criterion (or criteria).
  • The targets, the indicators and the sources of information are determined together in the third step.

Evaluation targets and others

When the evaluation question pertains to an intended result or impact, the target level is usually derived from a verifiable objective or borrowed from a performance assessment framework.

Performance monitoring may however be of little or no help in the instance of evaluation questions relating to cross-cutting issues, sustainability factors, unintended effects, evolving needs and problems, coherence, etc.

.

INDICATOR

.

What does this mean?

The evaluation team may use any kind of reliable data to assess whether an intervention has been successful or not in relation to a judgement criterion and a target.

Data may be collected in a structured way by using indicators. Indicators specify precisely which data are to be collected. An indicator may be quantitative or qualitative. In the latter case the scoring technique may be used.

Unstructured data are also collected during the evaluation, either incidentally, or because tools such as case studies are used. This kind of evidence may be sound enough to be a basis for conclusions, but it is not an indicator.

What is the purpose?

  • To collect and process data in a form that can be used directly when answering questions.
  • To avoid collecting an excessive amount of irrelevant data and to focus the process on the questions asked.

Evaluation indicators

The main evaluation indicators are those related to judgement criteria, that specify the data needed to make a judgement based on those criteria.

An indicator can be constructed specifically for an evaluation (ad hoc indicator) and measured during a survey, for example. It may also be drawn from monitoring databases, a performance assessment framework, or statistical sources.

A qualitative indicator (or descriptor) takes the form of a statement that has to be verified during the data collection (e.g. parents' opinion is that their children have the possibility of attending a primary school class with a qualified and experienced teacher).

A quantitative indicator is based on a counting process (e.g. number of qualified and experienced teachers). The basic indicator directly results from the counting process. It may be used for computing more elaborate indicators (ratios, rates) such as cost per pupil or number of qualified and experienced teachers per 1,000 children of primary-school age.

Indicators may belong to different categories: inputs, outputs, results or impacts. 

Evaluation indicators and others

When an evaluation question pertains to an intended result or impact, it is worth checking whether this result or impact has been subject to performance monitoring. In such cases, the evaluation team uses the corresponding indicators and data, which is a considerable help, especially if baseline data have been recorded.

Performance monitoring may, however, be of little or no help in the instance of evaluation questions relating to cross-cutting issues, sustainability factors, unintended effects, evolving needs or problems, coherence, etc.

Quality of an indicator

An indicator measures or qualifies with precision the judgement criterion or variable under observation (construct validity). If necessary, several less precise indicators (proxies) may be used to enhance validity.

It provides straightforward information that is easy to communicate and is understood in the same way by the information supplier and the user.

It is precise, that is, associated with a definition containing no ambiguity.

It is sensitive, that is, it generates data which vary significantly when a change appears in what is being observed.

Performance indicators and targets are often expected to be SMART, i.e. Specific, Measurable, Attainable, Realistic and Timely. The quality of an evaluation indicator is assessed differently. 

Indicators and effects: a warning!

The indicator used to evaluate an effect is not in itself a measurement or evidence of that effect. The indicator only informs on changes, which may either result from the intervention (effect) or from other causes.

The evaluation team always has to analyse or interpret the indicator in order to assess the effect.

Categories of indicators

- Indicators and the intervention cycle

Indicators are used throughout the intervention cycle. They are first used to analyse the context; then, for the choice and validation of the intervention strategy, afterwards for monitoring outputs and results and, finally, for the evaluation.

Indicators and intervention design

Context indicators may be used to support the identification of the needs, problems and challenges which justify the intervention.

As far as possible, objectives and targets are defined in a measurable way by using indicators.

Indicators, monitoring and performance assessment

Monitoring systems and performance assessment frameworks also use indicators which derive from the diagram of expected effects (also called results chain).

Monitoring indicators primarily relate to inputs and outputs. Performance indicators primarily focus on intended results and impacts. The EC's Result Oriented Monitoring (ROM) does not rely that much on indicators. It delivers systematic assessments of external aid projects in the form of ratings with a view to intended results and impacts.

Indicators and evaluation

Evaluation indicators are used to help answering specific evaluation questions. Depending on the question, they may relate to the needs, problems and challenges which have justified the intervention, or to the achievement of intended outputs, results and impacts, or to anything else.

- Global and specific indicators

Global or contextual indicators apply to an entire territory, population or group, without any distinction between those who have been reached by the intervention and those who have not. They are mainly taken from statistical data. This site offers help to look for contextual indicators.

Specific indicators concern only a group or territory that has actually been reached. With specific indicators, changes among those affected by the intervention can be monitored. Most of these indicators are produced through surveys and management databases.

- Indicators and intervention logic

Input indicators

Input indicators provide information on financial, human, material, organisational or regulatory resources mobilised during the implementation of the intervention. Most input indicators are quantified on a regular basis by the management and monitoring systems (providing that they are operational).

Output indicators

Output indicators provide information on the operators' activity, especially on the products and services that they deliver and for which they are responsible. To put it simply, one could say that outputs correspond to what is bought with public money.

Result indicators

Result indicators provide information on the immediate effects of the intervention for its direct addressees. An effect is immediate if the operator notices it easily while he/she is in contact with an addressee. Because they are easily recognised by the operators, direct result indicators can be quantified exhaustively by the monitoring system.

Impact indicators

Impact indicators provide information on the long-term direct and indirect consequences of the intervention.

A first category concerns the consequences that appear or last in the medium or long term for the direct beneficiaries.

A second category of impacts concerns people or actors that are not direct beneficiaries.

Impact indicators cannot be produced in general from management information. They require statistical data or surveys specially conducted during the evaluation process.

Indicators derived from scoring

What does this mean?

Scoring (or rating) produces figures that synthesise a set of qualitative data and or opinions. Scoring is guided by a scoring grid (or scorecard) with varying degrees of detail.

From an evaluation point of view, both words scoring and rating can be used.

What is the point?

Scoring allows the production of structured and comparable data on judgement criteria that do not lend themselves to a measurement using quantitative indicators.

How to construct a scoring grid

  • Examine several possible dimensions for the criterion that has to be assessed (sub-criteria).
  • For each dimension or sub-criterion, write a short sentence defining the success of the intervention (descriptor of full success).
  • For each dimension or sub-criterion, write another sentence (descriptor) defining the failure of the intervention.
  • Write one or more sentences (descriptors) that represent intermediate levels of success.
  • Associate a score with each descriptor (e.g. from 0 to 10, from 0 to 3, from -3 to +3).
  • Weight the sub-criteria if necessary.
  • Test the grid on a few pilot examples.
  • Discuss the test with the reference group if relevant

How to use the scoring grid

Scoring grids usually apply to projects or components of the intervention and allow for comparing these.

The evaluation team puts together all the data it has on the project or intervention to be assessed. It then chooses the level (or descriptor) in the scoring grid that corresponds best (or the least badly) to this information. The score results from this choice. 

Recommendations

The more detailed the scoring grid the less subjective the score will be and the more comparable the scores allocated by two different evaluators will be.

.

FROM QUESTIONS TO INDICATORS

.

Example of a question
  • To what extent has EC support improved the capacity of the educational system to enrol pupils from disadvantaged groups without discrimination?

From questions to judgement criteria

The judgement criterion (also called reasoned assessment criterion) specifies an aspect of the evaluated intervention that will allow its merits or worth to be assessed in order to answer the evaluation question, . For instance:

Judgement criterion derived from the question
  • Capacity of the primary school system to enrol pupils from ethnic minority X with satisfactory quality.

The judgement criterion gives a clear indication of what is positive or negative, for example: "enhancing the expected effects" is preferable to "taking potential effects into account".

A more precise judgement criterion than the question

The question is drafted in a non-technical way with wording that is easily understood by all, even if it lacks precision.

The judgement criterion focuses the question on the most essential points for the judgement.

Yet the judgement criterion does not need to be totally precise. In the first example the term "satisfactory quality" can be specified elsewhere (at the indicator stage).

Not too many criteria

It is often possible to define many judgement criteria for the same question, but this would complicate the data collection and make the answer less clear.

In the example below, the question is treated with three judgement criteria (multicriteria approach):

  • "capacity of the primary school system to enrol pupils from ethnic minority X with satisfactory quality"
  • "capacity of the primary school system to enrol pupils from the poorest urban areas with satisfactory quality"
  • "capacity of the primary school system to enrol girls ".

A judgement criterion corresponding to the question

The judgement criterion should not betray the question. In the following example, two judgement criteria are considered for answering the same question:

  • "capacity of the primary school system to enrol pupils from ethnic minority X with satisfactory quality"
  • "primary school leavers from ethnic minority X pass their final year exam "

The first judgement criterion is faithful to the question, while the second is less so in so far as it concerns the success in primary education, whereas the question concerns only the access to it. The question may have been badly worded, in which case it may be amended if there is still time.

Also specify the scope of the question

Most questions have a scope (what is judged) and a judgement criterion (the way of judging). In addition to the judgement criterion, it is therefore often desirable to specify the scope of the question, for example: "European aid granted over the past X years", "design of programme X", "the principle of decentralisation adopted to implement action X".

Also specify the type of cause-and-effect analysis

Some questions imply a cause-and-effect analysis prior to the judgement. It may therefore also be useful to specify the type of analysis required by means of terms such as "has European aid led to", "has it contributed to", "is it likely to".

From judgement criteria to indicators

An indicator describes in detail the information required to answer the question according to the judgement criterion chosen, for example:

Indicator derived from the judgement criterion
  • Number of qualified and experienced teachers per 1000 children of primary-school age in areas where ethnic minority X concentrates

Not too many indicators

It is possible to define many indicators for the same judgement criterion. Relying upon several indicators allows for cross-checking and strengthens the evidence base on which the question is answered. However, an excessive number of indicators involves a heavy data collection workload without necessarily improving the soundness of the answer to the question.

In the examples below three indicators are applied to a judgement criterion ("capacity of the primary school system to enrol pupils from ethnic minority X with satisfactory quality"):

  • "Number of qualified and experienced teachers per 1000 children of primary-school age in areas where ethnic minority X concentrates"
  • "Number of pupils per teacher in areas where ethnic minority X concentrates"
  • "Level of quality of the premises (scale 1 to 3) assigned to primary education in areas where ethnic minority X concentrates ".

Indicator corresponding to the judgement criterion

The indicator should not betray the judgement criterion. Two indicators are considered below:

  • "Number of qualified and experienced teachers per 1000 children of primary-school age in areas where ethnic minority X concentrates."
  • "Primary education enrolment rate in areas where ethnic minority X concentrates"

The first indicator corresponds faithfully in so far as it describes an essential aspect of the judgement criterion. The second indicator is less faithful because it fails to reflect the concept of "satisfactory quality". Its construct validity is not good.

Unambiguous indicators

An indicator must be defined without any ambiguity and understood in the same way by all the members of the evaluation team. For instance, in the above examples it is necessary to specify what a "qualified and experienced teacher" is. This can be done with reference to an existing definition, or else a definition can be formulated as precisely as possible until there is no more ambiguity whatsoever.

Indicators independent from the observation field

The same indicator should be able to serve to collect data in several contexts, for example:

  • "Number of qualified and experienced teachers per 1000 children of primary-school age" - in areas where ethnic minority X concentrates.
  • "Number of qualified and experienced teachers per 1000 children of primary-school age" - in areas where ethnic minority X is absent.

In this case the same indicator is applied in both types of area and serves as a comparison, on the basis of which a judgement is formulated.

Quantitative and qualitative indicators

The following two examples present an alternative between a quantitative indicator and a qualitative indicator for treating the same judgement criterion:

  • "Number of qualified and experienced teachers per 1000 children of primary-school age" (quantitative)
  • "Surveyed parents confirm that their children have the possibility of attending a primary-school class and benefit from a qualified and experienced teacher" (qualitative);

An indicator is preferably associated with a target

The target indicates which comparison should be made in order to answer the question, for example: "In the areas where ethnic minority X concentrates, the indicator is at least as good as in the entire country in average".

The target and the indicator are often specified interactively in successive steps. It is important not to digress from the judgement criterion during this process.

When the evaluation question pertains to an intended result or impact, the target is usually derived from a verifiable objective or borrowed from a performance assessment framework.

Feasible indicators

The indicator makes it possible to focus and structure data collection but serves no purpose as long as data does not exist. To ensure the feasibility of an indicator, it is necessary to indicate the source of the information to use, for example:

  • management data from the national education system,
  • periodical national surveys on the education system,
  • questionnaire survey in several specially selected areas and in the framework of the evaluation.

If no source is available or feasible, the indicator should be changed. If no feasible indicator can be found, excluding the question should be envisaged.

Example of a country evaluation

Question

Question
To what extent does the EC ensure coherence between its support and its other policies?

  • The question relates to a family of evaluation criteria: coherence/complementarity

Judgement criterion

Judgement criterion
The expected effects of the intervention, and the effects of other EC policies affecting the partner country are likely to reinforce each others.

The judgement criterion is derived from the question in the following way:

  • It shows the type of success that the question is supposed to evaluate, that is, "coherence between the aid and other EC policies".
  • It specifies several concepts, primarily that of coherence but also the term "other policies".
  • It concerns the way of judging and not what is judged. That is why the beginning of the question concerning "To what extent does the EC ensure" has been removed.

Indicator

Indicator
Positive/negative synergy between the expected effects of the intervention and the expected effects of other EC policies, as regards affected groups in the partner country.

The indicator is derived from the judgement criterion in the following way:

  • It corresponds faithfully to the judgement criterion.
  • It describes in detail the data to be collected in order to apply the judgement criterion chosen. However, terms such as "positive/negative synergy" still have to be defined further. The definition must be drafted as precisely as possible, until there is no more ambiguity whatsoever.
  • It is qualitative, although a quantitative indicator could have been defined, for example: "proportion of positive synergies among identified synergies".
  • It makes it possible to define a target, for example by comparing the respective importance of positive and negative synergies.
  • Its feasibility still has to be verified by ensuring that one or more sources of information and evaluation tools will be available, for example: opinion of a panel of independent experts, questionnaire administered at the end of a focus group composed of administrators.

Example of a sector evaluation

Question

Question
To what extent has EC support enhanced the capacity of the educational system to enrol pupils from disadvantaged groups without discrimination?

The question refers to a family of evaluation criteria: effectiveness.

Judgement criterion

Judgement criterion
Capacity of the primary school system to enrol pupils from ethnic minority X with satisfactory quality.

The judgement criterion is derived from the question in the following way:

  • It shows the type of success that the question is supposed to evaluate, that is, the best "capacity of the education system to enrol pupils from disadvantaged groups without discrimination".
  • It clarifies several concepts such as "educational system", "disadvantaged groups" and "discrimination". The term "satisfactory quality" has yet to be specified.
  • It concerns the way of judging and not what has been judged. That is why the beginning of the question concerning "EC support" has been removed.
  • It focuses on the judgement and not on the causal analysis. That is why the term "has … enhanced" has been removed.

Indicator

Indicator
Number of qualified and experienced teachers per 1000 children of primary-school age in areas where the ethnic minority X concentrates.

The indicator derives from the judgement criterion in the following way:

  • It describes in detail the information required to judge according to the judgement criterion chosen. But this is not enough. In particular, it is necessary to specify what a "qualified and experienced teacher" is. This can be done by referring to an existing definition or else by formulating a definition as precisely as possible until there is no more ambiguity whatsoever.
  • The indicator corresponds faithfully to the judgement criterion ("capacity of the primary school system to enrol pupils with satisfactory quality"). It does not encompass all the dimensions of the judgement criterion but highlights what is considered as essential.
  • It is quantitative, but a qualitative indicator could also have been defined, for example: "surveyed parents confirm that their children have the possibility of attending a primary school class and benefit from a qualified and experienced teacher".
  • It will make it possible to define a target, for example by comparing "areas where ethnic minority X concentrates" with "the entire country".
  • Its feasibility still has to be verified by ensuring that one or more sources of information are available, for example: management data of the national education system, periodical national surveys on the educational system, questionnaire survey run as part of the evaluation.
Evaluation methodological approach > EN: Methodological bases and approach > Evaluation methods > Methodological design

Methodological design

.

This section is structured as follows:

.

WHAT DOES IT MEAN? 

.

What is the purpose?

To set up the method that will allow the external evaluation team to answer the questions and to come to an overall assessment. In addition to selected questions, judgement criteria, indicators and targets, the evaluation method includes:

  • a strategy for collecting and analysing data
  • selected investigation areas
  • a series of specifically designed tools
  • a work plan.

When does it take place?

The evaluation team starts designing a provisional method as early as from the drafting of its proposal in order to draw up cost estimates. A key assumption at this stage is the extent to which the evaluation will rely on secondary data or will involve specific data collection work in the field.

The main frame of the method is then established during the inception stage, in line with the evaluation questions, judgement criteria, indicators, data collection tools and analysis strategy.

The method is refined and finalised before the field phase and fully described in the first phase report (desk).
The final report includes a short and sharp presentation of the evaluation method, together with its limitations, if there are any. The method is fully described in annex, including initial design, problems encountered, solutions found, method actually implemented, and limitations.
The evaluation method is designed through an iterative process at three levels:

  • A question-by-question approach, allowing the evaluation team to prepare the design tables with an aim to adequately answer each question.
  • An overall approach which cuts across the questions, and which allows the evaluation team to optimise the evaluation method as a whole, whilst matching time and resource constraints.
  • A series of specifically developed tools.

.

DESIGN TABLE PER QUESTION

.

What does this mean?

The design table explains how an evaluation question will be answered, including the chain of reasoning which connects data, findings and conclusions.
An example of a design table is provided on this site.

When is it constructed?

A design table is developed for each question and progressively refined in successive versions:

  • Preliminary version appended to the inception report.
  • Draft version(s) prepared during the first phase of the evaluation as the methodological design is progressively optimised.
  • Final version attached to the first phase report (desk).

Opening the table

The first lines of the table summarise the steps which have already been taken (see from question to indicator), i.e.

  • Text of the question
  • Comment
  • Scope of the question
  • Judgement criterion (or criteria)
  • Indicators
  • Targets

Sub-questions

What does this mean?

Together with the judgement criteria, indicators and targets, the sub-questions are meant to explain how the evaluation team will answer the question, for instance:

  • How have indicators changed over the evaluated period?
  • To what extent can the change be qualified as a success?
  • How far have EC activities contributed to explaining the observed change? Through which mechanisms? Is there evidence that such mechanisms have been working as assumed?
  • How far does evidence support alternative explanations? Is there evidence that the observed change is owing to other development partners, other EC policies, the Government or other external factors?

Who does what, and when?

A set of evaluation questions is drawn up at the inception stage, together with sub-questions.
The responsibilities are as follows:

  • The reference group validates the questions.
  • The external evaluation team breaks down each question into sub-questions in a design table.

What is the purpose?

The sub-questions describe the chain of reasoning through which the evaluation team plans to answer the question. The intended reasoning is indicative but it is worth clarifying it in advance because:

  • The reference group members get an opportunity to provide specific advice and inputs
  • All evaluation team members understand why they are collecting and analysing data, and therefore work more effectively
  • The team members who are not familiar with evaluation receive useful guidance on which data are to be collected and analysed.

It may furthermore be worth suggesting provisional sub-questions together with the draft set of key questions. Members of the reference group will then realise that many of their expectations will be satisfied through answering the sub-questions. It will help them to accept the principle of a limited list of well focused evaluation questions.

Sub-questions pertaining to indicators

These sub-questions may pertain to:
(1) the current level / status of the indicators, possibly with a break-down per country, area, social group, etc., for instance:

  • What is the current value of quantitative indicator X at country level, and for targeted areas/groups?

(2) changes in the indicators, for instance:

  • Do stakeholders perceive a positive change in qualitative indicator Y over the evaluated time period?

As seen in the examples above, the sub-questions may be quantitative or qualitative.

Question:

  • To what extent has EC support improved the capacity of the educational system to enrol pupils from disadvantaged groups without discrimination?

Sub-question:

  • What is the change in the number of experienced and qualified teachers per 1000 primary school age children respectively at country level, in poor urban areas and in areas where ethnic minority X concentrates?

In this example, the sub-question is simply meant to show how the indicator (number of experienced and qualified teachers) will be applied.

Sub-questions pertaining to analysis

These sub-questions are written with a view to:
(3) Confirming assumptions about the success of the intervention and substantiating a positive answer to the question, for instance:

  • Has the design of EC support included a commitment to monitor performance indicators related to effect X?
  • Was such monitoring actually undertaken?
  • Was the monitoring data subject to periodic discussion among development partners?
  • Have partners taken action as a follow-up to such discussions?
  • Were such actions designed with a view to achieving effect X?
  • Etc.

(4) Challenging assumptions about the success of the intervention and substantiating a negative answer to the question, for instance:

  • Have other development partners pushed for monitoring performance indicators related to effect X?
  • Have non-state actors contributed to putting the issue of achieving effect X onto the political agenda?
  • How far did other partners contribute towards shaping the actions taken in favour of disadvantaged groups?

Question:

  • To what extent has EC support improved the capacity of local authorities to design adequate rural development strategies?

Sub-question:

  • To what extent has the EC been able to put decentralisation at the top of the agenda in its policy dialogue with the Government?

In this example the sub-question relates to a specific short-term result (promotion of decentralisation through policy dialogue) which is a driver to the wider impact highlighted in the question (capacity building).

Sub-questions pertaining to judgement

These sub-questions are meant to assist in the formulation of conclusions involving explicit value judgements. They are written with a view to:
(5) applying and possibly refining the judgement criteria in the specific context of the intervention, for instance:

  • Do stakeholders spontaneously focus on the same judgement criteria as those selected for the evaluation? If not, why not?

(6) applying or developing the targets in the specific context of the intervention, for instance:

  • Which are the areas / groups in the country with the best performance as regards the selected judgement criterion? Among them, which ones can legitimately be compared with targeted areas / groups?

Question:

  • To what extent has EC support been efficient in strengthening the road network?

Sub-question:

  • To what extent has the EC strengthened the road network? In this example, the sub-question relates to a prerequisite (the road network is strengthened) before applying the judgement criterion (efficiency in strengthening the road network).

Analysis strategy

Four strategies can be considered:

  • Change analysis, which compares measured / qualified indicators over time, and against targets
  • Meta-analysis, which extrapolates upon findings of other evaluations and studies, after having carefully checked their validity and transferability
  • Attribution analysis, which compares the observed changes and a "without intervention" scenario, also called counterfactual
  • Contribution analysis, which confirms or disconfirms cause-and-effect assumptions on the basis of a chain of reasoning.

The three last strategies are appropriate for the questions which require a cause-and-effect analysis. The first one is appropriate in other instances.

Investigation areas

The evaluation team may consider collecting and analysing data at the level of the intervention as a whole, or investigating some areas more specifically, for instance:

  • All sub-questions will be addressed through an investigation into a selection of countries and will include country notes and a visit to each country.
  • In addition to using national statistics, the evaluation team investigates a selection of districts respectively typical of (a) the targeted group /area, and (b) best performing groups / areas.
  • The evaluation team will carefully select ten projects which will be subject to an in-depth investigation in order to address some of the sub-questions.

Example

 

 

.

OPTIMISING THE OVERALL DESIGN

.

What is the purpose?

To finalise the overall evaluation method in a way which cuts across the evaluation questions and which makes a good enough mix of evaluation tools, considering the available time and resources.

When does it take place?

The evaluation method is designed through an iterative process at three levels:

  • A question-by-question approach, allowing the evaluation team to prepare the design tables with the aim to adequately answer each question.
  • An overall approach which cuts across the questions, and which allows the evaluation team to optimise the evaluation method as a whole, whilst matching time and resource constraints.
  • A series of specifically developed tools.

Several iterations may be needed in order to allow the evaluation team to optimise the overall design whilst ensuring a high quality to each question.

Selecting tools

In parallel with the design tables, which are established on a question-by-question basis, the evaluation team designs its overall evaluation method which covers all questions and includes a number of tools such as:

  • A documentary analysis applying to a given set of reports
  • An analysis of context indicators
  • A series of interviews with stakeholders belonging to a given category
  • A questionnaire
  • A series of case studies
  • A series of focus groups or participatory meetings
  • An expert panel
  • The building and analysis of an ad hoc data base

These examples are all but limitative. The evaluation team may even have to develop a completely new tool, where relevant.

Combining tools and questions

The evaluation team draws up the list of all evaluation tools suggested in the design tables. Each tool is then considered from the viewpoint of its capacity to help answering several questions and sub-questions, for instance:

  • A series of interviews with Government officials: this tool may help answering questions related to "alignment of EC strategy on Government priorities", "policy dialogue in major policy domains", and "co-ordination among development partners".
  • A focus group gathering participants interested in primary education: this tool may help answering questions related to "efficiency of selected funding modalities", "mainstreaming of gender equality", and "improved access to basic education for the poor".

An image of this process is given by the matrix below in which the first tool helps answering sub-questions Aa, Cf and Eb, etc.

This image suggests that the set of tools will provide the evaluation team with possibilities of cross-checking data on sub-questions Aa and Eb. Tools and questions are combined in order to optimise such possibilities of cross-checking.
The above diagram is for explanatory purpose only. In the real world, the use of a matrix may be unpractical if the number of tools and/or sub-questions exceeds a dozen, which is typically the case in a country strategy evaluation. A pragmatic approach consists of drawing up a list of all questions and sub-questions addressed when developing each tool. An example is provided on this site.

Verifying the adequacy of tools

The evaluation team confirms that each tool envisaged is adequate in the sense that:

  • It fulfils the required function, e.g. a focus group is adequate for collecting beneficiaries' perceptions; a case study is adequate for understanding cause-and-effect mechanisms; a series of interviews is adequate for collecting the views of officials in partner institutions, and so on.
  • It is in line with the selected analysis strategy, e.g. a series of case studies is adequate in the instance of a contribution analysis; a questionnaire to beneficiaries and non-beneficiaries is adequate in the instance of an attribution analysis; a panel of experts is adequate in the instance of a meta-analysis, and so on.
  • It fits into the identified constraints, e.g. a questionnaire survey is appropriate if, and only if a satisfactory sample can be drawn; a database can be analysed if, and only if missing data are in limited numbers.

The evaluation team explains and justifies its main technical choices, with alternative options, pros and cons, and associated risks.

Articulating tools

The evaluation team assesses whether the selected tools may reinforce one another, for instance:

  • A series of interviews may help in identifying relevant papers which will be submitted to a documentary analysis.
  • A series of interviews may help in selecting the stakeholders to be invited to a focus group, and the issues to be discussed.
  • A series of interviews may help in refining the questions to be put to a sample of beneficiaries.
  • A series of case study monographs may be reviewed by a panel of experts with a view to strengthening the analysis and to deriving more accurate findings.

Preparing the overall assessment

In the process of progressively optimising its design, the evaluation team examines all the design tables in a cross-cutting manner with a view to preparing its final synthesis, i.e. an overall assessment that draws upon the answers to all evaluation questions.
More than being a mere compilation of findings and conclusions, the overall assessment may involve specific analyses which deserve to be designed in advance, for instance:

  • Specific themes like policy dialogue with the partner country's Government, decentralisation of EC procedures, etc. This calls for adequate sub-questions to be inserted in several design tables.
  • If the selected questions, sub-questions and tools leave a significant part of the support unexplored, then a few additional studies may be planned in order to provide a broader picture of the overall support (e.g. meta-analysis of monitoring reports). An overall analysis of management databases may also be undertaken with the same purpose.

Allocating resources

In the process of designing its method, the evaluation team tries to adequately share its limited resources between questions and sub-questions. Some questions deserve to be addressed with costly tools such as questionnaire surveys of end users, several case studies, focus groups, etc. Other questions should rather be answered on the basis of a documentary analysis only and a few interviews with EC and Government officials.
One reason why a question should be allocated more resources is the need for credibility. If a question is considered to be politically sensitive and if the answer is likely to trigger debate, then it deserves a stronger design, possibly with more cross-checking of sources, larger samples, several focus groups or case studies, etc.
It is also wise to invest substantial resources in a question that raises feasibility problems. The following are examples of specific difficulties calling for a particularly strong design:

  • A question relates to an innovative aspect of the intervention in an area where expertise is scarce.
  • A question relates to a component of the intervention which has just reached the end users in the field, and the implementing actors have no idea on how the targeted people are reacting.
  • A question relates to a far-reaching impact that is logically distant from EC activities and the abundance of external factors makes it difficult to assess cause-and-effect relations.

A question is rarely unevaluable in the absolute. It is more likely to be an accumulation of difficulties and constraints that leads to feasibility problems. At the earliest stages of the evaluation, it is often possible to amend a difficult question so as to make it more evaluable. This can be done, for example, by limiting the scope of the question or choosing to apply it to a less distant effect or to a probable effect if the real effect is not yet observable. Once a difficult question has been validated, the evaluation team has to design an appropriate method, and to allocate adequate resources.

Cost and time constraints

Successive versions of the method are designed within the evaluation team until the following constraints are matched:

  • The implementation of the evaluation tools fit into the overall time schedule of the evaluation process
  • The cost of the evaluation tools (human resources, technical costs, travel and daily subsistence) fits into the overall budget of the evaluation
  • The availability of qualified workforce in the field is sufficient for implementing evaluation tools professionally

.

DEVELOPING A TOOL

.

No evaluation method without tools

Evaluation tools are needed in order to collect primary and secondary data, to analyse data, and to formulate value judgements (or reasoned assessments). The external evaluation team may carry out some of these tasks without evaluation tools, but several tools are always needed for answering each evaluation question.
Tools range from simple and usual ones like database extracts, documentary analysis, interviews or field visits, to more technical ones like focus groups, modelling, or cost benefit analysis. This site describes a series of tools that are frequently used.
By using an appropriate mix of tools, the evaluation team strengthens the evidence basis of the evaluation, the reliability of data, the validity of its reasoning, and the credibility of its conclusions.

How are tools chosen?

Tools are chosen during the iterative process of designing the overall evaluation method, with an aim to:

  • Contribute to answering all questions and sub-questions, and to formulating an overall assessment
  • Gather / collect data, assist in analyses and formulation of value judgement (or reasoned assessment)
  • Facilitate cross-checking and triangulation
  • Reinforce each other through appropriate combinations
  • Match contextual constraints like: availability of expertise and data, allocated resources, time schedule.

When is it developed?

All evaluation tools are developed progressively and finalised before the beginning of the field phase, although some tools need to be implemented, and therefore developed earlier in the process, e.g. interviews with key stakeholders at the inception stage, analysis of management databases at the desk phase, etc.

Developing a tool

Whilst the set of evaluation tools is to be selected as a part of the overall evaluation design, each tool is to be developed in a separate way. An example of developed tool is provided on this site.
Development a tool may be a matter of a section in the inception or desk report. However the task may needs to be further formalised, including in the form of fully fledged terms of reference, when several members of the evaluation team work separately, for instance if the works extend to different countries, or if the tool is being sub-contracted.
Tool development proceeds through seven steps as follows:

Questions and sub-questions

The evaluation team lists the questions and sub-questions that have to be addressed by the tool. It refers to the design tables.

Technical specifications

The evaluation team develops the technical specifications of the tool through a preparatory stage. Technical specifications depend on the type of tool. They cover issues like:

  • Sampling, selection of interviews, of case studies, of documents …
  • Questions, themes of a documentary analysis, content of a monograph, fields of a database …
  • Mode of implementation
  • Duration of interview, focus group, visit, …

Caution! - When developing a questionnaire or an interview guide, the evaluation team should not proceed by copying and pasting evaluation sub-questions. If evaluation questions and sub-questions are naïvely passed to informants, there are considerable risks of biases.
Technical specifications need to be further formalised when several members of the evaluation team work separately, especially if the works extend to different countries.

Risk management

The evaluation team assesses the main risks with data collection and analysis, as well as potential biases. As far as relevant, it prepares second best solutions in case the tool cannot be applied satisfactorily. The following lines provide examples of risks associated with evaluation tools, and examples of second best solutions:

  • Considering database X, if more than 20% data are still missing after 4 weeks, then analyse the available data and ask expert Y to comment upon the validity of findings.
  • Considering the series of interviews X, if more than 30% informants cannot be reached after 3 weeks, then collect alternative "second hand" information from expert Y.
  • Considering the questionnaire X, if the number of respondents falls below 200, then gather a focus group meeting and cross-check results with that of the questionnaire survey.

This list is not limitative.

Mode of reporting

The outputs vary from one tool to another. They may take the form of tables, lists of quotations, lists of verbatims, monographs, etc. The evaluation team decides on how the outputs will be reported, for instance:

  • Full length data delivered to the evaluation team leader
  • Full length data included in a CDROM attached to the final report (in which case, some documents may need to be anonymised)
  • Tables or boxes to be inserted in the final report

The evaluation team also decides on how to report about the implementation of the tool and the associated limitations if any, e.g.

  • Note to be inserted in the methodological annex appended to the final report
  • Short methodological comment to be inserted in the final report itself.

Responsibilities

The tasks and roles are shared among the evaluation team members, e.g.

  • Who is to implement the tool?
  • Who will ensure harmonisation in case several team members implement the tool in various countries / areas?
  • Who is to decide upon second best alternatives in case of difficulties?
  • Who is responsible for delivering data?
  • Who will write the methodological comments?
  • Who is to assure quality?

Quality

Quality criteria are precisely defined. Depending on the tool they may cover issues like:

  • Coverage of themes, questions, issues
  • Accuracy
  • Identification of potential biases
  • Respect of anonymity rules or other ethical rules
  • Formal quality, language, spelling, layout

This site proposes a series of quality check-lists for frequently used tools.

Time schedule and resources

In some instances, it may be necessary to specify practicalities like:

  • date of start / end
  • human resources allocated
  • travel arrangements
  • technical expenditures

Example

Evaluation tool: "Interviews Education"
Name of
tool
Interviews with stakeholders at national level in the area of education
Questions and sub-questions addressed Question E - Education

  • Has the design of EC support to primary education included a commitment to monitor the quality of educational services supplied to disadvantaged groups?
  • Was such monitoring actually undertaken?
  • Were the monitoring data subject to periodic discussion among development partners?
  • Have partners taken action as a follow up of such discussions?
  • Were such actions designed with a view to providing disadvantaged groups with high quality primary education?
  • Have other development partners pushed for monitoring the quality of educational services provided to disadvantaged groups?
  • Have non-state actors contributed to raising the issue of disadvantaged groups on the political agenda?
  • How far did other partners contribute to shaping the actions taken in favour of disadvantaged groups?
  • Is it acceptable for stakeholders to focus on the quality of teachers and premises as a way to judge discrimination in the access to primary schools?

Question D - Policy dialogue

  • What has the EC's input into policy dialogue been in the area of primary education? How does this input compare to that of other partners?
  • Did the EC input focus on the quality of educational services supplied to disadvantaged groups?

Question X - Cross-cutting issues

  • To what extent has the EC mainstreamed gender in its support to primary education? How does it compare to other partners in this respect?
  • Did the EC input focus on the quality of educational services supplied to disadvantaged groups?
Technical specifications Ten interviews at the Ministry of Education, Primary Education Inspectorate, and at the National Teachers Union and NGOs most active in the field of primary education.
Face-to-face 40-minute interviews; contacts made by the evaluation team; EC Delegation informed; guarantee of anonymous treatment; minutes not sent to informants; informants invited to the final discussion seminar.
Risk management Some informants may express themselves in a purposely biased way. Interviews should therefore focus on facts rather than opinions.
Mode of reporting (internal to evaluation team)
  • Identification fiches for all interviews, with full details
  • Minutes including profile of interviewee, verbatims (at least one per interview and per sub-question if relevant), document provided, contact provided, comment on potential biases
  • Overall synthesis per sub-question
Mode of reporting (in the final report)
  • List of interviewees (to be inserted in the overall list of interviewees appended to the report)
  • Anonymised verbatims (in a CDROM attached to the report)
  • Methodological description of the tool with comments on biases (about 50 lines to be inserted in the methodological appendix)
Responsibilities Local education expert: contacts with informants, draft interview guide, interviews and reports
Team leader: approval of interview guide
Quality Quality criteria: coverage of sub-questions with verbatims, self-assessment of biases
Quality control by team leader
Informants may be contacted again by another team member for verification. They should be informed of that.
Time schedule Start date, interviews between …and …
Date of reporting
Resources … person x days

.

FEASABILITY (EVALUABILITY) OF A QUESTION

.

What is this?

Certain questions are easy to answer while others may raise evaluability problems. It is therefore necessary to assess the feasibility of evaluation questions from the outset.

What is the purpose?

  • To ensure that the evaluation provides reliable and credible answers to the questions asked.
  • To exclude or amend questions that are too difficult to answer.
  • To adjust the available time and other resources in case difficult questions have to be answered.

What has to be taken into account?

To establish whether a question is evaluable, we check:

  • Whether the concepts are stable (Are the main terms of the question understood by everyone in the same way?)
  • Whether explanatory assumptions can be identified (What are the external factors and the cause-and-effect assumptions?)
  • Whether access to the field and data collection entail major difficulties.

What are the most frequent limitations?

A highly innovative activity

If the question concerns an innovative instrument, activity or procedure, the following difficulties may arise:

  • It is difficult to define the terms of the question without ambiguity.
  • There is a lack of expertise to understand the cause-and-effect mechanisms.

A very recent activity

If the question concerns a recently implemented activity:

  • The activity has not yet produced observable effects
  • The informants have not yet stabilised their opinions.

Managerial weaknesses

If the question concerns activities in which there are or were managerial weaknesses, the following difficulties may be encountered:

  • The monitoring data and reports are inadequate or unreliable.
  • The managerial difficulties have generated conflicts that limit access to certain informants or cause those informants to bias their answers.

In case of a strong suspicion of illicit or illegal practices, it is preferable to postpone the evaluation question for later and to start with an audit.

A scope that is too complex

If the question concerns a multi-dimensional or multi-sectoral scope, the following difficulties may be encountered:

  • In view of the available time and budget, there are too many data to collect, informants to meet and analyses to perform, and they are too dispersed.

A far-reaching impact

If the question concerns a far-reaching impact which is connected to the evaluated activity through a long chain of causes and effects, then the following difficulties may be encountered:

  • There are so many external factors and they are so influential that it becomes impossible to analyse the contribution of the intervention.

An intervention that is too marginal

If the question concerns a very small activity compared to other neighbouring policies or to the context, the following difficulties may be encountered:

  • " The evaluated activity does not attain the critical mass that would allow an analysis of its contribution.

Recommendations

A question is rarely unevaluable in the absolute. It is more likely to be an accumulation of difficulties and constraints that makes the question difficult.
When a question is considered too difficult, it is often possible to amend it to make it more evaluable. This can be done, for example, by limiting the scope of the question or choosing to apply it to a less distant effect or to a probable effect if the real effect is not yet observable. This option may be preferable to exclusion of the question.
Policy-makers or any other actor may insist on asking a question that is clearly too difficult. In such an instance, it is useful to provide a full explanation of the difficulties and to show that evaluations have limitations and cannot answer all questions.

Evaluation methodological approach > EN: Methodological bases and approach > Evaluation methods > Data collection

Data collection

.

This section is structured as follows:

.

For the purpose of answering questions, the evaluation team collects data that are already available (secondary data) and applies data collection tools with a view to obtaining new information (primary data).

.

USING AVAILABLE DATA

.

What are they?

To avoid duplicating efforts, running up unnecessary costs and tiring the informants, it is recommended wherever possible to rely on existing information (secondary data), such as administrative data, research, previous evaluations, management and monitoring databases, statistics.

This information can be obtained at a lower cost. It can help to provide partial answers to some of the questions asked.

Why is it important?

  • Existing (secondary) data are cheaper and quicker to gather than primary data.
  • If the evaluation team does not acknowledge relevant existing data, this will undermine the credibility of the whole evaluation.

Main channels for identifying and gathering secondary data

  • Managers, implementing agencies, operators and partners
  • Experts in the domain under consideration
  • The Internet
  • Statistical offices and monitoring bodies
  • Scientific and professional literature

Reliability problems

Before making use of secondary data, particular attention should be paid to the following points:

  • Definitions: do the data measure/reflect what is to be addressed?
  • Missing data
  • Accuracy: are the sources and measurement method reliable?
  • Age: when did measurement actually take place?
  • Reliability over time: did definitions or measurement methods change in the period covered?
  • Comparability: are definitions and measurement methods consistent from one place to another?
  • Aggregating data: are pooled data consistent with one another?
  • Disaggregating: is it possible to break down data into sub-territories or sub-sectors while keeping a large enough statistical basis?

Recommendations

  • Where possible, multiple sources of evidence should be used so as to follow cross-checking.
  • Be aware of vested interests when using secondary data. Those responsible for their compilation may have reasons for showing an optimistic or pessimistic picture. For instance, it has been reported that officials responsible for estimating food shortages exaggerated the figures before sending aid requests to potential donors.

.

MOST FREQUENT DIFFICULTIES WITH DATA COLLECTION

.

What are they? How to cope with them?

Even if the data collection programme has been wisely prepared, the evaluation team often encounters problems during its field work. The most frequent difficulties occur with:

- Access to informants

The sampling process proves to be difficult. Decide whether or not a reduced sample size is likely to provide statistically valid findings. If not, apply another technique such as the focus group.
An informant does not express him/herself freely Focus interviews on facts rather than opinions.

Propose to keep collected information anonymous and explain how this will be secured.

An informant expresses him/herself in a way which seems purposely biased Focus demands on facts, not on opinions.

Cross-check with other information sources

- Cultural gap

An informant or an information source can be accessed in the local language only. The evaluation team should include at least one member who is fluent in the local language (translation and interpretation always generate important information losses).
There is a large cultural gap between the evaluation team and the surveyed group. The evaluation team should include one or several members capable of bridging the gap between the two cultures.

- Lack or weakness of data

An information source proves to be incomplete. If possible, extrapolate missing data and cross-check with other sources.
An information source proves to be unreliable. If possible, understand the biases, adjust data and cross-check with other sources.

Recommendations

  • Any evaluation creates a feeling of uncertainty, which makes some stakeholders reluctant to co-operate, if not hostile. In such cases keep a positive attitude, emphasise the intended use of the evaluation, promise impartiality, and focus on facts rather than opinions.
  • If an information source is not accessible or if a survey technique is not manageable, change the data collection work plan in order to collect similar information through other sources.
  • Pay sustained attention to biases and risks of unreliability. Strive to understand them. Report on them.
  • Avoid relying on one single information source in order to facilitate cross-checking at the analysis stage. This will also make it easier to manage if one of the sources cannot be used.

.

RELIABILITY OF COLLECTED DATA

.

What are the risks?

While gathering information, the evaluation team faces various risks of biases which may undermine the reliability of collected data.

Why should biases be considered carefully?

  • For improving the reliability of data collection
  • For assessing the quality of the evaluation
  • For understanding the limitations of conclusions which draw on unreliable data

Most frequent biases

- Confirmation bias

This risk is a threat to all data collection approaches. It results from a tendency to seek out evidence that is consistent with the intervention logic, rather than evidence that could disprove it.

When subject to this bias, the evaluation team and informants tend to focus on intended effects and systematically to overlook external factors, unintended effects, negative effects, interactions with other policies, outside stakeholders, alternative implementation options, etc.

This bias is avoided by relying on independent and professional evaluators.

- Self-censorship

In some instances, informants may be reluctant to freely answer questions, simply because they feel at risk. They tend to rigidly express the views of their institution or their hierarchy.

This bias is combated by guaranteeing confidentiality and anonymity in the treatment of answers. The interviewer should also insist on factual questions and avoid collecting opinions.

- Informants' strategy

Those who have stakes in the intervention may distort the information they provide, with the aim of obtaining evaluation conclusions closer to their views.

This bias will be reduced if the whole range of stakeholders is included in the data collection work plan and if various sources of information are cross-checked.

- Unrepresentative sample

This bias may be a matter of concern if the evaluation team generates quantitative data through a questionnaire survey. It should also be considered when using secondary data obtained from a questionnaire survey.

In this instance, the evaluation team should verify that the sample of surveyed informants is large enough and representative of the population as a whole.

- Question induced answers

This bias and the following ones are frequent in interviews and questionnaires. The way in which questions are asked by interviewers or the interviewer's reaction to answers can generate a bias which is either positive or negative. Even the order of the questions in a questionnaire may change the substance of the answers. This bias will be limited by having questionnaires designed and tested by experienced professionals.

- Empathy bias

Interviewees may not have a pre-determined opinion about the questions put to them. They try to make up their mind in a few seconds when responding to the interviewer or to the questionnaire. While doing so, they may be strongly influenced by the context.

Especially in the case of interviews, the evaluation team has to create a friendly (empathetic) atmosphere, at least for the sake of achieving a high rate of answers and fast completion of the survey.

The combination of the two introduces a systematic positive bias in the answers, which tends to overestimate the benefits of the intervention and to underestimate the role of external factors.

This bias is prevented by relying on properly trained interviewers.

- Sample selection bias

People who agree to be interviewed may not be representative of the overall target audience.
This bias could be controlled by undertaking a special qualitative survey on a few "non-respondents", although this exercise brings additional costs.

Recommendations

  • Rely on an independent and professional evaluation team in order to limit confirmation biases.
  • Systematically mix positive and negative questions in order to reduce empathy bias and question bias.
  • Be highly credible when promising confidentiality and anonymity in order to limit respondents' self-censorship - and keep such promises strictly.
  • Never rely on a single category of stakeholder (e.g. programme managers, beneficiaries) in order to reduce strategic bias.
Evaluation methodological approach > EN: Methodological bases and approach > Evaluation methods > Analysis

Analysis

.

This section is structured as follows:

.

KEY ELEMENTS

.

An analysis is required to convert data into findings, which themselves call for a judgement in order to be converted into conclusions. The analysis is carried out on a question-by-question basis, in the framework of an overall design cutting across all questions.

Data, evidence and findings

Any piece of qualitative or quantitative information that has been collected by the evaluation team is called data, for instance:

  • Document X indicates that the number of pupils has grown faster than the number of teachers in poor rural areas (data)

A piece of information is qualified as evidence as soon as the evaluation team assesses it as reliable enough, for instance:

  • Document X, quoting Education Ministry data that are considered reliable, indicates that the number of pupils has grown faster than the number of teachers in poor rural areas (evidence)

Findings establish a fact derived from evidence through an analysis, for instance:

  • The quality of primary education has decreased in poor rural areas (finding)

Some findings are specific in that they include cause-and-effect statements, for instance:

  • The EC has not significantly contributed to preventing the quality of primary education from decreasing in poor rural areas (cause-and-effect finding)

Findings do not include value judgements, which are embedded in conclusions only, as shown below:

  • The EC has successfully contributed to boosting the capacity of the educational system to enrol pupils from disadvantaged groups, although this has been at the expense of quality (conclusion).

Strategy of analysis

Four strategies can be considered:

  • Change analysis, which compares measured / qualified indicators over time, and/or against targets
  • Meta-analysis, which extrapolates upon findings of other evaluations and studies, after having carefully checked their validity and transferability
  • Attribution analysis, which compares the observed changes with a "without intervention" scenario, also called counterfactual
  • Contribution analysis, which confirms or disconfirms cause-and-effect assumptions on the basis of a chain of reasoning.

The first strategy is the lightest one and may fit virtually all types of questions, for instance:

  • To what extent are the EC priorities still in line with the identified challenges?
  • To what extent has the support taken into account potential interactions and conflicts with other EC policies?
  • To what extent has the EC mainstreamed a given cross-cutting issue in the implementation of its interventions?

The three last strategies are better at answering cause-and-effect questions, for instance:

  • To what extent has the EC contributed to achieving effect X?
  • To what extent has the EC contributed to achieving effect X sustainably?
  • To what extent has the EC contributed to achieving effect X at a reasonable cost?

The choice of the analysis strategy is part of the methodological design. It depends on the extent to which the question raises feasibility problems. It is made explicit in the design table.
Once the strategy has been selected and the data collected, the analysis proceeds through all or part of the following four stages: data processing, exploration, explanation, confirmation.

Data processing

The first stage of analysis consists in processing information with a view to measuring or qualifying an indicator, or to answering a sub-question. Data are processed through operations such as cross-checking, comparison, clustering, listing, etc.

  • Cross-checking is the use of several sources or types of data for establishing a fact. Systematic cross-checking of at least two sources should be the rule, although triangulation (three sources) is preferable. The cross-checking should involve independent documents. A document that quotes another document is not an independent source. An interviewee who has the same profile as another interviewee is not an independent source.
  • Comparison proceeds by building tables, graphs, maps and/or rankings. Data can be compared in one or several dimensions such as time, territories, administrative categories, socio-economic categories, beneficiaries and non-beneficiaries, etc. The evaluation team typically measures change by comparing quantitative indicators over time. Comparisons may also be qualitative, e.g. ranking a population's needs as they are perceived by interviewees.
  • Clustering proceeds by pooling data in accordance with predefined typologies, e.g. EC support per sector, beneficiaries per level of income.
  • Listing proceeds by identifying the various dimensions of something, for instance the various needs of the targeted group as expressed in a participatory meeting, the various effects of a project as perceived by field level stakeholders, the various strengths and weaknesses of the EC as perceived through interviews with other donors' staff.

Provisional findings emerge at this stage of the analysis. Further stages aim to deepen and to strengthen the findings.

Exploration

The exploratory analysis aims to improve the understanding of all or part of the evaluated area, especially when knowledge is insufficient and expertise is weak, or when surprising evidence does not fit available explanations.

The exploratory analysis delves deeper and more systematically into the collected data in order to discover new plausible explanations such as:

  • New categories / typologies
  • Unforeseen explanatory factors
  • Factors favouring / constraining sustainability
  • Unintended effects
  • New cause-and-effect assumptions.

The exploratory stage may not be needed for all questions. When such an analysis is carried out, brainstorming techniques are appropriate. The idea is to develop new plausible explanations, not to assert them.

Explanation

This next stage ensures that a sufficient understanding has been reached in terms of:

  • Precisely defined concepts, categories and typologies
  • Plausible cause-and-effect explanations
  • Identification of key external factors and alternative explanations.

Depending on the context and the question, the explanation builds upon one or several of the following bases:

  • Diagram of expected effects
  • Expertise of the evaluation team
  • Exploratory analysis

A satisfactory explanation (also called explanatory model) is needed for finalising the analysis.

Confirmation

The last stage of the analysis is devoted to confirming the provisional findings through a valid and credible chain of arguments. This is the role of the confirmatory analysis.

To have a finding confirmed, the evaluation undertakes a systematic self-criticism by all possible means, e.g. statistical tests, search for biases in data and analyses, check for contradictions across sources and analyses.
External criticism from experts or stakeholders is also considered.

.

ANALYSIS STRATEGY

.

Cause-and-effect analysis

What does this mean?

Approach through which the evaluation team asserts the existence of a cause-and-effect link, and/or assesses the magnitude of an effect.

Attribution or contribution

- Attribution analysis

Attribution analysis aims to assess the proportion of observed change which can really be attributed to the evaluated intervention. It involves building a counterfactual scenario.

- Contribution analysis

Contribution analysis aims to demonstrate whether or not the evaluated intervention is one of the causes of observed change. It may also rank the evaluated intervention among the various causes explaining the observed change. Contribution analysis relies upon chains of logical arguments that are verified through a careful confirmatory analysis.
It comprises the following successive steps:

  • Refining the cause-and-effect chains which connect design and implementation on the one hand, and the evaluated effect on the other. This step builds upon available explanations pertaining to the evaluated area. Explanations derive from the diagram of expected effects drawn in the first phase of the evaluation, from the evaluation team's expertise, and from exploratory analyses.
  • Gathering evidence related to each link in the cause-and-effect chain, including findings of similar studies, causal statements by interviewees, and evidence from in-depth inquiries.
  • Gathering evidence related to other explanations (other interventions, external factors).
  • Developing a step-by-step chain of arguments asserting that the intervention has (or has not) made a contribution, and possibly ranking the intervention among other contributions.
  • Submitting the reasoning to systematic criticism until it is strong enough.

Analytical approaches

- Counterfactual

The approach is summarised in the diagram below:


The "policy-on" line shows the observed change, measured with an impact indicator, between the beginning of the evaluated period (baseline) and the date of the evaluation. For instance: local employment has increased, as has literacy. The impact accounts for only the share of this change that is attributable to the intervention.
The "policy-off" line, also called the counterfactual, is an estimate of what would have happened without the intervention. It can be obtained with appropriate approaches like comparison groups or modelling techniques. Impact is assessed by subtracting the policy-off estimate from the observed policy-on indicator.
The assessed impact, derived from an estimate of the counterfactual, is itself an estimate. In other words, impacts cannot be directly measured. They can simply be derived from an analysis of impact indicators.
Only a counterfactual allows for a quantitative impact estimate. When successful, this approach therefore has a high potential for learning and feedback. It is nevertheless relatively demanding in terms of data and human resources, which makes it somewhat unusual in evaluation practice in developing countries.

- Case studies

Another analytical approach relies on case studies. It builds upon an in-depth inquiry into one or several real life cases selected in order to learn about the intervention as a whole. Each case study monograph describes observed changes in full detail. A good case study also describes the context in detail and all significant factors which may explain why the changes occurred or did not occur.
In a case study approach, the evaluation team analyses the whole set of collected facts and statements and checks whether they support assertions like "the change can be attributed to the intervention", "the change can be attributed to another cause", "the absence of change can be attributed to the intervention", etc.
Just one case study may fully demonstrate that the intervention does not work as intended and may provide a convincing explanation for that. However, it is worth confirming such a finding by one or two additional case studies.
On the other hand, it takes more case studies to demonstrate that the intervention works, because all alternative explanations should be carefully investigated and rejected.
If professionally implemented, case studies provide for a highly credible and conclusive contribution analysis. The approach is nevertheless fairly demanding in terms of time and skilled human resources.

- Causal statements

The approach builds upon documents, interviews, questionnaires and/or focus groups. It consists in collecting stakeholders' views about causes and effects. Statements by various categories of stakeholders are then cross-checked (triangulated) until a satisfactory interpretation is reached. A panel of experts may be called to help in this process.
A particular way of implementing this approach consists in collecting beneficiaries' statements about impacts or direct results. Typically, a sample of beneficiaries is asked questions like "How many jobs would you say have been created/lost in your firm as a result of the support received?" or "To what extent is your present situation/behaviour attributable to your participation in the intervention?" In this approach the interviewee is asked to apply the policy-off scenario on his/her own.
Evaluation teams tend to prefer this approach which is far more feasible, but nobody should forget that the difficulty is transferred to the respondents. Most often, interviewees do not have a clear view of the policy-off scenario. They try to make up their minds in a few seconds during the interview and in doing so are subject to all kinds of biases.
When interviewing beneficiaries, the evaluation team often faces difficulties due to deadweight, since interviewees tend to exaggerate the effect of the evaluated intervention on their own behaviour or situation. In other words, they tend to underestimate the changes that would have occurred in the absence of the intervention. This results in a bias which is called deadweight.
In order to avoid this bias, the evaluation team should never rely upon a single naive question like "How many new jobs have been created as a result of the support received?" or "How much has your income increased as a result of the project?" By contrast, multiple triangulated questions may enable the evaluation team to assess and reduce the bias. Beneficiaries' statements are called "gross effects" (including bias) whilst the evaluation team's estimate is called a "net effect" (corrected from bias).

- Meta-analysis

This approach builds upon available documents, for instance:

  • Previous works pertaining to the evaluation as a whole (monitoring, audit, review, etc.)
  • Recent reports related to a part of the intervention, e.g. a project, a sector, a cross-cutting issue (evaluation reports but also monitoring, audit, review, etc.)
  • Lessons learnt from other interventions and which can be used in answering the question.

In performing meta-analyses, the evaluation team needs to (1) assess the quality of information provided by the reviewed documents, and (2) assess the transferability to the context of the evaluation underway.

- Generalisation

The first two approaches (counterfactual and case studies) have the best potential for obtaining findings that can be generalised (see external validity), although in a different way. Findings can be said to be of general value when all major external factors are known and their role is understood. Counterfactual approaches build upon explanatory assumptions about major external factors, and strive to control such factors through statistical comparisons involving large samples. Case studies strive to control external factors through an in-depth understanding of cause-and-effect mechanisms.

  Counterfactual Case studies
External
factors
Identified in advance Identified in advance or discovered during the study
Control of
external
factors
Quantitative, large samples, statistical techniques In-depth understanding of cause-and-effect mechanisms

Recommendation

The evaluation team should be left with the choice of its analysis strategy and analytical approach.

Cause-and-effect questions

What does this mean?

Cause-and-effect questions pertain to the effects of the evaluated intervention. They are written as follows:

  • To what extent has [the intervention] contributed to [the expected change]?
  • How far has [the intervention] helped to achieve [the expected change]?

These questions call for an observation of change, and then an attribution of observed change to the intervention, or an analysis of the intervention's contribution to observed changes.
Questions pertaining to direct and short-term effects can easily be answered. Far-reaching effects (e.g. poverty reduction) raise feasibility problems.

Causality and evaluation criteria

Effectiveness and impact questions tend to be cause-and-effect questions in the sense that they link the evaluated intervention (the cause) to its effects.
Efficiency and sustainability questions are also cause-and-effect questions since actual effects have to be analysed first, before being qualified as cost-effective or sustainable.
Generally speaking, relevance and coherence questions are not cause-and-effect questions. Typical examples are:

  • How far are EC objectives in line with needs as perceived by the targeted population?
  • To what extent are the effects of the intervention and the effects of other EC policies likely to reinforce one another?

The latter example involves causes and effects, but only in a prospective and logical manner. The evaluation team is not expected to assert the existence of cause-and-effect links and/or to assess the magnitude of actual effects.
Exceptionally, some relevance questions may call for cause-and-effect statements, for instance:

  • How far did the involvement of non-state actors in the design of the EC strategy contribute to better adjustment of priorities to needs as perceived by the targeted population?

Questions pertaining to the EC value added may be cause-and-effect questions if the evaluation team attempts to assert the existence or the magnitude of an additional impact, due to the fact that the intervention took place at European level.

Caution!

Questions which do not require a cause and effect analysis do nevertheless call for a fully-fledged analysis covering all or part of data processing, exploration, explanation and confirmation.

.


Counterfactual

.

What does this mean?

The counterfactual, or counterfactual scenario, is an estimate of what would have occurred in the absence of the evaluated intervention.
The main approaches to constructing counterfactuals are

  • Comparison group
  • Modelling

What is the purpose?

By subtracting the counterfactual from the observed change (factual), the evaluation team can assess the effect of the intervention, e.g. effect on literacy, effect on individual income, effect on economic growth, etc.

Comparison group

One of the main approaches to counterfactuals consists in identifying a comparison group which resembles beneficiaries in all respects, except for the fact that it is unaffected by the intervention. The quality of the counterfactual depends heavily on the comparability of beneficiaries and non-beneficiaries. Four approaches may be considered for that purpose.

Randomised control group

This approach, also called experimental design, consists in recruiting and surveying two statistically comparable groups. Several hundred potential participants are identified and asked to participate or not in the intervention, on a random basis. The approach is fairly demanding in terms of preconditions, time and human resources. When the approach is workable and properly implemented, most external factors (ideally all) are neutralised by statistical rules, and the only remaining difference is participation in the intervention.

Adjusted comparison group

In this approach a group of non-participants is recruited and surveyed, for instance people who have applied to participate but who have been rejected for one reason or another. This approach is also called quasi-experimental design. In order to allow for a proper comparison, the structure of the comparison group needs to be adjusted until it is similar enough to that of participants as regards key factors like age, income, or gender. Such factors are identified in advance in an explanatory model. The structure of the comparison group (e.g. per age, income and gender) is adjusted by over- or under-weighting appropriate members until both structures are similar.

Matching pairs

In this approach a sample of non-participants is associated with a sample of beneficiaries on an individual basis. For each beneficiary (e.g. a supported farmer), a matching non-participant is found with a similar profile in terms of key factors which need to be controlled (e.g. age, size of farm, type of farming). This approach often has the highest degree of feasibility and may be considered when other approaches are unpractical.

Generic comparison

The counterfactual may be constructed in abstracto by using statistical databases. The evaluation team starts with an observation of a group of participants. For each participant, the observed change is compared to what would have occurred for an "average" individual with the same profile, as derived from an analysis of statistical databases, most often at national level.

Comparative approaches

Different forms of comparison exist, each with pros and cons, and varying degrees of validity.

  • An "afterwards only" comparison involves analysis of the differences between both groups (participants and non-participants) after the participants have received the subsidy or service. This approach is easy to implement but neglects the differences that may have existed between the two groups at the outset.
  • A "before-after" comparison focuses on the evolution of both groups over time. It requires baseline data (e.g. through monitoring or in the form of statistics, or through an ex ante evaluation), something which does not always exist. Baseline data may have to be reconstructed retrospectively, which involves risks of unreliability.

Strengths and weaknesses in practice

A well-designed comparison group provides a convincing estimate of the counterfactual, and therefore a credible base for attributing a share of the observed changes to the intervention. A limitation with this approach stems from the need to identify key external factors to be controlled. The analysis may be totally flawed if an important external factor has been overlooked or ignored. Another shortcoming stems from the need to rely upon large enough samples in order to ensure statistical validity. It is not always easy to predict the sample size which will ensure validity, and it is not infrequent to arrive at no conclusion after several weeks of a costly survey.

Modelling

The principle is to run a model which correctly simulates what did actually occur in reality (the observed change), and then to run the model again with a set of assumptions representing a "without intervention" scenario. In order to be used in an evaluation, a model must include all relevant causes and effects which are to be analysed. These are at least the following:

  • Several causes including the intervention itself and other explanatory factors.
  • The effect to be evaluated.
  • A mathematical relation between the causes and the effect, including adjustable parameters.

Complex models (e.g. macro-economic ones) may include hundreds of causes, hundreds of effects, hundreds of mathematical relations, hundreds of adjustable parameters, and complex cause-and-effect mechanisms such as causality loops. When using a model, the evaluation team proceeds in three steps:

  • A first simulation is undertaken with real life data. The parameters are adjusted until the model reflects all observed change correctly.
  • The evaluation team identifies the "primary impacts" of the intervention, e.g. increase in the Government's budgetary resources, reduction of public debt, reduction of interest rates, etc. A set of assumptions is elaborated in order to simulate the "without intervention" scenario, that is to say, a scenario without the "primary impacts".
  • The model is run once again in order to simulate the "without intervention" scenario (i.e. the counterfactual). The impact estimate is derived from a comparison between both simulations.

Modelling techniques are fairly demanding in terms of data and expertise. The workload required for building a model is generally not proportionate to the resources available to an evaluation. The consequence is that the modelling approach is workable only when an appropriate model and the corresponding expertise already exist.

.

EXTERNAL FACTORS

.

What are they?

Factors which are embedded in the context of the intervention and which hinder or amplify the intended changes while being independent from the intervention itself.

External factors are also called contextual, exogenous or confounding factors.

Why are they important?

  • To understand their influence in order to properly disentangle the effects of the intervention from those of other causes.
  • To identify the contextual factors which may prevent the transferability of lessons learned.
  • To adjust samples of participants and non-participants in order to make them truly comparable and to achieve internal validity.

Typical examples

Factors explaining participation in the intervention:

  • Potential applicants are (are not) familiar with the implementing agency.
  • They belong (do not belong) to a "club" of recurrent beneficiaries.
  • They have (do not have) a systematic grant-seeking behaviour.
  • They have (do not have) social or economic difficulties preventing them from applying for grants / loans.

Factors explaining the achievement of specific impacts:

  • Trainees had a high (low) level of education before entering the programme. In addition to the intervention this factor partly explains the skills they acquired.
  • The target public had a high (low) level of knowledge before being reached by an awareness-raising campaign. In addition to the intervention this factor partly explains their current level of awareness.
  • Beneficiary enterprises were large (small), young (old), and growing (declining) when they received support. In addition to the intervention these factors partly explain their current level of competitiveness.

Factors explaining global impact

  • Local agriculture has benefited from exceptionally favourable weather during the programming period (or the opposite). In addition to the intervention this factor partly explains the change in farmers' income.
  • International economic trends were exceptionally favourable during the programming period (or the opposite). In addition to the intervention this factor partly explains the evolution of exports.

When dealing with such external factors, the evaluation may usefully consult the contextual indicators that are available on the web.

How can they be identified?

In a given evaluation, external factors are potentially numerous and it is crucial to highlight the most important ones. The following approaches may help:

  • Making the intervention logic explicit and uncovering implicit cause-and-effect assumptions
  • Analysing previous research and evaluation work
  • Taking expert advice
  • Carrying out qualitative surveys

Identifying external factors is one of the main purposes of the exploratory analysis.

Recommandations

Do not try to identify all possible external factors when clarifying the intervention logic in the structuring phase of the evaluation. They are simply too numerous. This task should be undertaken only when working on a given evaluation question, and only if the question involves a cause-and-effect analysis.

.

EXPLORATORY AND CONFIRMATORY ANALYSIS

.

Exploratory analysis

What does this mean?

If necessary, the evaluation team delves into the collected data in order to discover new plausible explanations such as:

  • New categories / typologies
  • Explanatory factors
  • New cause-and-effect assumptions
  • Factors favouring / constraining sustainability

What is the purpose?

  • To improve the understanding of all or part of the evaluated area, especially when existing knowledge and expertise are inadequate
  • To develop and refine explanatory assumptions.

How to carry out the exploratory analysis

The analysis explores the set of data (quantitative and qualitative) with a view to identifying structures, differences, contrasts, similarities and correlations. For example, the analysis involves:

  • Cross-cutting analyses of several case studies
  • Statistical comparisons cutting across management data bases, statistical data bases, and/or the results of a questionnaire survey
  • Comparisons between interviews and documents.

The approach is systematic and open-minded. Brainstorming techniques are appropriate. Ideas emerge through the first documentary analyses, interviews, and meetings. The exploration may continue through the field phase.

Confirmatory analysis

What does this mean?

Provisional findings progressively emerge during the first phases of the evaluation team's work. They need to be confirmed by sound and credible controls. That is the role of the confirmatory analysis.
In the particular case of cause-and-effect questions, the analysis is meant to disentangle several causes (the intervention and the external factors) in order to demonstrate the existence and/or assess the magnitude of the effect.

What is the purpose?

  • To ensure that the findings are sound and able to withstand any criticism when the report is published
  • To ensure that the findings are credible from the intended users' viewpoint
  • In the particular case of cause-and-effect questions, to distinguish actual effects from observed change

How is a confirmatory analysis performed?

For a finding to be confirmed, it is systematically criticised by all possible means, e.g.:

  • If the finding derives from a statistical analysis, are the validity tests conclusive?
  • If the finding was suggested by a case study, is it contradicted by another case study?
  • If the finding derives from a survey, can it be explained by a bias in that survey?
  • If the finding is based on an information source, is it contradicted by another source?
  • Is the finding related to a change that can be explained by external factors that the evaluation team may have overlooked?
  • Does the finding contradict expert opinions or lessons learned elsewhere and, if so, can this be explained?
  • Do the members of the evaluation reference group have arguments to contradict the finding and, if so, are these arguments justified?

Recommendations

  • Devote relatively long interactions to the discussion of the final report in order to allow for a careful confirmatory analysis. Ensure that the evaluation team has put aside sufficient resources for that purpose.
  • Not all findings require the same level of confirmation. Concentrate efforts on findings that support the most controversial conclusions, the lessons that are the most likely to be transferred, or the recommendations that are the most difficult to accept.
  • In order to enhance the evaluation's credibility, it is valuable to present the criticisms that the findings withstood during the confirmatory analysis, in an annex.

.

VALIDITY

.

What does this mean?

Validity is achieved when:

  • Conclusions and lessons are derived from findings in a way which ensures transferability (external validity)
  • Findings are derived from data without any bias (internal validity)
  • Collected data reflect the changes or needs that are to be evaluated without bias (construct validity)

What is the purpose?

A lack of validity may expose the evaluation to severe criticism from those stakeholders who are dissatisfied with the conclusions and recommendations, and who will point out any weaknesses they may have found in the reasoning.

Validity is part of the quality criteria. It should be given an even higher level of attention when the intended users include external stakeholders with conflicting interests.

External validity

Quality of an evaluation method which makes it possible to obtain findings that can be generalised to other groups, areas, periods, etc. External validity is fully achieved when the evaluation team can make it clear that a similar intervention implemented in another context would have the same effects under given conditions.

Only strong external validity allows one to transfer lessons learned. External validity is also sought when the evaluation aims at identifying and validating good practice.
External validity is threatened when the analysis fails to identify key external factors which are influential in the context of the evaluated intervention but would have a different influence in another context. External factors should not only be identified; the magnitude of their consequences should also be assessed. In the instance of a survey, this calls for larger and more diverse samples. In the instance of case studies, it requires a multiplication of the number of cases.

Internal validity

This is the quality of an evaluation method which, as far as possible, limits biases imputable to data collection and analysis. Internal validity is fully achieved when the evaluation team provides indisputable arguments showing that the findings derive from collected facts and statements.

Internal validity is a major issue in the particular case of cause-and-effect questions. When striving to demonstrate the existence and/or to assess the magnitude of an effect, the evaluation team is exposed to risks such as:

  • Overlooking cause-and-effect mechanisms which contradict initial assumptions.
  • Deriving impact estimates from a comparison of samples (e.g. participants and non-participants) that are not similar enough.
  • Having findings that do not withstand statistical tests because samples are not large enough.

Construct validity

This is the quality of an evaluation method which faithfully reflects the changes or needs that are to be evaluated. Construct validity is fully achieved when key concepts are clearly defined and when indicators reflect what they are meant to.

Construct validity is threatened if the evaluation team does not fully master the process of shifting from questions to indicators. Construct validity is also at risk when the evaluation team uses indirect evidence like proxies.

Recommandations

  • Never start analysing data without a thorough understanding of the context. Pay enough attention to identifying key external factors.
  • The valid analysis of an impact requires time and resources. For this reason, cause-and-effect questions may need more resources than others.
Evaluation methodological approach > EN: Methodological bases and approach > Evaluation methods > Judgment

Judgment

.

This section is structured as follows:

.

CONCLUSIONS AND LESSONS

.

What does this mean?

Conclusions provide clear answers to the evaluation questions. They incorporate value judgements. Lessons are transferable conclusions to subsequent cycles of the same intervention or of other interventions.

What is the purpose?

  • To ensure that the report provides true answers to the questions asked.
  • To clearly identify the points at which the evaluation involves value judgements, and to ensure that those judgements are not implicit or badly justified, as this would be a weakness.

How should they be formulated?

Conclusions and lessons stem from the preceding steps as follows:

Conclusions answer the questions

The questions asked at the beginning of the evaluation find their answers by means of the conclusions. A conclusion may answer several questions and several conclusions may answer a single question.

Provided that all the questions asked have been answered, the evaluation team can present additional conclusions to take into account unexpected and important information and results.

The conclusions follow from data and findings

Upon writing a conclusion, what is being judged is one aspect of the intervention, for example: a strategic guideline (Is it relevant?), a practice (Is it efficient?), an expected effect (Was it obtained?), or an unexpected one (Is it positive?).

Thus, conclusions stem from collected data and evidence, from analysis and interpretations performed, from findings and new knowledge generated.

Conclusions are based on judgement criteria

To formulate its conclusions, the evaluation team applies the judgement criteria (also called "reasoned assessment criteria") that were agreed upon in the first phase (desk) of the evaluation. Data collection and analysis are structured according to these criteria. As long as this is possible, the findings are compared against targets.

  • Example: The support has contributed towards increasing the number of qualified and experienced teachers by 30% in the areas where ethnic minority X concentrates (finding), which has allowed that area to catch up with the country average (target).

At the stage of the draft final report, the evaluation team may have to refine its judgement criteria and targets. In such a case, the issue is discussed with the reference group.

A lesson is a transferable conclusion

A lesson is a conclusion that can be transferred to subsequent cycles of the same intervention or to other interventions.

  • Example: By connecting its disbursements to specifically designed performance indicators, the EC can successfully contribute towards improving the capacity to enrol pupils from disadvantaged groups.

How should they be presented?

One chapter of the report introduces the conclusions relative to each question, as well as the conclusions that emerge from points not raised by the questions. The conclusions are organised in clusters in the chapter in order to provide an overview of the assessed subject.

The chapter does not follow the order of the questions or that of the evaluation criteria (effectiveness, efficiency, coherence, etc.)

It features references to the sections of the report or to annexes showing how the conclusions derive from data, interpretations, analysis and judgement criteria.

The report includes a self-assessment of the methodological limits that may restrain the range or use of certain conclusions.

A paragraph or sub-chapter picks up the 3 or 4 major conclusions organised by order of importance, while avoiding being repetitive. This practice allows to better communicate the evaluation messages that are addressed to policy makers within the Commission.

The conclusion chapter features not only the successes observed but also the issues requiring further thought on modifications or a different course of action.

Suggestions

  • Drafting a good conclusion requires that attention be paid to clarifying and justifying value judgements. It is therefore preferable to focus on a few key conclusions rather than on a large number of poorly justified ones. The evaluation team and the people in charge of quality assurance are therefore advised to carefully reread the final report and to eliminate any unessential and/or unintended value judgements.
  • It is difficult to calmly discuss the judgement criteria and the targets with the reference group at the end of the evaluation because everyone immediately sees the influence of the discussion on the conclusions. That is why it is preferable to go into as much detail as possible in the explanation of criteria at the inception stage.
  • Whenever possible, the evaluation report states the findings (which follow only from facts and analysis) separately from the conclusions (which involve a value judgement). This approach demands explicit judgement criteria and enhances quality.
  • The evaluation team presents its conclusions in a balanced way, without systematically favouring the negative or the positive conclusions.
  • If possible, the evaluation report identifies one or more transferable lessons, which are highlighted in the executive summary and presented in appropriate seminars or meetings so that they can be capitalised on and transferred.

.

RECOMMENDATIONS

.

What is this?

The recommendations are derived from conclusions.They are intended to improve or reform the intervention in the framework of the cycle under way, or to prepare the design of a new intervention for the next cycle.

What is the purpose?

  • To optimise use of the evaluation in the form of feedback.
  • To create a positive approach and easier take-up when the evaluation reveals problems.

How to draft and present them

The recommendations must be related to the conclusions without replicating them. A recommendation derives directly from one or more conclusions.

The recommendations must be clustered and prioritised. The report mentions the addressees of the recommendations, e.g. EC Delegation, services in charge of designing the next intervention, etc.

The recommendations are useful, operational and feasible, and the conditions of implementation are specified. Wherever possible and relevant, the main recommendations are presented in the form of options with the conditions related to each option, as well as the predictable consequences of the implementation of each option.

The recommendations are presented in a specific chapter. This chapter highlights the recommendations derived from the three or four main conclusions. 

How to promote them

The recommendations are valuable as far as they are considered and, if possible, taken up by their addressees.

To promote their take-up, the manager drafts a fiche contradictoire in order to:

  • List the recommendations in a shortened form
  • Collect the addressees' responses
  • Inform on actual follow-up to the recommendations, if any.

Advice

  • " If a recommendation does not clearly derive from the conclusions, it probably reflects preconceived ideas or the tactical interests of one of the stakeholders. Its presence in the report could then discredit the evaluation. The evaluation manager must therefore be careful to refuse any recommendations that are not based on collected and analysed data, and on the conclusions.

.

ETHICAL PRINCIPLES

.

What is this?

The conclusions include value judgements on the merits and worth of the intervention. This dimension of the evaluation exercise is particularly sensitive and the evaluation team has therefore to respect specific ethical principles.

What is the purpose?

  • To guarantee an impartial and credible judgement (also called "reasoned assessment").
  • To ensure that the judgement does not harm anyone.

What are the main principles?

- Responsibility for the judgement

The conclusions are primarily a response to questions. Members of the group are partially responsible for the judgement in so far as they orientate it through the evaluation questions they validate.

The external evaluation team also intervenes in the preparation of the judgement by making proposals to define the questions, clarify the judgement criteria and set the targets.

In the synthesis phase, the evaluation team applies the judgement criteria agreed on, as faithfully as possible, and produces its own conclusions. The conclusions are discussed within the reference group but remain the entire responsibility of the evaluation team.

As part of the quality assurance process, the evaluation manager can require sounder justification of a judgement, or better application of an agreed judgement criterion. By contrast, he or she cannot require the removal or amendment of a conclusion if it is methodologically sound.

- Legitimacy of the judgement

The questions and criteria take into account the needs and point of view of the public institution that initiated the evaluation.

The members of the reference group contribute different points of view, which reinforces the legitimacy of the evaluation.

During the desk phase the evaluation team holds interviews, which may enable it to identify other points that were not expressed by the reference group members. It makes them known in reference group meetings and may take them into account in the judgement criteria.

More generally, the evaluation team has a responsibility to bring to light important findings and judgement criteria which have arisen during the evaluation process, even if they are not covered by the evaluation questions, provided that such points are legitimate.

A point of view is legitimate if:

  • It is expressed by stakeholders or in their name.
  • It expresses an aspect of the public interest and not the individual interest of one person or the private interest of an organisation.
  • It is compatible with basic human rights.

- Impartiality of the judgement

The impartiality of the judgement concerns the entire evaluation, that is, the choice of questions and judgement criteria, the determination of targets and the formulation of conclusions.

The entire process is exposed to risks of partiality, for example:

  • The evaluation team favours its own preconceptions.
  • The evaluation team implicitly favours the point of view of one of the stakeholders.
  • The evaluation team does not hear, understand or take into account the point of view of one of the stakeholders.
  • The evaluation team systematically focuses on the negative or positive conclusions.

When there are differences in the way of judging, in the judgement criteria or in the target levels, impartiality consists in:

  • Making sure that evaluation team members are familiar with and respectful of beliefs, manners and customs of concerned groups.
  • Respecting all cultures and standpoints, whilst conforming to universal values as regards minorities and particular groups, such as women. In such matters, the United Nations Universal Declaration of Human Rights (1948) is the operative guide.
  • Being aware of asymmetrical power relations, and correcting the biases arising from them.
  • Making sure that all the opinions are heard, even if they are not conveyed by the loudest voices or the majority.
  • Reporting on differences in the reports and at reference group meetings.
  • Explaining the choices transparently (Who made the choice? Why? What were the alternatives?).

In case of divergence, a solution may consist in judging in relation to several criteria and/or formulating several conclusions that correspond to different points of view. This solution has the drawback of diluting the conclusions and thus of making the evaluation less conclusive.

It is often preferable to make choices and to explain them transparently.

- Protection of people

The conclusions concern the merits of the evaluated intervention, not the people who implement it or benefit from it.

Individuals' names are cited only when this enhances the credibility of the evaluation. The evaluation team must respect people's right to provide information in confidence and ensure that sensitive data cannot be traced to its source. Before citing a person or organisation, the evaluation team or any other evaluation actor anticipates and avoids the risks involved for that person or organisation.

Evaluations sometimes uncover evidence of wrongdoing. Bad professional practices are never reported in a traceable way. However, the evaluation team member who encounters illegal or criminal acts deals with them as any other citizen should do. In the later case, the issue should be discussed with the evaluation manager.

Evaluation methodological approach > EN: Methodological bases and approach > Evaluation methods > Quality assurance

Quality assurance

.

This section is structured as follows:

.

KEY ELEMENTS

.

What is this?

Establishing quality check-points at the main phases of the process, defining the responsibilities of quality assurance, and establishing the rules of the game in case of quality defects.

Quality assurance is organised on a step-by-step basis: successive aspects of quality are secured at each step, thus creating a sound base for the subsequent step.

What is the purpose?

  • To enhance the conclusions of the evaluation in users' eyes by showing that this is impartial and rigorous work that meets professional standards.
  • To gradually construct quality and thus avoid discovering a quality defect in the final report stage.
  • To clarify relations between the evaluation manager, the reference group and the external evaluation team.

Rules of the game

The rules of quality assurance are specified in the terms of reference and pointed out when the external evaluation team is engaged.

These rules concern:

  • Approval of documents
  • Quality criteria
  • Dissemination of the quality assessment of the final report

The quality assurance process benefits from the contribution of all actors whilst limiting the potential conflicts that might arise between them, e.g.

  • The actors who hold responsibilities in the evaluated intervention (the evaluees) are in a good position for assessing the relevance of data collected and the fairness of interpretations. Their assessment may, however, be distorted by a confirmation bias. They should therefore be given only an advisory role, e.g. through their participation in the reference group.
  • The evaluation team leader has a major role in assuring quality, especially in designing an adequate method, and in securing the accuracy and fairness of the report. He/she may, however, overweight the views of powerful stakeholders, especially those who are likely to commission other evaluations in the future. The evaluation manager's quality assessment helps to prevent such a risk.

Key steps

Inception report

In the inception report stage, quality assessment is used to ensure that the evaluation team has clearly understood the scope and aims of the evaluation, as well as the logic of the evaluated intervention and the questions to answer.

First phase report (desk)

In the first phase report stage (desk), quality assessment is used to check the quality of the documentary analysis as well as the appropriateness of the proposed method for the field phase, including the indicators, data sources and analytical approach.

Final report

In the final report stage, quality assessment checks whether the evaluation provides valid and impartial answers to the questions asked, and whether the form of the report is suited to the targeted users' needs.

At this stage the quality assessment concerns first the draft version of the report and then the final version.

Key players

It is expected that all key players involved in the quality assurance process have a sufficient evaluation capacity, except reference group members. If a key player is not qualified enough, he/she ensures that appropriate support is provided.

Evaluation team leader

The evaluation team leader is primarily concerned with preventing major risks that threaten quality, e.g. overlooking a major question, not consulting an important stakeholder, elaborating upon unreliable data, judging in a partial way.

He/she also ensures that each report is submitted to a detailed quality check before it is released to the evaluation manager.

Quality assessor(s)

The company or consortium in charge of the evaluation contract appoints one or more quality assessors who carefully check each report for quality before it is released to the evaluation manager. The quality assessor should be experienced in evaluation and should not belong to the evaluation team.

Evaluation manager

The evaluation manager has the ultimate responsibility for assessing whether or not the methodological quality of a report is sufficient for allowing the next step to be taken.

In this process he/she takes advice from the reference group members, and relies upon technical support as far as necessary.

Reference group

Reference group members receive all draft reports for comments. They may comment on the factual bases substantiating the evaluation, which is a contribution to quality assurance. When their comments apply to the content of conclusions, they are passed on to the evaluation team which freely decides on whether or not to take them into account.

.

APPROVAL OF DELIVERIES

.

What is this?

Formal recognition by the evaluation manager that a delivered document has an adequate content and meets the applicable quality criteria in methodological terms.

What are the approval steps?

Generally, a document is approved in the following way:

  • The manager checks that the document has the required form and content, and has no major quality defect. If it does, he/she immediately requests a new version.
  • The manager asks for the opinion of the individuals or services that have to be consulted, either at a meeting (reference group) or by email. He/she gives a deadline after which there is no guarantee that comments will be taken on board.
  • The manager lists his/her own requirements regarding quality improvements, attaches other requests received, distinguishes requests applying to the methodological quality from requests applying to the substance of the text, and sends all the comments to the external evaluation team.
  • The evaluation team takes into account all the requests for quality improvements, either by amending its document accordingly or by mentioning in an annex how the request will be treated at a later stage, or else by explaining in an annex why the request has not been treated.
  • A new version of the document is soon presented to the manager who accepts or refuses it.
  • When approval is required, it is also a prerequisite for moving on to the next stage.

Which documents are approved and by whom?

The following documents require approval:

  • Inception report
  • First phase report (desk)
  • Pilot mission report (when the evaluation includes missions in several countries)
  • Draft final report
  • Final report

Approval is generally performed by the evaluation manager and may be confirmed by his/her superior.

The reference group members validate the set of evaluation questions, i.e. group members formally accept, or at least do not reject, the set of evaluation questions. If necessary, divergences are resolved by the evaluation manager. In that case dissenting views are noted in subsequent reports. 

Rules of approval and independence

The approval process does not contradict the external evaluation team's independence. In this respect the requests for amendments made during the approval exercise need to be differentiated:

  • Comments concerning methodological quality have to be taken into account. If the evaluation team considers that it cannot take a demand into consideration, this should be fully justified. An assessment of the methodological quality is attached to the final report.
  • Comments concerning the substance of the document, statements and conclusions are freely taken into account or rejected by the evaluation team. The team must nevertheless, in a note or in the annex, mention the requests not taken into account and comment upon its decisions in this respect.

.

QUALITY ASSURANCE SUPPORT

.

What is this?

Assessing the quality of an evaluation is a multifaceted task. The commissioning body and the stakeholders are in the best position to assess whether their needs have been treated impartially. The experts and those who implement the intervention are in a good position for assessing the relevance of data collected and the fairness of interpretations. It is the evaluation manager's role to assure that the evaluation has sound methodological and factual bases. The manager receives a quality assurance support if he/she lacks training or experience.

Who can play this role?

Within the EC headquarter Evaluation Unit, managers have the required training and skills to assess the methodological quality of reports. All evaluations are monitored by two members of the Unit, both of whom participate in quality assessment.

In the case of devolved evaluations, the manager uses the quality criteria proposed by this site as well as the indications given to fill out the quality assessment grid, if necessary with the help of the Evaluation Unit.

The Evaluation Unit can perform cross-cutting quality assessments autonomously.

Evaluation methodological approach > EN: Methodological bases and approach > Evaluation tools
Evaluation methodological approach > EN: Methodological bases and approach > Evaluation tools > Toolbox

Toolbox

.

This section is structured as follows:

.

What type of tools?

.

The evaluation's four tasks

Evaluations are usually divided into four tasks which are not strictly sequential: the evaluation's organisation (based on the evaluation questions wording leading to the overall assessment), the collection of quantitative and qualitative information, the analysis of the information collected, and the evaluation questions assessment leading to the formulation of conclusions and recommendations.
Objectives of these four tasks:

  • The evaluation's organisation contributes to the selection (or definition) of evaluation questions, judgement criteria, and indicators for these criteria, so as to determine the evaluation's methodology.
  • The information collection gathers all the primary and secondary data (quantitative and qualitative) available for answers to the evaluation questions.
  • The analysis of the information is used to assess the assistance policy contribution to the observed evolutions of indicators.
  • The assessment of each evaluation question leads to the formulation of an answer and conclusions for the evaluation.

The elaboration of an evaluation methodology based on the evaluation questions (which are designed for the development of an overall assessment) is crucial in the selection of tools used for each of the evaluation's four stages.

Importance of the documentary stage

Although no tool presented in this methodology is specifically dedicated to this purpose, the collection of information from the European Commission services and on-site (particularly the information collection using CRIS database) is a component of the evaluation process.
The following indicative list sets out the main documentary sources (secondary data) to be consulted for a country evaluation:

  • Overall agreements, bilateral/multilateral and sector-based/thematic agreements (trade, investment, science and technology, etc.), agreements of co-operation, partnership, association, and conclusions of bilateral and multilateral conferences
  • Country/Regional Strategy Papers (CSPs and RSPs) and National/Regional Indicative Papers (NIPs and RIPs)
  • The available annual reports and mid-term reviews
  • The European Court of Auditors' reports
  • Governmental official papers (such as the Poverty Reduction Strategy Papers (PRSP) and available sector-based strategies) and papers produced by multilateral and bilateral co-operation agencies (strategy papers, project papers, etc.)
  • The available thematic or sector-based project and programme evaluations
  • Reports of monitoring (ROM) on projects and programmes in progress
  • The European Commission's general regulations and all regulation documentation and political agreements covering the evaluation's period

Presentation of the tools

Thirteen tools have been developed. They are usually familiar to the evaluators. Their specificities are described here.

Objectives diagram and impact diagram

The objectives diagram displays the classification of the objectives to be achieved for the strategy implementation, from the European Union's global objective to activities carried out for operational programmes. The impact diagram displays the classification of activities, outcomes and expected impacts. The expected impacts are the objectives in terms of results.
The tool reconstitutes the intervention rationale and the expected impacts; as such, it plays a crucial role in the organisation stage of the evaluation's complex interventions through the wording of evaluation questions.

Problem diagram

Projects and programmes in development assistance aim at satisfying priority needs through the resolution of a range of issues. It is theoretically possible to construct a diagram taking the shape of a tree, with the trunk (the core problem), roots (the causes) and branches (the consequences and impacts).
The problem diagram, associated with the impact diagram, validates the relevance of a project, a programme or a strategy by relating expected impacts to the problems they should be contributing to solve.

Decision diagram

The decision diagram displays the process during which the strategic objectives and the overall co-operation policies with developing countries, which are defined by the European Union's assistance agreements, are converted into short-term and medium-term bilateral co-operation decisions.
Complementing the objectives diagram, the decision diagram facilitates the analysis of the strategy in terms of internal coherence (logical succession of the choices) and external relevance (contextual elements and position of stakeholders).

Interview

The interview collects information from stakeholders and beneficiaries throughout the evaluation stages: facts and verification of facts, opinions and points of view, stakeholders analyses and suggestions.

Focus group

The focus group is a means of discussing information, opinions and judgements already collected. The tool explains why opinions have been expressed (and the analyses supporting them) and checks their consistency. Focus groups are frequently used to collect the beneficiaries opinions concerning their participation in a programme and what they had drawn from it (positive and negative aspects). They are an alternative to interviews. Whatever their usage, the focus group's specificity is a collection of opinions which have been moderated by an in-depth discussion rather than a collection of spontaneous opinions.

Survey

The survey collects comparable answers from a sample of the population. When the sample is representative, the survey displays statistical measures which can be useful for quantified indicators.

Case study

Case studies are the preferred evaluation tool when "how" and "why" questions are being posed, because they allow a detailed examination of the actual elements in line with the evaluation goals. In contexts allowing or requiring it, the case(s) can be selected to yield general conclusions for the overall evaluation.

Expert panel

The expert panel is a group of independent specialists, recognised in at least one of the fields addressed by the programme under evaluation. The panel yields a collective assessment which is nuanced, argued and supported by the knowledge and experience of the experts.

SWOT

SWOT analysis combines the study of the strengths and weaknesses of an organisation, a geographical area, or a sector, with the study of the opportunities and threats to their environment. Frequently used in ex ante evaluations, it can also be used in ex post evaluations to assess the orientations taken.

Context indicators

The tool ranks a country through the comparison of its context indicators with that of other countries. A context indicator is a datum which produces simple and reliable information describing a variable relative to the context. The tool evaluates development dynamics through the comparison of the level and evolution of a country's main indicators with that of other countries with similar contexts.

Multicriteria analysis

In ex ante situation, multicriteria analysis is a decision-making assistance tool. In ex post evaluations, it usefully contributes to the formulation of a judgement based on a range of heterogeneous criteria.

Cost-effectiveness analysis

The tool identifies the economically most efficient way to fulfil an objective. It compares the efficiency of projects or programmes with comparable impacts. It usefully contributes to the formulation or validation of a judgement on the selection of the most efficient projects and programmes.

Cultural and social analysis

In country evaluations, the cultural and social analysis identifies the constitutive components of social, ethnic, religious and interest groups. It also highlights all the values shared in a society as well as its internal divisions.

Absence of statistical tools

In the context of assistance to developing countries, statistical data are often difficult to collect and their relevance is limited by the delays in their publication and a weak reliability. The available data are general and descriptive, and only allow for straightforward analysis. Thus, it is common practice to develop tools based on easily available data.

Rules for the methodology's application

Generally speaking, the feasibility of field work or the limitations of such a task should be checked. Indeed, specific contexts within the country where the study is to be carried out, such as logistical and implementation costs constraints, can constrain the choice of tools.
Prior to the presentation of the selection criteria for the most appropriate tools for the methodology to be applied, rules for the methodology's application should be recalled:

  • No tool can answer a question or fulfil one of the four stages of the evaluation by itself, which means that evaluations need a combination of tools.
  • Each of the tools is adapted to a specific stage, sometimes several.
  • Several tools are used concurrently in the same stage with different approaches in order to facilitate the triangulation of information.
  • The selection of evaluation tools depends on the tasks to be achieved and the context: the degree of difficulty of the intervention, quality of the available expertise, nature of the information to be collected, multiplicity of the interlocutors, etc.
  • The tool selection must be guided, except for specific reasons, by the homogeneity of the detail and the degree of precision of the information required for the analysis.

In short, the evaluation team should use several tools and choose the most efficient combination of tools for the specific context of the evaluation.

.

When to use which tools?

.

The tools and the evaluation's four functions

A first selection can be made when the tools are classified within the four tasks of the evaluation.
The table below sets out the tasks for which each tool is normally used and other tasks where it could be usefully incorporated. Most tools have a main function and one or more secondary functions. This list is indicative only.

1420534573_tools1.png

Organisation tools

The organisation of the evaluation can usefully be supported by a series of tools called organisation tools. In homogeneous project or programme evaluations, the normal organisation tool is the logical framework which describes the objectives (overall and specific) of the intervention, the issues to which the evaluation answer and the expected outcomes. A problem diagram can usefully complement the logical framework. Evaluations where the scope includes a range of heterogeneous objectives and activities (such as geographic evaluations), theoretically require three tools, the objectives diagram and the impact diagram being the core organisation tools. In complex programmes or strategies, the problem diagram brings precision on the relevance of objectives, identifies the goals and issues of the interventions, as well as the problems neglected by the evaluation's objectives. The decision diagram complements the objectives diagram with information about the reasons for the programme's orientations, and especially the basis for the rejection and negotiation of options. These tools are also useful for the analysis and judgement stages (polyvalent nature of the tools).

Collection, analysis and judgement tools

During these three evaluation stages, a large range of tools is available, complementary to each other and/or polyvalent. As a consequence, numerous factors are taken into account to make an efficient choice. Although no rule securing an optimal choice exists, a logical process can facilitate the development of a homogeneous methodology which will provide well-grounded answers to the evaluation questions.

Polyvalent nature of the tools

The resources allocated for an evaluation are not sufficient for the implementation of all the tools mentioned previously. Choices must be made with respect to the evaluation's priorities and the maximisation of the use of resources. The wider the scope and questioning, the greater the risk of dispersion, which means that the evaluation team must ensure that their observations and analyses provide answers to the most essential issues of the evaluation. The evaluation team should remember that several tools are usually relevant for the same evaluation stage and can be used to confirm the information, and that the tools are often polyvalent and can answer several questions at the same time.

Selection criteria for the tools

Besides the tools specific functions and their ability to be implemented in one of the four stages of the evaluation, other selection criteria should be examined.
The selection criteria set out below should guide the evaluator through a series of choices needed for the development of the methodology.

  • Knowledge of the techniques. Although the majority of the tools presented are easily implemented, some may require prior experience. This is particularly the case for tools necessitating specific group moderating skills, and complex tools such as multicriteria analysis, cost-effectiveness analysis or surveys.
  • Need for specific data. The implementation of some tools requires the collection of specific data without which their conclusions would be ill-founded. For example, the implementation of cost-effectiveness analysis is impossible without indicators of effectiveness measuring comparable projects. Their availability and reliability must therefore be checked before the cost-effectiveness analysis is used.
  • The prerequisites for the tool's usage. This issue is particularly important for tools whose implementation takes place during the field stage in the partner country. As the implementation of such tools often generates high costs, their relevance within the overall methodology, cost-effectiveness and efficiency (particularly for the data collection) must be secured.
  • Implementation time. Some tools, such as the surveys and some types of focus group investigations and expert panels, need a preparation stage before their implementation on-site
  • The availability of qualified local experts, capable of conducting specific tools in the local language. This issue is particularly crucial for tools requiring group moderating skills (focus groups, etc.) for which available and skillful experts are sometimes hard to find.

The table below grades each tool using the five criteria and indicates those demanding particular attention. Each criteria is awarded a grade from 1 to 3. Grade 1 means that the criteria does not constitute a particular problem for the tool; grade 3 means that it is recommended that the feasibility of the tool's implementation should be checked with regards to this criteria.

The tools' specific requirements

Tools Knowledge
of the
techniques
Specific
information
Costs Delays Qualified
local
experts
Objectives
diagram and
impact
diagram
2 1 1 1 1
Problem
diagram
2 1 1 1 1
Decision
diagram
2 1 1 1 1
Interview 1 1 1 1 1
Focus group 2 1 1 2 3
Survey 3 1 3 3 2
Case study 1 2 2 3 1
SWOT 1 3 2 2 1
Context
indicators
2 1 1 1 1
Expert panel 2 1 1 2 2
Multicriteria
analysis
2 3 2 2 2
Cost-
effectiveness
analysis
3 3 3 2 1
Strong requirements (3) , medium (2), poor (1)

A grade 3 does not mean that the tool should not be used. Indeed, the priority should be given to the tools providing the best answers to the evaluation questions, and then, the evaluator can check the possibility of using them in the context and with the available resources of the evaluation.

Development of a homogeneous methodology

  • The methodology's development must target the organisation of a homogeneous and efficient technique. It is useless to schedule the implementation of sophisticated tools for one stage of the evaluation if the other stages are not addressed with the similar precision or logical process, or if the other tools cannot provide the required information or use the information collected.
  • The collection stage must be carefully managed because if it leads to incomplete findings, the organisation of a new information collection is challenging during the field stage. The same attention should be given to analysis and judgement tools to be implemented in the country under evaluation.
  • To do so, it is useful to assess the risk of an unsuccessful implementation for each tool, in order to plan alternative solutions if needed be and limit the impact of such a failure on the evaluation as a whole.

In essence, the methodology should be constructed with a range of available tools and take into account their advantages and limitations, the conditions for their implementation in the context of development assistance evaluations, the prerequisites for their implementation and the limits of their findings due to the context.

Analysis of the tools

Objectives diagram and impact diagram

Main function Organisation of the evaluation
Advantages Reconstructs the intervention rationale and the expected impacts.
Contributes to the wording of evaluation questions.
Plays a crucial role in the evaluation's organisation stage for complex programmes (such as country evaluation strategies).
Potential constraints Limited access to documentation.
Uncertain information.
Expected outcomes Wording of evaluation questions.
Reconstruction of the programme's intervention rationale.
Prerequisites Compulsory documentary research.
Expertise of the evaluator.

Problem diagram

Main function Organisation of the evaluation
Advantages Validates the relevance of a project, programme or strategy by relating the expected impacts with the problems they contribute to solve.
Potential constraints Limited access to documentation.
Uncertain information.
Expected outcomes Validation of the programme's intervention rationale.
Prerequisites Compulsory documentary research.
Complementary to the objectives diagram.
Expertise of the evaluator.

Decison diagram

Main function Organisation of the evaluation
Advantages Facilitates the analysis of the strategy in terms of internal coherence ((logical succession of the choices) and external relevance (contextual elements and position of the stakeholders).
Potential constraints Limited access to documentation.
Uncertain information.
Expected outcomes Explanation of the decision process and determination of strategic and/or political objectives. Analysis of the strategy's coherence and relevance.
Prerequisites Compulsory documentary research.
Complementary to the objectives and problem diagrams.
Expertise of the evaluator.

Interview

Main function Observation / Collection                     
Advantages Collects information from actors and beneficiaries throughout the evaluation stages.                                               
Potential constraints Limited availability and collaboration of the stakeholders.
Inappropriate selection of interlocutors (lack of representativity of all the stakeholders).
Subjectivity and spontaneity of the opinions expressed.
Poor degree of comparison between answers.
Expected outcomes Facts and verification of facts, opinions and points of view, analyses of the actors and suggestions.
Prerequisites Availability of the stakeholders.
Time planning required.

Focus group

Main function Collection / Observation
Advantages Highlights the reasons for the opinions expressed (including the analyses supporting them) and checks their consistency.
Potential constraints Limited availability and collaboration of the stakeholders.
Risk of inappropriate or arbitrary selection of interlocutors (lack of representativity of all the stakeholders).
Subjectivity and spontaneity of the opinions expressed.
Expected outcomes Opinions ponderated after in-depth discussion instead of spontaneous opinions.
Prerequisites Information collection prior to the focus group setting (the tools aims at debating information, opinions and judgement which have already been collected).
Local assistance to facilitate the organisation of meetings.

Survey

Main function Collection / Observation
Advantages Collects comparable answers from a sample of the population.
Potential constraints Limited availability and collaboration of the stakeholders.
Difficult analytical process of the information requiring a complex organisation.
Detailed knowledge of the techniques.
Expected outcomes When the sample is representative, it can yield significant statistical information which can be useful for quantitative indicators.
Prerequisites Required expertise for data processing.
Time planning required.
Costs.

Case study

Main function Analysis
Advantages Answers to "how" and "why" questions through a detailed examination of actual cases.
Potential constraints Too specific situations analysed (impossible to develop a general rule).
Risk of unjustified extrapolation/generalisation.
Risk of inappropriate/arbitrary selection of case studies (with no relationship to the evaluation's objectives).
Access to information.
Expected outcomes Answers to questions requiring in-depth study (how and why?).
Where possible, case studies can be selected to provide the conclusions with a general rule applying to all the evaluation.
Prerequisites Easy access of the sites and availability of the beneficiaries.
Time.
Costs.

Expert panel

Main function Analysis
Advantages Comprises independent specialists, recognised at least in one of the fields addressed by the programme under evaluation.
Yields a collective point of view which has been debated and reached through consensus.
Potential constraints Limited availability of the experts.
Subjectivity of the judgements/analyses.
Expected outcomes Collective, nuanced and well-argued judgements which are supported by the experts' knowledge and experience.
Prerequisites Accessibility of the selected experts.
Information collection prior to the organisation of the expert panel.
Knowledge of meeting moderating techniques.

SWOT

Main function Analysis
Advantages Used frequently in ex ante evaluations; can assess the orientations carried out in ex post evaluations.
Potential constraints Excessive general and sometimes incomplete information which does not allow the evaluator to undertake an in-depth analysis.
Risk of impartiality in the analysis of the strengths and weaknesses.
Absence of control over the tool.
Requires a consensual approach of the analysis (perception of the threats/opportunities).
Expected outcomes Quick diagnostic of an organisation, a territory, a programme or a sector form the analysis of the strengths and weaknesses combined with the analysis of the environment/context in terms of opportunities and threats.
Prerequisites Information collection prior to the analysis (SWOT analysis requires very precise information, otherwise it may be limited to an abstract and intellectual exercise).
Knowledge of the tool.

Context indicators

Main function Analysis
Advantages Assess development dynamics through the comparison of the level and evolution of the main indicators with that of countries having similar contexts.
Potential constraints Inappropriate choice of context indicators.
Lack of sufficient/relevant information needed to complete the indicators.
Expected outcomes Situation of a country through a comparison of its context indicators with that of other countries.
Prerequisites Requires a good data collection, implemented on a regular basis so as to measure the context indicators evolution.
May require the implementation of surveys.

Cultural and social analysis

Main function Analysis
Advantages Identifies the components of social, ethnic, religious and interest groups, as well as all the converging and diverging points at the basis of common values within a society.
Potential constraints Limited availability of the experts.
Subjectivity of the judgements/analyses.
Level of abstraction of the analysis; distance from reality.
Expected outcomes Approximate notion of social capital in a given territory.
Prerequisites Accessibility of the selected experts.
Information collection prior to the analysis.
Knowledge of meeting moderating techniques.

Multicriteria analysis

Main function Judgement
Advantages Decision-making assistance tool which usefully contributes to formulate judgements supported by a range of heterogeneous criteria in ex post evaluations.
Potential constraints Insufficiency/lack of precision of the information.
Lack of knowledge of the techniques.
Expected outcomes Judgements on evaluation questions.
Prerequisites Required information (prior collection and analyses to be provided).
Knowledge of the techniques.
Time.
Costs.

Cost-effectiveness analysis

Main function Judgement
Advantages Compares the effectiveness of projects or programmes targeting similar impacts.
Usefully contributes to the formulation or validation of the judgement on the selection of the most effective projects or programmes.
Potential constraints Insufficiency/lack of precision of the information.
Lack of knowledge of the techniques.
Expected outcomes Determines the activity achieving an outcome at a least cost.
Judgements on the selection of projects/programmes.
Prerequisites Required information (prior collection and analyses to be provided).
Knowledge of the techniques.
Time.
Costs.

.

Which combination of tools?

.

The combination rationale

Some tools require the implementation of other tools prior to their use. Such is the case when a tool yields useful information for the implementation of another tool, or when bringing a different viewpoint to the analysis, it strengthnens or nuances the conclusions reached which another tool. Before taking the decision to implement complex tools (such as judgement tools), the evaluator should check whether preliminary information which is capable of improving their performance are available, and find the optimum tool capable of yielding such information.
The most frequent cases of combinations of tool are listed in the table below. For example, the first line shows that objectives diagrams and impact diagrams require or may require the implementation of interviews and focus groups.

Tools required by other tools


The table is indicative and other combinations can be developed in particular contexts. Usually, collection tools (interviews, focus group and surveys) are the most frequently combined with analysis and judgement tools, because the latter require specific information for their implementation. Thus, special care should be given to analysis and judgement tools, so as to ensure the homogeneity of the methodology and the maximum performance of the tools throughout the evaluation.

The tools' frequency of use

The table shows that the interview is the tool the most frequently used by other tools, including collection tools (which is not surprising). Conversely, case studies are seldom used and belong to the category of meta-tools requiring the support of all the collection tools and, if needs be, analysis tools.
Although presenting an exhaustive table of the tool's combinations is not possible, recommendations should be made to ensure the homogeneity of the tools performance all through the evaluation:

  • Some tools are frequently used by other tools. For example, the interview is the collection tool the most often used by other evaluation tools.
  • Most of the tools are multifunctional. The decision to use a specific tool should thus be based on its conditions of use and on the constraints related to the implementation of additional tools.
  • Because of the polyvalent nature of the majority of the tools, criteria for rationalisation and optimisation must support their selection. For example, evaluation questions should be classified depending on the tools selected and vice-versa.

Example of combination

Testing of two country evaluation tools - the survey and the focus group - during the evaluation in Benin.
The test checked the possibility of providing elements to answer the following evaluation question through the implementation of a combination of two tools: "To what extent are the European Commission's sanitary strategy and interventions adapted to the fundamental needs of the population, and particularly the poor?"
The objectives of the test were to collect and analyse information and points of view from beneficiaries, in order to assess the global trends of the service provided by health centres. The test combined three surveys and four focus groups.
Three surveys:

  • Two structured questionnaires, one addressed to households and the other to the users of the health centres, which resulted in a total of 660 respondents
  • One open-ended questionnaire addressed to the local authorities, health professionals and people in charge of insurance companies and local associations, which resulted in a total of 100 respondents.

Four focus group investigations:

  • Two focus groups of beneficiaries (countrywomen) in two different villages.
  • One focus group with midwives and nurses.
  • One focus group with doctors from the private and public sector.

The last 2 focus group investigations were set up to complement the outcomes of the 2 focus groups with beneficiaries and that of the surveys. The goal was to confront different perspectives on a same situation where a few dysfunctional elements could be raised.
The confrontation of the survey's outcomes with that of the focus group contradicted the hypothesis at the start of the reorganisation of the public health system implemented by the European Commission in Benin. It also asserted that the access to health care in the public sector was very limited for the poor population.

.

What specific constraints and requirements?

.

Limitations and risks depend on the constraints and requirements specific to each of the available tools, whatever their categories.

Possible constraints

Access to information and sources of information

Most of the tools require relatively straightforward access to baseline documentation (Objectives diagram and impact diagram, Problem diagram, Decision diagram, Case study, Context indicators). Some tools strongly depend on the representative nature of the interlocutors and stakeholders, on their availability and co-operation, such as the Interview, the Focus group and the Survey.

Quality of the collected information

The quality of the collected information is crucial, and can be influenced by:

  • The insufficiency and/or approximate nature of the information (SWOT, Context indicators, Cost-effectiveness analysis, Multicriteria analysis, Cultural and social analysis)
  • Bias in the opinions expressed (Survey, Focus group)
  • Partiality in the judgements or bias in the analyses (Expert panel, Cultural and social analysis)
  • Partiality in the analysis of the strengths/weaknesses (SWOT)
  • An ill-founded generalisation (Case study)
  • Answers difficult to compare (Interview)
  • An inappropriate selection of context indicators (context indicators)
  • An excessive level of abstraction in the analysis and too great a distance from reality (Cultural and social analysis)

Knowledge of specific techniques

A good knowledge of the tool is needed for all the tools, and especially for: the Survey, the Cultural and social analysis, SWOT, Cost-effectiveness analysis, Multicriteria analysis.
The availability and experience of acknowledged experts must be confirmed, particularly for: the Case study, the Cultural and social analysis, the Expert panel, the Focus group.
The processing of answers can quickly become unmanageable because of a lack of sufficient competence (Survey).
A lack of experience can lead to inappropriate choices in study cases, with no direct relationship with the evaluation's objectives (Case study).
The attempt at consensus seeking on the perception of threats and opportunities requires a specific knowledge (SWOT).

Prerequisites

Human resources

The availability of acknowledged experts must be guaranteed whatsoever. It is a key condition for: the Cultural and social analysis, the Expert panel, SWOT, Cost-effectiveness analysis, Multicriteria analysis and Focus group.
The evaluator's expertise must be assessed for the organisation tools: Objectives diagram and impact diagram, Problem diagram, and Decision diagram.
Expertise is required for the data processing analysis developed in Surveys.
Experience in meeting management and moderation is crucial for: the Cultural and social analysis, the Expert panel, the Focus group.

Costs

Implementation costs should be examined particularly for the following tools: Cost-effectiveness analysis, Multicriteria analysis, Survey, Case study.

Time span

An appropriate time schedule is important for all tools, and particularly for the implementation of: Cost-effectiveness analysis, Multicriteria analysis, Survey, Case study and Interview.

Impact of preparatory stages

When a tool requires a preparatory information collection, its time span, cost and human resources should not be under-estimated. If needs be, access to sites and beneficiaries availability must be checked (Case study).

.

Check list for the tool's implementation

.

Check list for the evaluation team

  • Does the selection of evaluation questions allow the use of tools to efficiently provide answers?
  • Will the selection of tool help in formulating an overall assessment?
  • Does the implementation of the tools selected provide relevant answers to the evaluation's objectives?
  • Can each tool be adapted to the constraints and opportunities related to the specific conditions of the evaluation?
  • Does the organisation of each of the tools take into account all the prerequisites for their implementation?
  • Were the prerequisites for each tool (such as the competences, number of working days, expenses) precisely assessed? Are they relevant with regard to the expected outcomes? To the outcomes achieved?

Check list for contract managers

  • Are the answers provided for each of the evaluation questions supported by the implementation of an effective combination of tools?
  • Does the use of appropriate tools support the overall assessment?
  • Is the choice of each tool and their combination clearly and convincingly explained?
  • Are the available resources for the tool's implementation (experts, budget, time span) being effectively used?

Toolbox and directions for use

The aim of the evaluation is to produce operational recommendations supported by solid conclusions which are based on clear judgement criteria, solid and concrete information and rational argumentation.
Conclusions and recommendations are a means of discusssing about the programmes or policies with their authors and the operators involved. They must result from a rigorous process (the expert's own point of view is insufficient).
The development of a rigorous ad hoc methodology and the use of appropriate tools are the crucial components of this process. This website presents a series of evaluation tools and directions for use of the toolbox which:

  • Integrates the tools within the evaluation's methodology
  • Presents selection criteria for the tools
  • Gives guidance on an appropriate combination of tools
Evaluation methodological approach > EN: Methodological bases and approach > Evaluation tools > Problem diagram

Problem diagram

SUMMARY

.

Why are this tools used in evaluation?

.

The analysis of problems is a means to test the validity of the objectives of a project, a programme or a strategy. As a programme aims at solving a range of problems, the evaluation should be concerned with the validity of its analysis. The evaluators should therefore check:

  • The validity of the procedure: how have the problems been identified and classified?
  • The apparent coherence of the problems' linkage: are their causal links relevant?
Figure 1: Analysis of the problems in the project cycle and in the strategy policy co-ordination cycle

.

Programmes and strategies depend on the analysis of the situation. This analysis presents the primary data relative to the problems addressed by the programmes and strategies, and the information about the context (economic, social, cultural, institutional, environmental) in which they will be implemented.

This crucial analysis identifies:

  • The problems to be addressed by the activities in order of priority
  • The positive elements on which activities can rely
  • The actors to be involved in the activities
  • The structural and dynamic constraints which may hinder the success of the activities

Therefore, the evaluation team can be asked to assess the quality of the analysis, and the conformity between the analysis of the situation and the adopted strategy (or programme). The problem diagram, as an ex post construction, can be one of the tools used to check the coherence and the relevance of the analysis in respect of the main contextual problems.

.

What are the possible uses of these diagrams?

.

When it is impossible to directly establish objectives diagrams, problem diagrams play an essential role in the organisation stage of the evaluation. Problem diagrams present a summarised vision of the situation to be improved, partially at least, by the strategy. The classified objectives included in the strategy should be deduced from the problems diagrams.

.

Figure 2: The objectives deduced from the problems

.

The reconstruction of the problem diagram includes a step which differentiates between context problems and intervention problems. As a consequence, the diagram resulting from this selection should be completely convertible into a logically reconstructed objectives diagram, i.e. the higher-order overall objective corresponds to the core problem, and each row of subordinated objectives to its equivalent row of problems.

.

How is the problem diagram constructed?

.

Stage 1: How to identify the problems?

Record the references to teh problems found in the documentation: quotations from the evaluation's baseline documents are used to identify problems which may not be systematically depicted as such in the texts. Sometimes, problems are identified as assistance objectives or impacts targeted by these objectives. Problems are thus expressed as:

  • problems (presented as such)
  • objectives, whose goal is to resolve explicitly or implicitly a problem
  • expected impacts of the CSP assistance

The problems directly targeted by the intervention may not be explicitly identified. Main problems and context problems can be intermingled, which complicates the construction of the diagram. The evaluator will only be able to differentiate the two types of problems after the completion of the diagram. 

The crucial stage of the process is the identification of the core problem among the variety of the selected problems. Three situations can be encountered:

  • The core problem is mentioned in the documentation
  • The core problem is not clearly mentioned, whereas the overall objective is. In this case, the evaluator can deduce the core problem from the higher-order objective.
  • Neither the core problem, nor the overall objective are explicitly mentioned

In the last situation, the evaluator plays the role of the planning manager, and decides which of the problems will be the core problem. He/She may:

  • Take the decision, on the basis of his/her rationale
  • Benefit from the assistance of specialists during interviews or working sessions

The selection of the core problem should be conducted concurrently with the classification of the problems into levels. Indeed, the selection of the core problem should be supported and justified by the coherence of the whole diagram.

Stage 2: How to construct a problem diagram?

Figure 3: Stages involved in the drafting of temporary diagram 

Usage highlights the fact that these two types of problems/causes are the easiest to identify, whereas intermediary causes are the hardest to determine and classify. Thus, it is recommended that the development of the diagram starts with its extremities at the same time.

Figure 4 : Problem diagram in the transport sector in Tanzania

Test the temporary diagram

Where possible, the authors of the documentation referred to above should test the diagram in order to validate the classification of the problems by rows and links. The aim is to check that the evaluator's interpretation reflects the authors' intention correctly. If the authors are not available or, in order to complement their original contribution, the evaluator should consider asking for the participation of other actors responsible for the drafting process. 

Contacting the authors is usually possible when the documentation is recent and the authors are still in their position or contactable in one of the services. This task is challenging when the documentation is old and its authors are not easily identifiable or contactable.

Develop the final version of the problem diagram

The final version takes into account the opinions collected during the testing of the temporary diagram. It is an accurate account of the initial intentions logically reconstructed of the European Commission, taken from official documentation.

.

What are the preconditions for its use?

.

The need for human resources varies with the tasks to be achieved and the situation encountered: reconstruction of the problem diagram with the analysis of the situation displayed in the documentation; retrospective reconstruction of the problem diagram as it should have been established to support the relevance of the strategy or programme under assessment. 

Most of the graphic problems can be solved with software like MS PowerPoint. 

The evaluator can also benefit from specific software designed to assist decision-making, such as MS Visio, TeamUp-PCM for Windows, and Palisade PrecisionTree.

Figure 5 : the preconditions for its use
The time span Collect: 5 to 10 working days
Data analysis: 5 to 10 working days
Test: 0,5 to 5 working days
Human resources Multidisciplinary team of experienced evaluators, whose specialities should cover the thematic evaluation scope. Knowledge of the European Commission's strategies and programmes development procedures.
Knowledge of computer tools. Experience in the fields covered by the strategies and programmes
Specific knowledge of the country, sector and theme under consideration
Financial resources A budget less than €5,000 should be fixed for the problem diagram, objective diagram and decision diagram.

.

What are the advantages and limitations of the tool?

.

Figure 6 : The advantages and limitations of the tool
Advantages Presentation of the problems 
The diagram presents the various problems and their relationship with the core problem through a system of rows. It displays the causal logical links between them or, conversely, the poor logic of these links. 

Main problems and contextual problems 
In the analysis of the situation, the diagram distinguishes the problems relating to the activity's context from the problems to be solved by the strategy and the planning. As a consequence, its construction requires the highlighting of priorities of the development assistance, and explains why certain problems are considered as important features of the strategy while others are not. 

Definition of the objectives The problem diagram enables the evaluator to: 

  • Reveal the implicit objectives of the strategy (or the programme)
  • Check the validity of the objectives presented in the strategy and programming documentation

It contributes to the organisation of the evaluation around a crucial question, which should be systematically answered: To what extent have the objectives been achieved?

Limitations Quality of the analysis
The methodology for the analysis does not guarantee high quality of the data, for neither the methodology, nor the sources of information are usually mentioned in the strategic, political and programming papers. The evaluator must therefore systematically enlarge the assessment to include the sources of the analysis displayed in these documents. The methodology used, the nature of the main sources of information and the identity of the authors should also be noted. 

Selection of the main problem
The selection of the main problem, which is crucial for the construction of the diagram, is particularly challenging when the objectives of the activities are general and the whole range of the country's problems (or the region's) are to be considered. The documentation may show two main problems which lack links between them, or may appear insufficient for the determination of a single core problem. 

Knowledge of the situation in the country or the region
The evaluation team may lack a sufficient understanding of the country or the region under consideration to assess the relevance of the analysis undertaken, the determination and the logical setting of the main problem.

Evaluation methodological approach > EN: Methodological bases and approach > Evaluation tools > Problem diagram > Detailed presentation

Detailed presentation

.

This section is structured as follows:

.

WHAT IS A PROBLEM DIAGRAM? 

.

WHAT DOES A PROBLEM DIAGRAM REPRESENT?

What is the definition of a problem?

In development assistance, projects and programmes have two targets: to satisfy priority needs or achieve a specific goal, and in this view to solve a range of hindering problems. Generally, the success of a policy or strategy depends on the resolution of difficult issues for the partners and potential beneficiaries.

Problems seldom appear separately. Cause-and-effect relationships often link them and constitute a system, which can be presented as a tree-like diagram.

The importance of problems varies. It is thus theoretically possible to identify a core (or central) problem and derive from it a range of causes and effects.

Chart of the problem diagram

When a core problem is identified, the diagram takes the shape of a complete tree, provided with a trunk (the core problem), roots (the causes) and branches (the consequences and impacts).

Standard charts illustrate the core problem in the centre of the diagram, causes at the bottom, and consequences at the top. The diagram is read from the bottom upward.

Standard chart: the complete problem tree

 

 

A strategy, programme or project is all the more effective when it studies the fundamental causes of the problem(s) to be solved. Thus, once the core problem is identified, the diagram only represents the causes and is displayed as an inverted tree.

The diagram below illustrates a simple case, where three levels of problems are identified: fundamental causes, intermediary causes and the core problem.

Simplified problem diagram

 

In the context of the European Commission's development assistance, the problem diagram illustrates the core problem with its immediate and in-depth causes. It may also depict the core problem's main consequences which are addressed by the assistance strategy (defined in the strategy or policy papers and the corresponding programmes).

WHERE DOES THE PROBLEM DIAGRAM COME FROM?

The problem diagram comes from objective-based management whose use expanded during the 1960s in the American economic sphere and in Western economies.

Also called cause-and-effect diagram, it has been developed for a variety of fields, such as medical diagnosis, prevention and research on the causes of accidents, and total quality management.

It is used as a country assistance decision tool. It is linked to the objectives tree in the context of Goal Oriented Project Planning (GOPP, originally called ZOPP in German for Zielorientierte Projektplanung) which derives from the logical framework developed by GTZ at the start of the 1980's.

Used as a problem tree, it is:

  • A useful tool for the elaboration and wording of projects
  • Part of a planning methodology contributing to the development
    of an objectives tree and a logical framework
  • A participative decision tool which embraces the main stakeholders of the project and
    whose determination is the first stage of the project cycle sequence

Following GTZ's initiative, various public donors and operators (agencies and NGOs) have used the problem tree as a component of project planning and management. However, its application as a participatory and ownership tool does not always play the crucial role defined by the ZOPP designers.

WHAT SHAPE CAN IT TAKE?

Tree shape or complex diagram

Complex and simplified problem trees are considered to be standard charts. They are subject to the following basic rules:

  • Determination of a single core problem
  • Subdivision of the problem of a given row into 2 or more problems of the row below
  • Connection of a defined problem with a single problem of the row above
  • No horizontal, or feedback links (i.e. effects becoming causes and vice versa)

This simplification of reality is understandable in the context of the objectives-oriented participative planning process. It becomes a handicap in an evaluation because it avoids the complexity of reality and the rational limitations inherent in strategy and planning papers. Thus, the standard chart has been abandoned and the term "tree" replaced by "diagram".

Complex problem diagram

 

This diagram represents a complex problem system.

  • One of the intermediate causes in row (2) is directly linked to the core problem and not to another intermediate cause (1).
  • Two intermediate causes in row (3) are interrelated.
  • One of these causes is linked to two causes of the row above (intermediate 2).

Examples of more complex problem systems can be found, having two core problems and subdivisions of intermediate causes. In any case, such problem diagrams do not illustrate feedback links.

Vertical or horizontal diagram

Diagrams can either be vertically or horizontally oriented.

In their vertical shape, they read from top to bottom, or from bottom to top. Usually, the core problem is located in the upper part of the diagram.

In their horizontal shape, diagrams read from right to left, or left to right.

Other shapes

Other shapes have been developed, such as the fishbone diagram (also called Ishikawa diagram). Dedicated to the study of the cause resulting from a selected impact, it is also used in quality control process, medical diagnoses and accident prevention.

WHAT IS A LOGICAL LINK?

Definition

Problems and causes illustrated in the diagram are related to each other with horizontal or vertical links. These links are called "logical" when expressing an inference relationship (induction or deduction) which has been validated by experience. They highlight the fact that:

  • The introduction of a problem or cause at a certain row implies that of more deep-rooted problems or causes
  • A problem (cause) of a certain row can be deduced from a problem (cause) of an upper row
  • Two problems (causes) of the same row share a synergic relationship

The logic of the links between problems (causes) is essentially based on experience and development theories (originating from this experience). Experiments and theories usually provide good indicators for the determination of subordinate problems (or causes) in which a particular problem (or cause) in a certain row has its origin.

However, in terms of development, experiments do not always result in the same conclusions and, as a consequence, should not be considered universal. This can be explained by the dependency of the problems on various factors, which are often not well-known. Even when experts agree on the relevance of a core problem, they may disagree on the definition of intermediate and in-depth causes. The content of core problems, such as poverty alleviation, is continuously debated and evolving.

Verification of the diagram's logic

The logical links between problems (i.e. the problem diagram's logic) can be simultaneously checked with:

  • Recognised experts who have worked in different countries and situations
  • Designers and implementing managers of programmes included in the evaluation scope

These two groups of actors can validate the diagram's logic in whole or in part. Conversely, a link which is contradicted by the experience of the experts and operators can be classed as illogical. The borderline between the logical and illogical nature of a situation is determined when there is a doubt, and contradictory or ambiguous experiments.

.

WHY AND WHEN?

.

WHEN SHOULD A PROBLEM DIAGRAM BE USED?

Evaluation of standard projects and programmes

As a programme aims at solving a range of problems affecting potential beneficiaries (individuals or groups of beneficiaries), the evaluation should be concerned with the validity of its analysis. When a logical framework (ZOPP, Project Cycle Management) supports the establishment of the programme, its drafting usually includes the analysis of the problems. The evaluators should therefore check:

  • The validity of the procedure: how have the problems been identified and classified?
  • The apparent coherence of the problems' linkage: are their causal links relevant?

The assessment of the analysis of the problem and its corresponding diagram is a means to test the validity of the objectives system and its corresponding diagram, and to draw conclusions from them.

If the programme does not have a logical framework, at least the problems to be addressed should have been analysed. In this context, it may be useful to develop a rational diagram based on a study of the definition of the programme and programming documentation.

Evaluation of complex strategies and programmes

In complex strategies and programmes, the evaluation usually deals with a range of activities (projects and programmes) which lack explicit justification within a logical framework. The evaluation team may not be provided with an explicit and logically structured presentation of the objectives targeted by the donor, nor with thorough analysis of the difficulties which are supposed to be solved as a result of adopting such objectives.

The available documentation is not always clear about how the various types of problems have been taken into account. The evaluation team must therefore reflect upon questions such as:

  • Are the problems belonging to the context?
  • Or, are they selected because of the strategy's main objectives?
  • Why have the reasons for the problem's selection not been clearly stated?

In addition, the methodology which was used to analyse the situation and select the problems should be presented, which is not always done prior to the evaluation. Answers to the following questions should be considered as part of the evaluation:

  • Does this analysis result from documentary research?
  • Or, does it result from the conclusions of workshops and seminars?
  • Has a single author drafted it?
  • Or, has it been the result of a collective work?
  • Is/are the author(s) of the analysis also responsible for the drafting of the strategic orientations and the intervention programme?

Evaluation of the analysis of the initial situation

Programmes and strategies depend on the analysis of the situation. This analysis presents the primary data relative to the problems addressed by the programmes and strategies, and the information about the context (economic, social, cultural, institutional, environmental) in which they will be implemented. This crucial analysis identifies:

  • The problems to be addressed by the activities in order of priority
  • The positive elements on which activities can rely
  • The actors to be involved in the activities
  • The structural and dynamic constraints which may hinder the success of the activities

Therefore, the evaluation team can be asked to assess the quality of the analysis, and the conformity between the analysis of the situation and the adopted strategy (or programme). The problem diagram, as an ex post construction, can be one of the tools used to check the coherence and the relevance of the analysis in respect of the main contextual problems.

The evaluation team should not underestimate the amount of work, nor the level and variety of competences required (Human resources and working arrangement) to construct the problem diagram. Indeed, the evaluation team should not simply present a diagram with the problems developed in the documentation, but also reconstruct the problem diagram which should have been prepared during the development of the objectives diagram (and therefore, during the drafting of the strategy)

PROBLEM OR OBJECTIVE: WHICH COMES FIRST?

In project cycle

In the planning process and project management field (ZOPP, PCM), specialists place the preparation of the problem diagram before the objectives diagram. The typical sequence can be pictured as followed:

Analysis of the problems in a simplified sequence of the project cycle

 

Problems analysis may be undertaken before stakeholder analysis, but should always precede the objectives analysis. Indeed, in Project Cycle Management (PCM), objectives are always deduced from the problem analysis.

In the strategy policy co-ordination cycle

If the definition of the donor's activities in bilateral co-operation does not systematically result from the analysis of the problems, it will target one or more overall objectives (strategic and philosophical) which are explicitly or implicitly mentioned in the documentation. The donor (alone or in partnership) defines the main fields of the development assistance in accordance with these objectives.

The chart below illustrates the logical position of the problem analysis within the strategy policy co-ordination cycle and, particularly, in relation with the definition of the objectives.

Analysis of the problems in a simplified sequence of the strategy policy co-ordination cycle

 

The problem analysis and the resulting diagram derive from the definition of the intervention scope, and thus from the overall objectives of the donor (the European Commission). They also form the basis for the definition of specific objectives of the development assistance, which is to be provided during the period covered by the strategy.

WHAT ARE ITS ADVANTAGES AND LIMITATIONS?

Its advantages

Presentation of the problems

The diagram presents the various problems and their relationship with the core problem through a system of rows. It displays the causal logical links between them or, conversely, the poor logic of these links.

Main problems and contextual problems

In the analysis of the situation, the diagram distinguishes the problems relating to the activity's context from the problems to be solved by the strategy and the planning. As a consequence, its construction requires the highlighting of priorities of the development assistance, and explains why certain problems are considered as important features of the strategy while others are not.

Definition of the objectives

The problem diagram enables the evaluator to:

  • Reveal the implicit objectives of the strategy (or the programme)
  • Check the validity of the objectives presented in the strategy and programming documentation

It contributes to the organisation of the evaluation around a crucial question, which should be systematically answered:

To what extent have the objectives been achieved?

Its limitations

The diagram rationally organises the main difficulties on the basis of the analysis of the situation. It may entail the following limitations.

Access to information

The lack of data or difficult access to information may lower the quality of the analysis of the situation. The evaluator should therefore reflect upon the subsequent questions:

  • Is the analysis sufficiently comprehensive and does it cover the main aspects of the situation?
  • Is it topical, i.e. based on sufficiently updated data (which documents should be referred to?)?
  • Is it dynamic, i.e. does it take into account the observable trends and deep-rooted changes in the society?
Quality of the analysis

The methodology for the analysis does not guarantee high quality of the data, for neither the methodology, nor the sources of information are usually mentioned in the strategic, political and programming papers. The evaluator must therefore systematically enlarge the assessment to include the sources of the analysis displayed in these documents. The methodology used, the nature of the main sources of information and the identity of the authors should also be noted.

Main problem / contextual problem

The determination of the priorities for each problem results in their classification into two categories: contextual problems and main problems. Two questions must be addressed:

  • Is the determination of these priorities explained?
  • Does the available data enable the evaluator to assess the relevance of such an explanation?
Selection of the main problem

The selection of the main problem, which is crucial for the construction of the diagram, is particularly challenging when the objectives of the activities are general and the whole range of the country's problems (or the region's) are to be considered. The documentation may show two main problems which lack links between them, or may appear insufficient for the determination of a single core problem.

Tree-like illustration

The standard setting of the challenges into a tree diagram illustrates a straightforward classification (which types of problem diagrams can be used?), which does not always highlight the complexity of the situation and the interactions between issues. Indeed, the construction of diagrams depends on graphical conventions, such as:

  • There should be no illustration of interactions between problems of the same row
  • There should be no illustration of feedback links (effects become causes and vice versa)
  • Several boxes can illustrate several problems' single cause

More sophisticated representations should therefore be tried.

Knowledge of the situation in the country or the region

The evaluation team may lack a sufficient understanding of the country or the region under consideration to assess the relevance of the analysis undertaken, the determination and the logical setting of the main problem.

WITH WHICH TOOLS CAN IT BE COMBINED?

Objectives diagram

The problem diagram is usually associated with the objectives analysis (and as a consequence, with the objectives diagram) and the development of the logical framework.

The problem diagram is used for the construction of the objectives diagram, often through a simple transposition.

Decision diagram

The analysis of the programme's development process (decision diagram) can be used to explain the reasons underpinning the determination of the main problems and the core problem.

In intermediary evaluations of programmes, the evaluator may be asked to construct a decision and an objectives diagram, in addition to the problem diagram itself.

Effect diagram

In an ex post evaluation or in an evaluation with ex post components, the initial analysis of the situation and the selection of the main problems can be compared with the analysis of the situation undertaken at the end of the period assessed. They can also be compared to an effect diagram resulting from the strategy's implementation, in order to assess the contribution of the development assistance to the resolution of problems.

Other analytical tools

In the context of an ex post reconstitution of the events, the problem diagram may be supported by other analytical tools, such as:

  • SWOT analysis
  • Socio-cultural analysis

WHAT ARE THE PRECONDITIONS FOR ITS USE?

Available information

When the problem diagram is found in the basic documentation of the programme, or when it can be directly deduced from a logical framework, it must be presented with a study of the choices underpinning its construction. This requires the consultation of the documentation which has supported its construction, and the interviewing of the actors directly involved in its development.

When key actors and documentation are not available

Subsequent to the consultation with specialists, if required, the evaluator may present conclusions about the relevance of the problems system (illustrated in the diagram).

When the diagram has been developed on the basis of the documentation at the disposal of the evaluator, the latter should:

  • check the validity of the documentation, and enlarge the scope of the analysis to additional sources, if needed
  • test the validity of the hierarchical links and the problems' interrelationship with documentary research and/or interviews

If neither the archives documentation, nor the key actors are available, the evaluator must explain the hypotheses at the basis of the diagram's construction, by providing the reader with precise sources (quotations and comprehensive references of the documentation) underpinning the hypotheses.

.

HOW SHOULD THE PROBLEM DIAGRAM BE CONSTRUCTED?

.

STAGE 1: HOW TO IDENTIFY THE PROBLEMS

Record the references to the problems found in the documentation

Quotations from the evaluation's baseline documents are used to identify problems which may not be systematically depicted as such in the texts.

Sometimes, problems are identified as assistance objectives or impacts targeted by these objectives.

Problems are thus expressed as:

  • problems (presented as such)
  • objectives, whose goal is to resolve explicitly or implicitly a problem
  • expected impacts of the CSP assistance

What is to be recorded in the documentation?

 

The preliminary analysis of quotations can reveal:

  • Repetitions, partly due to the numerous presentations of the situation (executive summary, strategy, programme, annex)
  • Imprecise formulation (for example, generic "economic and social development" which could be more specific)

A thorough selection should therefore be undertaken.

Distinguish context problems from main problems

Among the various problems, the documentation often identifies problems which should be solved by the intervention. The evaluation team should therefore identify the problems related to the intervention in the analysis of the situation, in order to incorporate them into the diagram. The remaining problems should be considered part of the context.

The problems directly targeted by the intervention may not be explicitly identified. Main problems and context problems can be intermingled, which complicates the construction of the diagram. The evaluator will only be able to differentiate the two types of problems after the completion of the diagram.

Whatever the situation, the evaluator should present the list of all the problems under consideration in the annexes, split into two categories:

  • Context of the intervention
  • Object of the intervention

Identify the core problem

The crucial stage of the process is the identification of the core problem among the variety of the selected problems.

Three situations can be encountered.

The core problem is mentioned in the documentation
It is usually a wide and consensus problem, such as structural poverty, rebuilding after a conflict or a natural disaster.

The core problem is not clearly mentioned, whereas the overall objective is
In this case, the evaluator can deduce the core problem from the high-order objective.

These two situations do not need further developments: the evaluator records the core problem explicitly or implicitly mentioned in the documentation, whatever his/her opinion about the problem selected.

Neither the core problem, nor the overall objective are explicitly mentioned
The evaluator plays the role of the planning manager, and decides which of the problems will be the core problem. He/She may:

  • Take the decision, on the basis of his/her rationale
  • Benefit from the assistance of specialists during interviews or working sessions

Whatever the procedure, the selection of the core problem should be conducted concurrently with the classification of the problems into levels. Indeed, the selection of the core problem should be supported and justified by the coherence of the whole diagram.

Proceed to a first classification of the problems by rows

Several situations must be considered:

  • The row is explicit
  • The cause-and-effect link between two problems is explicit, which eases the location of one in relation to another
  • No indication is given about the problem's row (per se or relative). The members of evaluation team must draw their own conclusions and/or ask the assistance of an expert specialised in the fields where they lack the required competence.

Each of the problem's classification by row must be justified (with interpretations presented as explicit assumptions, if needed).

One of the main difficulties in the graphical illustration of problems is to distinguish between "short-cuts" and inconsistencies.

In a "short-cut", the succession of causal links is pictured as a causal link between the first and the last problem of the causal chain.

For example, the chain illustrating the impact of inadequacies of the road infrastructure on poverty can be illustrated as follows:

 

In the same document, quotations can be illustrated as follows:

 

Or:

This example does not portray inconsistencies but stylistic short-cuts.

 

STAGE 2: HOW TO CONSTRUCT A PROBLEM DIAGRAM

Establish a temporary diagram

The diagram may intend to faithfully reflect the problems system as stated in the strategic and planning documents with missing elements and inconsistencies. If no faithfully constructed diagram is explicitly requested, the evaluators will consider it as a step to the construction of a logically reconstructed problem diagram.

Stages for the drafting of temporary diagram

 

Should the diagram's construction start with the core problem or the in-depth causes?

Usage highlights the fact that these two types of problems/causes are the easiest to identify, whereas intermediary causes are the hardest to determine and classify. Thus, it is recommended that the development of the diagram starts with its extremities at the same time.

Identify the authors who have analysed the situation

Before testing the temporary diagram, the evaluator should identify the main and secondary authors who have contributed to the analysis of the situation (writing of a chapter, sector-based contributions, participation in working sessions, etc.).

It may be useful to check the impact of this analysis on the definition of the strategy and programme operational objectives.

Test the temporary diagram

Where possible, the authors of the documentation referred to above should test the diagram in order to validate the classification of the problems by rows and links. The aim is to check that the evaluator's interpretation reflects the authors' intention correctly. If the authors are not available or, in order to complement their original contribution, the evaluator should consider asking for the participation of other actors responsible for the drafting process.

Contacting the authors is usually possible when the documentation is recent and the authors are still in their position or contactable in one of the services. This task is challenging when the documentation is old and its authors are not easily identifiable or contactable.

Develop the final version of the problem diagram

The final version takes into account the opinions collected during the testing of the temporary diagram.

It is an accurate account of the initial intentions logically reconstructed of the European Commission, taken from official documentation.

HOW MANY DIAGRAMS MUST BE DEVELOPED?

A single diagram

If the analysis of the situation does not radically evolve during the evaluation and if the activities focus on a single core problem, one diagram is enough.

Several diagrams

  • When the country situation (the country's own or its strategic context) evolves during the evaluation period, and the analyses clearly reflect these changes.
  • When the evaluation team cannot identify a single core problem and when various core problems result from a fairly independent causes system, it is appropriate to establish as many diagrams as core problems. All the verification procedures must be used to support the various diagrams, and to prove the existence of a range of overall objectives.
  • If the information is sufficient, a diagram can be established for each of the priorities or sectors of concern of the assistance.

HOW SHOULD PROBLEM DIAGRAMS BE USED?

When it is impossible to directly establish objectives diagrams, problem diagrams play an essential role in the organisation stage of the evaluation.

Conversion into an objectives diagram

Problem diagrams present a summarised vision of the situation to be improved, partially at least, by the strategy. The classified objectives included in the strategy should be deduced from the problems diagrams. 

Planning stage

During the planning stage (of a project or a programme), the conversion of the problem diagram into an objectives diagram is not automatic, because each problem integrated into the diagram does not have the same priority.

Evaluation stage

During the evaluation stage, the reconstruction of the problem diagram includes a step which differentiates between context problems and intervention problems. As a consequence, the diagram resulting from this selection should be completely convertible into a logically reconstructed objectives diagram, i.e. the higher-order overall objective corresponds to the core problem, and each row of subordinated objectives to its equivalent row of problems.

Conversion of the problem diagram into an objectives diagram

Extract from 'A summary of the theory behind the LFA method', The Logical Framework Approach (LFA), SIDA - Swedish International Development Co-operation Agency, Draft, June 2002.

 

The objectives diagram displays the classification of objectives, from the higher-order objective to the projects and programmes that are designed to achieve the overall objective. The diagram can provoke comments about:

  • Its internal coherence
  • The relevance of the objectives, regarding the overall context (i.e. the overall strategy of the European Union in terms of country assistance, the situation of eligible countries and the objectives of major actors, the intervention scope and the objectives of other donors)
  • The appropriateness of the means allocated to achieve these objectives

These comments must be presented in the reports and notes of the evaluation organisation stage. They must also appear as an annex in the final report.

Definition of the themes for evaluation questions

The previous analyses raise questions about:

  • The consistency of the strategy with the European Union's overall objectives of its country assistance
  • The relevance of the strategy to the situation of eligible countries and the policies of the major actors (governments in particular)
  • The need for complementarities and synergies with other donors
  • The consequences of a lack of coherence in the objectives system
  • The appropriateness of the means to the objectives

These questions lead to the determination of a range of themes which could be investigated during the subsequent stages of the evaluation, particularly through evaluation questions.

However, evaluation questions cannot be automatically deduced from these analyses. Other questions may emerge during the organisation stage, formulated by the strategy implementing operators. Such questions can relate to the difficulties encountered during the planning and implementation stages. However, the number of evaluation questions is limited and additional questions should be the subject of a selection process.

HOW ARE THE FINDINGS PRESENTED?

Notes and reports of the organisation stage

Problem diagrams are established during the organisation stage, where the reports and notes should be provided. At this stage, the construction of the diagram must be precisely described.

The precise sources identifying the problems (exact quotations, references to original documents) must be provided. References to specific documents, interviews, and expertise must support the location of the problems in the diagram, and the assumptions developed during the diagram's construction must be explained.

Final report

The problem diagram should be incorporated into the final report because it is not only a stage in the development of the evaluation, but also an illustration of a possible analysis of the evaluation's strategy.

The report itself may include a short presentation of the diagrams, in addition to its development in the annexes.

Verbal presentation

The evaluation team will need to present its work (methodology and findings) to different groups of people (the evaluation reference group, participants in the debriefing session, etc.). The problem diagram is a very efficient tool for these situations, providing that it is readable without being over-simplistic.

To do so, a main diagram and several sub-diagrams developing fundamental sections of the latter should be presented, each of them not exceeding more than 20 items.

WHAT ARE THE PRECONDITIONS FOR THE DEVELOPMENT OF THE PROBLEM DIAGRAM?

Human resources and working arrangements

The need for human resources varies with the tasks to be achieved and the situation encountered: reconstruction of the problem diagram with the analysis of the situation displayed in the documentation; retrospective reconstruction of the problem diagram as it should have been established to support the relevance of the strategy or programme under assessment.

The right column of the table below illustrates a wide range of working days.

Type of work required for the design of a problem diagram (usual situation)

Tasks Types of competences Categories of professionals Number of working days (estimation)
Identification

Collection of the documentation

Study of the documentation

Logical process of thinking

Knowledge of the European Commission's strategies and programmes development procedures

Knowledge of computer tools

Medium or junior professionals 5-10 days

Analysis of the problems identified by the documentary work

Construction of the diagrams

Test of the diagrams

Construction of the final diagrams

Logical process of thinking

Experience in the fields covered by the strategies and programmes

Specific knowledge of the country, sector and theme under consideration

Multidisciplinary team of experienced evaluators, whose specialities should cover the thematic evaluation scope

Freelance experts

5-10 days

0-5 days

Travelling expenses

Strategy papers are drafted under the responsibility of the head office (DG RELEX or DG DEV), which is also responsible for the planning stage. EuropeAid is in charge of project drafting, which explains why the majority of the useful documentation (including the complete baseline documentation) is in Brussels. Some tasks may be devolved to Delegations in the future, whose responsibilities in this area will grow.

In a country evaluation context, the retrospective construction of the diagram requires at least 2 or 3 experts working on a mission of 5 to 10 days in the country under consideration.

Computer devices

Most of the graphic problems can be solved with software like MS PowerPoint.

The evaluator can also benefit from specific software designed to assist decision-making, such as MS Visio, TeamUp-PCM for Windows, and Palisade Precision Tree.

.

IN COUNTRY EVALUATION

.

WHEN SHOULD IT BE USED?

Donors must explicitly or implicitly explain the decision to provide assistance to a country or a region in a given timeframe. For this purpose, they must determine the objectives of the assistance designed to solve a range of problems (the objectives can be the donor's or the beneficiary's).

The purpose of an evaluation is not to analyse the problems and establish the related diagram, but rather to reconstitute the analysis which has already been undertaken and convert it into a diagram.

Consequently, in country evaluations, the problem diagram is particularly appropriate for the following two situations.

To examine the explicit problems included in the documentation

The conversion of the analysis into a diagram (Stage 2: how to construct the problem diagram) enables the evaluator to check the coherence of the analysis and its relevance to the objectives. For this purpose, a problem and an objectives diagram should be reconstructed (how should problem diagrams be used?) and be compared.

To analyse the non-explicit problems in the documentation

A study of the analysis of the situation facilitates the establishment of a problem diagram, whose identified problems should be solved by the assistance. This stage should ease the reconstitution of an objective diagram, or as a minimum, validate the draft of the diagram established in accordance with the documentation.

WHICH PROBLEMS MUST BE TAKEN INTO ACCOUNT?

In a bilateral assistance strategy, the European Commission is the donor, and the States and national public institutions are usually its partners. The European Commission and the States have their own overall objectives, corresponding to their own issues. Yet, to a certain extent and degree of detail, the problems become common.

The donor's range of problems

This type of problem corresponds to strategic issues meant to be resolved with financial assistance (for example, regional instability, weak absorption capacity of the local market, uncontrolled immigration).

The country beneficiary's range of problems

This range of problems is related to major political issues to which the States are confronted (macro-economic unbalances, social instability, weak growth, social or geographical disparities, etc).

The range of problems common to the donor and the country beneficiary

This type of problem has common origins which are relevant for both the donor and the country beneficiary, and are relative to the fields of the assistance scope (for example, health, SME, irrigation and environment).

Except for participatory evaluations, the evaluation team will prioritise the first and third type of problems, because they are directly relevant to the donor.

WHICH DOCUMENTATION SHOULD BE REFERRED TO?

Where can documents analysing problems be found?

Documents related to the European Commission strategy
Country Strategy Papers (CSP) do not systematically describe long-term problems, whose partial solution is to be found through the implementation of the European Commission's strategy. These problems are sometimes described in the presentation of the strategy objectives, or can be found in the European Council's political papers expressing the institution's interest in the country situation.

Documents related to development assistance programmes
The problems analysed by development assistance projects and programmes constitute the core of the chapter "Analysis of the situation" in the CSPs. This chapter, however, does not always distinguish the problems included in the context of the projects and programmes from the problems at the origin of such projects and programmes.

Thus, the construction of a problem diagram requires the evaluator to distinguish context problems from main problems.

Which documents should be selected?

Successive versions of the analysis of the situation can seldom be found (in strategy or programme papers). After ensuring that previous drafts do not yield more in-depth and valid analyses, the evaluation team mainly works on the final version.

The baseline documentation comprises the (CSP) and the corresponding National Indicative Paper (NIP), covering the timeframe of the evaluation scope. They are usually provided to the evaluation team at the start of the mission.

Often, the evaluation period (defined in the Terms of Reference) is covered by several strategy and programming papers, each of them including analyses and information about the situation of the country, specific sectors or regions.

Usually, the situation and problems are analysed at the start of the period covered by the strategy and/or the programme under assessment. Yet, the analysis may be affected by difficulties concerning the information, such as:

  • Late publication of the data
  • Difficult access to information
  • Poor or uncertain quality of the data

These difficulties, which may even weaken the relevance of the objectives, should not be underestimated by the evaluation team.

WHICH TYPES OF PROBLEM DIAGRAMS CAN BE USED?

Complete problem diagram

Used in participative planning, the complete problem diagram sketches on each side of the core problem the problems identified by the stakeholders.

Simplified problem diagram

The evaluation only focuses on the lower part of the diagram, because the projects usually deal with the causes of the problems.

Complex problem diagram

Usually, the evaluation team limits its task to the reconstruction of a simple tree shape diagram (simplified or complex, depending on the case) which is derived from the identification of the core problem.

Network diagram

This type of diagram is faithful to the complexity of an actual situation. Yet, its reconstitution requires a wealth of information which is seldom available. Furthermore, the conversion of such information into a diagram represents a challenging task.

EXAMPLES

 

BIBLIOGRAPHY

 

The logical framework and the objectives tree

  • Manuel de Gestion du cycle du projet, Commission Européenne, EuropeAid, mars 2001. [FR]
  • Guide récapitulatif des Formations - Gestion du Cycle de Projet, Commission Européenne, EuropeAid, Itad Ltd, Particip GmbH. version 1.1, février 2001. [FR]
  • 'Logical framework analysis', BOND Guidance Notes No 4, March 2003.
  • 'The Logical Framework Approach (LFA)', A summary of the theory behind the LFA method, Sida - Swedish International Development Co-operation Agency, Draft, June 2002.
  • 'The Logical Framework Approach', Handbook for Objectives-oriented Planning, NORAD, 4th edition, January 1999.
  • 'The logical framework approach', Australian agency for international aid (AusAid), AusGuide, AusGuidelines, 2000.
  • 'Logical frameworks: a critical assessment', Working paper # 264, Des Gasper, Institute of Social Studies, the Hague, December 1997.

Use of the logical framework in evaluations:

  • 'Using the Engendered Logframe for Monitoring and Evaluation', Helen Hambly Odame, ISNAR, August 2001.
  • 'Project appraisal 2 (4)', Logical Framework Approach to the Monitoring and Evaluation of Agricultural and Rural Development Projects, Coleman G., 1987, p. 251-259.

The other approaches:

  • Program Theory in Evaluation: Challenges and Opportunities, New Directions for Evaluation, a publication of the American Evaluation Association, Fall 2000.
Evaluation methodological approach > EN: Methodological bases and approach > Evaluation tools > Problem diagram > Check lists

Check lists

CHECK-LIST FOR EVALUATORS

.

Questions Answer
Preparation and design  
Has a clear agreement from the head of the evaluation been obtained on the period concerned and the relevant documentation?  
Has the dating of the documentation been confirmed by their authors or contributors?  
In case of inaccessibility or uncertainty about the relevance of the collected documentation, has the risk encountered been indicated?  
Implementation  
Are justified assumptions, underpinning the problems classification, formulated in the report?  
Have the wording and classification of problems been checked during a new reading of the documentation?  
Has the temporary diagram been tested with the authors of the documentation, or in case of unavailability of the authors, has the testing been enlarged to competent managers? If not, is this situation clearly stated in the report?  
During the establishment of the logically reconstructed diagram, has the baseline documentation been consulted to support hypotheses? Have highly skilled specialists been consulted, by means of a written notice if necessary?  
Have the opinions of the authors and other managers been compared to the documents and was there a triangulation of the perspectives?  
Are the documentary references and quotations presented in the report? Are the specialists who have been consulted quoted in the report?  
Has the problem diagram been compared to the effect diagram?  

.

CHECK-LIST FOR MANAGERS

.

Questions Answer
Has the temporary diagram been tested?  
Have the opinions of the authors and other managers been compared to the documents and was there a triangulation of the perspectives?  
Are the documentary references and quotations presented in the report? Are the specialists who have been consulted quoted in the report? Have the specialists consulted been quoted in the report ?  
Was the problem diagram compared with the objectives diagram and the effects diagram?  
Evaluation methodological approach > EN: Methodological bases and approach > Evaluation tools > Objectives diagram and the effect diagram

Objectives diagram and the effect diagram

SUMMARY

.

Why are these tools used in evaluation?

These diagrams are usually used as organisation tools. They provide a framework for the information collection and the undertaking of in-depth interviews during the assessment of the programme or strategy. They relate direct outcomes and field effects with the expected objectives and effects of the programme. The construction of the diagrams should therefore constitute one of the first tasks of the evaluation.

The objectives diagram illustrates the objectives classification, from the global objective to detailed operational objectives.

The effect diagram displays the classification of the results, outcomes and effects of what is intended from the implementation of the objectives system.

Figure 1: Rationale of the objectives diagram and the effect diagram

What are the possible uses of these diagrams?

In the evaluation context, diagrams are used to reconstruct the intervention logic of the European Commission's support to the country. This reconstructed logic will be shaped into one or more logical diagrams of effect. Prior to the preparation of effect diagram(s), the team will have to prioritise the stated cooperation objectives and translate these into intended effects. These intended effects will form the "boxes" of the diagram(s). Possible "holes" in the intervention logic will be indicated and filled on the basis of assumptions to be validated by the reference group. The effect diagram(s) will help to identify the main evaluation questions.

Figure 2: example of an objectives diagram

 

The faithfully reconstructed objectives diagram displays the objectives system and provides the evaluator with a first approach to the strategy and policies inner quality. Indeed, an unclear, incomplete or incoherent diagram means a lack of relevance in the resulting planning or a lack of faithfulness to the initial objectives system.

During the implementation of the strategy or programme, external events, such as the evolution of the world commodities market price, elections, political changes, or the conclusion of international agreements can influence the achievement of the objectives and expected outcomes. Comments dealing with the discrepancies between the expected and the observed outcomes should take these events into account. A would-be coherence
Diagrams establish logical links between objectives or effects. Each objective or effect is presented as logically dependent on a higher row objective or effect. The outputs of the activities implemented or scheduled by the programme appear as a contributions to the overall objective/effect and support the coherence of the objectives and effect system.

A strategy or a programme seldom address the full scope of the overall objective, which is limited to choices for each row of objectives or effects. Unless the evaluation can find an explicit explanation of the choices made in strategic documentation, it must provide an answer to series of questions:

  • Do these choices represent the only possibilities?
  • Are the objectives the most relevant ones for the row 'x'?
  • What external factors have led to this selection?
  • What reasons have led to the elimination of certain objectives?

How are they constructed?

After determining the evaluation scope, the evaluators should construct a diagram illustrating the objectives presented in the strategy and planning documents. The objectives diagram is drawn from this diagram. When the evaluation scope covers one or more strategy papers (geographical) or strategic policies (sector-based, thematic), it is recommended that one diagram per document is created (unless there is a logical continuity in the strategy or the policy). When a logical framework has supported the drafting of a programme, it clearly states the various levels of objectives targeted by the programme. The logical framework is consequently a reference point for the evaluation, as a consequence of the presentation of the objectives diagram's basic constituents. Although the establishment of result-based logical frameworks has not yet been generalised, the effect diagram can usually be deduced from the logical framework of the intervention's objectives. The objectives diagram and effect diagram can also be used in evaluations for projects and programmes whose rationale is not explicit in the logical framework. In practice, the objectives and intended effects of complex policies and strategies often lack explicit presentation and logical structure, whereas the justification for an evaluation is to be able to answer the following question:

  • Have the objectives or intended effects of the policy or the strategy under assessment been achieved?

What are the preparation stages for the construction of the diagrams?

Stage 1: delimit the evaluation scope

The terms of reference include information about the timeframe and the financial tools to be assessed. Note that projects and programmes originating from previous documentation can also be implemented during this timeframe. 

Stage 2: identify the objectives and effects

Collect the documentation required for the establishment of the diagrams. It should comprise:

  • Policies, strategies and programmes baseline documentation
  • Complementary official documentation

An overall analysis of the break points in the global strategy should be undertaken, without including the objectives of the projects and programmes in progress and scheduled for a different timeframe from the one assessed.

Establish a list of the objectives recorded in this documentation. 

Stage 3: construct a faithfully reconstructed objectives diagram

A provisional classification could be carried out on the basis of the distinction between three levels of objectives:

  • Higher-order objectives (goals)
  • Short-term and medium-term intermediate objectives
  • Operational objectives

La définition des relations logiques occupe une place centrale dans le classement. Elle est avant tout affaire d'expérience. 

Stage 4: convert the faithfully reconstructed objectives diagram into a logically reconstructed objectives diagram

Some faithfully reconstructed objectives diagrams may reveal logical defects in strategy or political papers, such as:

  • Strategy or political papers involving a range of objectives without sufficient details to classify them
  • No definition of the higher-order objective (goals)
  • Weak relevance of the causal links between objectives
  • The objectives of a given row do not result in any objectives

In order to prepare a comprehensive and coherent objectives diagram, the evaluation team will need all available documentation, its own expertise and, if required, that of experts.

Each of these rationalisation operations should be explained in a technical note. 

Stage 5: construct the effect diagram

Objectives diagrams and intended effect diagrams share the same rules of construction. The effect diagram is constructed from the conversion of each of the objectives into the corresponding intended effect:

  • The higher-order objective corresponds to the higher-order effect
  • Intermediate objectives correspond to intermediate effects or outcomes
  • Operational objectives correspond to results

Computer devices

Most of the graphics can be addressed with software such as MS PowerPoint.

How are the findings presented?

Objectives diagrams are established during the organisation stage, where reports and notes should be provided. At this stage, the diagram's construction must be precisely described.

For the faithfully reconstructed objectives diagram, the sources of the objectives/effects (quotations, references to the original documentation) must be provided. References to documentation, interviews and expertise must support the objectives' location in the diagram, and the assumptions developed during the construction of the diagram must be explained.

Figure 3 : example of the EC strategy in Tanzania

The process through which the logically reconstructed objectives diagram has been extracted from the faithfully reconstructed objectives diagram must also be clearly explained. 

Verbal presentations

The evaluation team will need to present its work (methodology and findings) to different types of people (the evaluation reference group, participants in debriefing sessions). The objectives diagram and/or the effect diagram are very efficient tools for this purpose, providing that they are readable without being over-simplistic.

To do so, a main diagram, and several sub-diagrams developing fundamental sections of the main diagram, should be presented, each of them not exceeding 20 items (boxes).

What are the preconditions for its use?

Figure 4: the preconditions for its use
The time span Collection of the documentation and identification of the objectives: 4 to 6 working days
Analysis and construction of the diagrams: 5 to 10 working days
Test of the diagrams: 1 to 4 working days
Human resources Knowledge of the European Commission's development procedures for strategies and programmes
Knowledge of strategies and programme documentation
Knowledge of computer tools
Logical process of thinking
Experience in the fields covered by the strategies and programmes
Specific knowledge of the country
Financial resources A budget around €5,000 should be fixed for the problem diagram, objective diagram and decision diagram.

What are the advantages and limitations of the tool?

Figure 5 : The advantages and limitations of the tool
Advantages Logical classification of the objectives and the effects: the diagram explicitly illustrates the objectives/effects and their various rows, from the global objective (more or less long term), down to the range of activities that have already been implemented or are to be undertaken.

It reveals the logical links between objectives/effects, or the lack of a linkage.

Definition of the evaluation questions. Through the diagram, the evaluator examines a range of questions whose aim is to help answer the crucial question: To what extent have the stated objectives and the intended effects been achieved?

Presentation of the strategy: when the objectives diagram is well structured and clearly presented, it is a valuable educational tool which facilitates the understanding of the strategy.

Limitations Replacing the objectives tree for the objectives diagram in the illustration of the objectives system avoids most of the construction constraints and their inherent limitations (with one exception: the representation of retroactive links).

A simplified representation of reality: as a chart, the diagram is a simplified representation of reality, and its educational value depends on the selection of a sensible degree of simplification.

In order to avoid an over-simplification of the facts, the evaluator can develop sub-diagrams focusing on specific parts of the main diagram.

Evaluation methodological approach > EN: Methodological bases and approach > Evaluation tools > Objectives diagram and the effect diagram > Detailed presentation

Detailed presentation

.

This section is structured as follows:

.

WHAT IS AN OBJECTIVES DIAGRAM AND WHAT IS AN EFFECT DIAGRAM?

.

WHAT DO THEY REPRESENT?

Objectives and effects: definitions

Development assistance (in terms of projects, programmes and strategies) usually focuses on an objective to be achieved, or an intended effect.

The objective is expressed in terms of an intervention, whereas the impact is expressed in terms of consequence of the intervention.

Various glossaries of evaluations provide definitions for these two terms, such as in the OECD's glossary (Glossary of Key Terms in Evaluation and Result-Based Management).

The objective of the external assistance covers a variety of aspects, from factual interventions (roads, training, etc.) to macro-economic and social changes. In evaluation, the term impact conveys the idea of wide-scope and long-term effects. By convention, medium term effects are called outcomes, and short-term effects results. The tool will refer to the term effect, so as to include the development assistance's range of outputs (results, outcomes and effects).

When the diagram is used as a structuring tool, the effects presented in the diagram are the effects intended from the assistance.

The diagram can also be used as an evaluation tool for the analysis of the programme's outcomes. The effects presented in the diagram are the observed effects.

At the planning stage, the assistance goals are usually expressed in terms of objectives. At the evaluation stage, the concepts of objectives and effects can be used either way.

The role of a diagram in the planning process

The objectives system

Usually, a long term and global strategic objective, assumed to be a first level objective, is fulfilled through the completion of a range of second level objectives. This is true even where the first level objective is straightforward. Each second level objective depends on the completion of several third level objectives, and so on, down to operational objectives (the intervention projects).

Therefore, an objective is usually understood as a means to achieve a superior level aim, while depending on the completion of subordinate means or objectives.

The objectives system is the presentation of all the objectives of all levels with their respective links. 

The effect system

The outputs of the implementation of a project, a programme or an assistance strategy include direct results, the short-term outcomes which are linked to these results, and the longer term impacts (direct and indirect). Such outputs can be called 'effects' and are linked together in a range of causal relations or synergies at the basis of the effect system. 

The objectives tree

The need for a logical classification of the range of objectives in objective-based planning is at the origin of the drafting process of programmes and projects. The objectives system is usually presented as a system of roots and a trunk, hence its common name, the objectives tree.

Such tree-like illustrations, however, are subject to some basic rules:

  • There can only be one first row objective, also called the global objective
  • Each subordinate objective is related to one objective in the row above
  • Interactions between objectives in the same row and feedback links (i.e. effects becoming causes and vice versa) are not represented

The level expresses the place of the objectives in the cause-and-effect system. In the diagram, the level is represented by the row.
Standard objectives tree

When these rules are applied, objectives trees are a simplified illustration of the objectives system defined during the drafting of the strategy (or the programme).

In the context of the European Commission development assistance, objectives trees show the classification of objectives to be achieved in a geographical (region or country), sector-based or thematic strategy. They range from the European Union's long-term global objective, down to the activities carried out in operational programmes.

The role of the objectives diagram and effect diagram in an evaluation

What actually happens in thematic, sector-based or geographical evaluations is more complex than that illustrated in objectives and effect systems. Indeed, strategies and programmes under evaluation do not systematically result from objective-based planning. Even if they do, the logical classification of objectives may result from decisions based on circumstances rather than from a rational selection derived from the fundamental issues.

This situation may result in:

  • Two or more first row objectives which cannot be reduced to a single global objective
  • Objectives of a given row which are linked to several objectives of the immediate row above
  • Objectives of a given row which are related to an objective belonging to 2 rows above
  • Interactive objectives at the same level

The use of standard objectives trees in evaluations assumes that the definition of the objectives is rational and the strategic scope can be reduced to a simplified illustration.

These limitations explain why the use of objectives diagrams, whose shape can fluctuate more, is favoured to strictly codified objectives trees.

The effect diagram

International development assistance has tended to evolve progressively in the fields of planning and management from an objective-based approach to a result-based approach. Thus, the evaluation must take into account the effect system linked to the programme or the strategy to be assessed, and organise it. The term effect diagram is used to describe the theoretical organisation of the effect system (outputs, short-term results and intermediate impacts) which leads to the overall intended impact.

The effect diagram displays the classification of the results, outcomes and impacts of what is intended from the implementation of the objectives system. Its tree-shape connects the actual activities which have been planned, and the outputs, which should produce direct results, to the medium term intermediate impacts and the long-term global impacts. The tree can also be read from the bottom to the top, the long-term overall impact being reached after the implementation of intermediate impacts, results, outputs and interventions.

WHERE DOES THE OBJECTIVES DIAGRAM COME FROM?

The objectives diagram comes from objective-based management whose use expanded during the 1960s in the economic sphere of the United States and in Western economies.

It was first adapted to the requirements of the United States Department of Defense and, in the late sixties, to the USAID country assistance management as a component of the logical framework.

Thus, the first well-known use of objectives diagram (including USAID), relates to the drafting of logical frameworks.

The objectives diagram belongs to a series of country assistance decision tools, and can be combined with the problem diagram in the context of Goal Oriented Project Planning (GOPP, originally called ZOPP in German for Zielorientierte Projektplanung). 

At the start of the 1980s, GTZ (German Technical Cooperation) conducted a methodological revision of ZOPP, to be used as an educational tool to determine the stages of the participatory process which produces the logical framework.

The typical sequence can be illustrated as followed:

Analysis of the problems of a simplified sequence in the project cycle

In this context, the objectives diagram is:

  • a useful tool for the drafting of projects
  • a planning methodology tool, which comes from the problem tree and contributes to the logical framework
  • a participative decision tool which embraces the main stakeholders of the project, whose determination is the first stage of the project cycle sequence

Following GTZ's initiative, various public donors and operators (agencies and NGOs) have used the problem tree as a component of project planning and management.

Its use as a participatory and appropriation tool has not always played the paramount role defined by the ZOPP designers, indeed, in many cases, the stakeholder analysis was insufficiently comprehensive and the participation procedures were extremely formal, resulting in the problem tree becoming a formal exercise, even a manipulation tool. The logical framework was therefore limited to rigid project management tool, ignoring the fact that it is first and foremost a learning and negotiation tool. As a consequence, emphasis was put on over-simplistic wording, forgetting that these corresponded to the educational objective of the tool.

The biggest criticism of the objectives diagram is the assumption that a consensus can be reached for development objectives and strategies among stakeholders (donors, operators and beneficiaries).

Inadequacies and modifications at the programme drafting stage should be examined during the evaluation, which should focus particularly on the drafting process of the objectives system and avoid tools requiring or yielding a simplistic representation of the system.

WHAT SHAPES CAN AN OBJECTIVES DIAGRAM AND AN EFFECT DIAGRAM TAKE?

Tree shape or complex diagram

The standard objectives tree is subject to the following basic rules:

  • A single global objective
  • Subdivision of the objective of a given row into 2 or more objectives of a row below (level 2)
  • Relation of a defined objective with a single objective of a row above (level 3)
  • No horizontal links
  • No feedback links (i.e. effects become causes and vice versa)

When trees are drafted in the context of objectives-oriented planning, the first row objective is usually called the "global objective". The lower row objectives have various names, but preferably "intermediate objectives". Several rows can be dedicated to intermediate objectives. The objectives located at the bottom of the tree are called "operational objectives".

The objectives diagram set out below is a complex objectives system:

  • One of the intermediate objectives in row (2) is directly linked to the global objective and not to another intermediate objective (1).
  • Two intermediate objectives in row (3) are interrelated.
  • One of these objectives is linked to two objectives of the row above (intermediate 2).
  • Row 2 of the intermediate objectives appears incomplete (lack of subdivisions to the lower row objectives).

Complex objectives diagram

 

Examples of more complex objectives systems can be found, having two global objectives and subdivisions of intermediate objectives, however, such objectives diagrams cannot present feedback links.

A structured presentation of the effect system (the most recent one) has taken into account the limitations of the tree shaped representation and adopted the flexible shape of the diagram. As a consequence, the systems of results, outcomes and effects can have the same degree of complexity as the objectives system, and can be illustrated by similar diagrams.

Effect diagram

Vertical or horizontal diagram

Diagrams can either be vertically or horizontally oriented.
In their vertical shape, they read from top to bottom, or from bottom to top.
In their horizontal shape, they read from right to left, or left to right.

WHAT IS A LOGICAL LINK?

Definition

The objectives and effects illustrated in the diagrams are related to each other with horizontal or vertical links. These links are called "logical" when expressing an inference relation (induction or deduction) which has been validated by experience. They highlight the fact that:

  • The adoption of an objective/effect at a certain row implies that of subordinated objectives/effects
  • An objective/effect of a certain row can be deduced from an objective/effect of a row above (immediate or not)
  • Two objectives/effects of the same row share a synergetic relationship

The logical nature of the links between objectives/effects are essentially based on experience and development theories (derived from this experience). Experience and theory can determine which subordinated objectives/effects should result from the implementation of objectives/effects of a certain level, in order to be properly fulfilled.

However, in terms of development, experiments do not always result in the same conclusions and, as a consequence, should not be considered universally applicable. This can be accounted for by the dependency of the objectives/effect on various factors, often not well-understood.

For example, even when experts agree on the relevance of a global objective, they may disagree on the definition of intermediate objectives, and in particular, operational objectives. The content of overall objectives such as poverty alleviation is continuously debated and evolving.

Verification of the logical nature of the diagram

The logical links between objectives/effects (i.e. the logical nature of the objectives diagram and effect diagram) can simultaneously be checked with:

  • Recognised experts who have worked in different countries and situations
  • Designers and implementing managers of programmes included in the evaluation scope

These two groups of actors may validate the logical nature of the diagram unanimously or individually.

The evaluation should clearly present the process of the validation and its outcome.

.

WHY AND WHEN ?

.

WHEN SHOULD AN OBJECTIVES DIAGRAM BE USED?

Three fundamentals in development assistance programmes

 

An assistance programme can be viewed from different perspectives: its results, internal procedures, partnerships, etc. In terms of efficiency, its immediate results and/or impacts should be understood in relation to the various objectives and/or effects defined in the assistance programme. This implies that the objectives must be identified, or easily recognisable, and classified, or easily put into a hierarchy.

In programmes established on the basis of the logical framework

When a logical diagram has supported the drafting of a programme, it clearly states the various levels of objectives which are targeted by the programme. The logical framework is consequently a reference point for the evaluation, as a consequence of the presentation of the objectives diagram's basic constituents. Although the establishment of result-based logical frameworks has not yet been generalised, the effect diagram can usually be deduced from the logical framework of the intervention's objectives. The evaluator should reproduce the objectives diagram included in the programme, or, if it is not available, reconstruct it with the help of the logical framework's first column, and, if necessary, convert it into an effect diagram.

In more complex policies and strategies

The objectives diagram can also be used in evaluations for projects and programmes whose rationale is not explicit in the logical framework. In practice, the objectives and intended effects of complex policies and strategies often lack explicit presentation and logical structure, whereas the justification for an evaluation is to be able to answer the following questions:

  • Have the objectives or intended effects of the policy or the strategy under assessment been achieved?
  • To what extent have these objectives or intended effects been achieved?

In order to carry out the evaluation, the evaluator must determine and rank the objectives/effects of the strategy or the policy. In this context, an objectives diagram combined with an effect diagram is an effective tool for the reconstruction and representation of the objectives system and/or effect system.

For programmes established on the basis of the logical framework, as well as in more complex policies and strategies, the construction of the objectives diagram and effect diagram should be one of the evaluation's first tasks. The diagram orients the first stage of the information collection and the undertaking of interviews, whose goal is to develop an in-depth knowledge of the policy or programme to be assessed.

WHAT ARE ITS ADVANTAGES AND LIMITATIONS?

Its advantages

Logical classification of the objectives and the effects

The diagram explicitly illustrates the objectives/effects and their various rows, from the global objective (more or less long term), down to the range of activities that have already been implemented or are to be undertaken.

It reveals the logical links between objectives/effects, or the lack of a linkage. 

Definition of the evaluation questions

Through the diagram, the evaluator examines a range of questions whose aim is to help answer the crucial question:

To what extent have the stated objectives and the intended effects been achieved?

Presentation of the strategy

When the objectives diagram is well structured and clearly presented, it is a valuable educational tool which facilitates the understanding of the strategy.

Its limitations

Replacing the tree illustration for the diagram avoids most of the construction constraints and their inherent limitations (with one exception: the representation of retroactive links). 

A simplified representation of reality

As a chart, the diagram is a simplified representation of reality, and its educational value depends on the selection of a sensible degree of simplification.

In order to avoid an over-simplification of the facts, the evaluator can develop sub-diagrams focusing on specific parts of the main diagram. 

A would-be coherence

The diagram establishes a logical link between an objective and another belonging to the row above, or between an effect and another belonging to the row above. Each subordinate objective is presented as logically dependent on a higher row objective, and each effect is presented as logically dependent on a higher row effect. The outputs of the activities implemented or scheduled by the programme appear as a contribution to the global objective/impact and support the coherence of the objectives and effect systems.

Yet, the objectives diagram and the effect diagram (whether they are directly drawn from the programme's logical framework or reconstructed from explanatory documentation) conceal a series of choices. Each objective or effect of a given row is achieved through the implementation of subordinate objectives or effects whose selection must be explained. Indeed, a strategy or a programme seldom address the full scope of the overall objective, which is limited to choices for each row of objectives or effects. Unless the evaluation can find an explicit explanation of the choices made in strategic documentation, it must provide an answer to series of questions:

  • Do these choices represent the only possibilities?
  • Are the objectives the most relevant ones for the row 'x'?
  • What external factors have led to this selection?
  • What reasons have led to the elimination of certain objectives?

Selected and rejected objectives

 

Selected and rejected effects

 

If the evaluator does not investigate such questions, the evaluation of relevance and coherence of a programme or strategy may be superficial.

WITH WHICH TOOLS CAN IT BE COMBINED?

Problem diagram

The objectives diagram often comes (but not systematically) from the problem tree.

Problem diagrams illustrate the overall aspects of a situation which should at least be partially improved by the strategy, whose ranked objectives should be deduced from the diagram.

In the planning process, such a deduction is not systematic. Indeed, for various reasons, all the problems listed in the diagram cannot be analysed with the same priority.

The relationship between the problem and the objectives diagrams

 

 

In evaluations, the reconstruction of the problem diagram includes a stage during which the problems caused by the context are distinguished from those following the implementation of activities. The resulting diagram should be easily translated into an objectives diagram: the focal problem should correspond to the global objective, and each level of problems should be matched by an equivalent row of objectives.

Logical framework

In strategy or planning papers, the objectives diagram is often linked to the development of the logical framework, which provides to the diagram its left column and general structure.

Decision diagram

The analysis of the programme's development (decision diagram) can be used to explain the reasons for the selection of objectives for various rows (and particularly the highest row objectives).

In mid-term programme evaluations, the evaluator may be asked to re-establish some of the tools which have been previously mentioned in this methodology, in addition to the objectives diagram.

WHAT ARE THE PRECONDITIONS FOR ITS USE?

Study of the initial documentation

When the objectives diagram and the effect diagram are found in the initial documentation of the programme, or when they can be drawn from the logical framework, they should be presented, with the study of the choices made during their construction. This implies a study of the documentation which was used in the construction of the diagrams and interviews with stakeholders directly involved in their drafting.

In sector-based evaluations, baseline documents can be found in all the sector-based strategy documentation originating from the European Commission (such as communication papers, guidelines, handbooks, etc.), from the partner States and other donors.

Baseline documents also include other strategic documents (such as CSPs) and their corresponding national indicative programmes (NIPs) which are relevant to the evaluation scope under study, as well as bilateral agreements such as Association Agreement, Agreement Cooperation and their monitoring documents.

When key actors and documentation are not available

When neither the documentation nor stakeholders are available, the evaluator should discuss the relevance of the objectives system presented in the diagram with specialists, and formulate a judgement.

When the evaluator has reconstructed the diagram on the basis of the available documentation, the evaluation team must:

  • check the validity of the documentation and, if required, complete it
  • test the validity of the links and interrelationships with additional documentary research and interviews

If archives and stakeholders are not accessible, the evaluation team must explain and describe their own assumptions underpinning the construction of the diagram. The team must therefore provide the reader of the diagram with the sources (quotations and documentary references) on which the diagram is based.

HOW CAN THE OBJECTIVES DIAGRAM BE USED IN PROJECT EVALUATIONS?

A means to concentrate on the intervention rationale of the project

Usually, projects are provided with a logical framework. The analysis should focus on this logical framework, and study the vertical and horizontal coherence of the matrix with the context. In the absence of a logical framework, the intervention rationale should be reconstructed on the basis of the project's documentation (Stage 2: identify the objectives and the impacts).

There may be a delay between the design of the project and its start. In this case, the relevance of the logical framework should be checked when the project is launched.

Any other interventions in the field of application of the project should be taken into account, in order to assess the coherence of the project's interventions with the general context of the assistance.

Which objectives should be taken into account?

The evaluation studies the objectives presented in the logical framework, or in the reconstituted logical framework.

The ownership of the project's objectives should be measured for the various types of stakeholders, particularly in the context of a participative evaluation.

Once completed, the logically reconstructed objectives diagram results in two concurrent evaluative approaches:

  • The analysis of the project's quality criteria (relevance, efficiency, effectiveness, sustainability) in the various levels of intervention (activities, outcomes, specific objectives).
  • The internal consistency of the project's strategy (vertical coherence) and its relevance with the European Union's overall objectives, the Member State's policies, and the International donors' position.

.

HOW SHOULD THE OBJECTIVES AND EFFECT DIAGRAM BE CONSTRUCTED?

.

The evaluation should first identify the outcomes of the assistance and compare them with the objectives to be achieved. The structuring stage requires the analysis of the objectives, from which the intended effects are deduced. 

After determining the evaluation scope, the evaluators should construct a faithfully reconstructed objectives diagram from the strategy and planning documents. A logically reconstructed objectives diagram and an effect diagram (intended effects) will be derived from the faithfully reconstructed objectives diagram.

STAGE 1: HOW IS THE EVALUATION SCOPE DELIMITED

Usually, the evaluation scope is strictly defined in the terms of reference which includes information about the timeframe and the financial tools to be assessed.

In any case (concerning the evaluation timeframe or the financial tools), the managing authority must provide the evaluation team with a precise definition of the scope, although the evaluation team must feel free to warn the latter about the consequences of a too restrictive scope.

Period of the evaluation

The period to be assessed fully or partially corresponds to the period covered by the strategy and/or planning papers.

However, during this timeframe, projects and programmes which are being implemented may have already been described and authorised in previous documentation.

What importance should be given to the objectives or the effects of such projects and programmes?

Two situations should be considered, depending on the nature of the projects and programmes implemented during the period under study (i.e. their objectives are in continuity with or differing from the objectives planned in previous projects and programmes).

Projects and programmes whose objectives differ

The evaluator should undertake a general analysis of the differing topics in the overall strategy and type of activities, and avoid the study of specific objectives (or effects) included in these projects and programmes. 

Projects and programmes whose objectives are in continuity

In this case, the study of their specific objectives (or effects) should be included in the analysis of the objectives.

Financial tools

Explicit terms of reference

Terms of reference can focus the evaluation on the activities carried out by a specific institution (a General Directorate of the European Commission) with a specific financial tool. The scope is strictly limited. The evaluation team should however examine other institutions, include them in the category "other donors", and seek the elements of coherence in their activities which could be missing from the objectives system or the effect system of the institution under evaluation.

The evaluation team should also examine the coordination between activities under assessment and others, and formulate its conclusion on the possible effects of insufficient coordination. 

Inexplicit terms of reference

When terms of reference do not determine the evaluation scope, the analysis of the objectives or the effects should cover the range of activities, whatever the implementing institution and the financial tool implemented.

STAGE 2: HOW TO IDENTIFY THE OBJECTIVES AND EFFECTS

Collect the documentation

As a first step, the documentation needed for the construction of the diagram is collected in the relevant Unit of DG RELEX or DG DEV. It comprises:

  • Policies, strategies and programmes baseline documentation
  • Complementary official documentation

The documentation varies with the type of evaluation (geographical, sector-based and thematic).

Record the references to the objectives in the baseline documentation

The task should begin with a careful analysis of the baseline documentation, to assemble a record of all references (quotations, sources) to the objective (understood as the target of the European Commission's assistance) and any explicit mention of the links between objectives.

The documentation seldom uses the term "objectives" and only introduces the term in expressions such as "with a view to", "in order to", "so as to", etc. Objectives can be called "priorities", "fields of intervention", or sometimes "focal sectors".

A precise table of such terminology must be established.

Distinguish strategic from background objectives

Les citations relevées dans les documents de référence sont traduites en objectifs. Chacun des objectifs est assorti d'un numéro d'identification renvoyant aux citations.
The quotations taken from the reference documentation are converted into objectives and each is given a unique identification number.

The documentation identifies the objectives which should be targeted by the assistance. The remaining objectives are treated as background items.

In an annex to the reports (how are the findings presented?), the evaluation team should present the list of all the objectives, marshalled into two categories: strategic and background objectives.

STAGE 3: HOW TO CONSTRUCT A FAITHFULLY RECONSTRUCTED OBJECTIVES DIAGRAM

Classify the objectives by rows

Several situations can be encountered:

  • The row of the objective is explicit. This is often the case for the objective of the first row, also called the overall or long-term objective.
  • The cause-and-effect link between two objectives is explicit, which facilitates the understanding of the interconnections between them.
  • No indication is given about the objective's row (per se or in relation to others). The evaluator should refer to additional documentation in order to find an indication of a cause-and-effect link; if this task fails, it should be deduced and/or the evaluator should ask the assistance of experts (human resources and working arrangement).

Each of the classifications by row should be explained by means of an interpretation expressed as an assumption.

A provisional classification could be carried out on the basis of the distinction between three levels of objectives:

  • Overall objectives
  • Short-term and medium-term intermediate objectives
  • Operational objectives

The definition of the logical relationship is paramount in the classification. This task is a question of experience, because the decision of the evaluation team that two events are logically connected depends on the judgement of experts in the field under study, and managers responsible for the implementation of the strategy and policies.

Establish a temporary diagram

The first faithfully reconstructed objectives diagram can be drawn from this first classification. It usually reveals incoherencies that will have to be corrected in accordance with the evaluator's interpretation (these corrections must be indicated in the diagram).

Should the construction of the diagram start with the overall objective
or the operational objectives (objectives describing the attended results)?

Usage has shown that these two types of objectives are more easily identified than intermediate objectives, which are also more difficult to rank. Thus, it is recommended that the roots of the diagram and the extremities of the branches are developed concurrently.

In the diagram, each objective is presented as a box with a heading. The evaluator should also indicate the identification number which refers to the quotation list. Providing the boxes with colours corresponding to the level to which the objective belongs may also be helpful.

Each link should refer to one or more quotations (in accordance with the identification number).

The diagram should also mention the specific means (financial and non financial) used in the implementation of each operational objective.

Identify the authors of the objectives' wording

Before testing the temporary diagram, the evaluator should identify the main and secondary authors of the objectives' wording (writing of a chapter, sector-based contributions, participation in working meetings, etc.).

It may be useful to check the consequence of the wording on the definition of the operational objectives of the programme planning.

Test the temporary diagram

The authors of baseline documentation and additional documents should test the diagram in order to validate the classification of the objectives by rows and links. The aim is to check that the evaluator's interpretations accurately reflect the authors' intentions. If the authors are not available or, with a view to complement their contribution, the evaluator can seek the participation of other actors responsible for the drafting process (both from the writing and the discussion process).

Contacting the authors is usually possible when the documentation is recent and the authors are still in position or contactable in one of the services. This task is challenging when the documentation is old and its authors are not easily identifiable, nor reachable.

At the end of the interviews, the evaluator may also encounter difficulties evaluating what was actually done during the drafting process and the justifications of respondents for the decisions taken.

Establish the final version of the faithfully reconstructed objectives diagram

This version takes into account the opinions collected during the test of the temporary diagram.

It should be considered an accurate report of the initial intentions of the European Commission, taken from the official documentation.

STAGE 4: HOW TO CONVERT THE FAITHFULLY RECONSTRUCTED OBJECTIVES DIAGRAM INTO A LOGICALLY RECONSTRUCTED OBJECTIVES DIAGRAM

Advantages and limitations of a faithfully reconstructed objectives diagram

Advantages of the faithfully reconstructed objectives diagram

Some faithfully reconstructed objectives diagrams display a strong internal coherence: apart from the overall objective or the overall expect impact, each objective 'x' leads to an higher (immediate or not) objective and is supported by one or more subordinate objectives. This is particularly the case when faithfully reconstructed objectives diagrams are extracted from a carefully established logical framework. Faithfully reconstructed objectives diagrams can be deemed as completely logical and stand as logically reconstructed objectives diagrams.

Logical links of a given objective 'x' (in objectives diagrams)

The establishment of a logically reconstructed objectives diagram is not required when the objectives system of the strategy and planning documents is thoroughly and consistently presented.

Limitations of the faithfully reconstructed objectives diagram

Some faithfully reconstructed objectives diagrams may reveal logical defects in strategy or political papers, such as:

  • No definition of the overall objective
  • Weak relevance of the causal links between objectives
  • The objectives of a given row do not result in any objectives or are not linked to subordinate objectives which would have provided information about the implementation process
  • Several unclassified objectives

In these cases, the definition of the evaluation thematic scope and the evaluation questions is difficult. The team should therefore draft a logically reconstructed objectives diagram.

Preparation of the logically reconstructed objectives diagram

In order to prepare a comprehensive and coherent objectives diagram, the evaluation team will need all available documentation, its own expertise and, if required, that of experts. This task should be completely transparent (how are the findings presented?): whenever possible, the diagram should display the reclassification of objectives, the changes affecting links between objectives, and the introduction of missing intermediate objectives. Each of these rationalisation operations should be explained in a technical note.

Stage 5: how to construct an effect diagram

Objectives diagrams and intended effect diagrams share the same rules of construction. 

The effect diagram is constructed from the conversion of each of the objectives into the corresponding intended effect.

 

.

HOW ARE THE DIAGRAMS USED?

 

Objectives diagrams and effect diagrams are a crucial tool in the organisation stage of the evaluation. They play many roles:

  • The objectives provide a description of the strategy (geographic) or the policies (sector-based, thematic)
  • The diagrams determine themes for evaluation questions
  • They present the strengths and weaknesses of the objectives system to be achieved or the intended effects system in an informative and transparent way

Description and analysis of the strategy

Objectives diagrams demonstrate a synthesis of the geographical strategy displayed in the official documentation. When the evaluation scope covers one or more strategy papers (geographical) or strategic policies (sector-based, thematic), it is recommended that one diagram per document is created (unless there is a logical continuity in the strategy or the policy). 

They illustrate the objectives/effects classification, from the overall objective/impact to projects and programmes scheduled to implement the objective/effects. The logically reconstructed objectives diagramand the effect diagram are informative means to reflect upon:

  • Its internal coherence
  • The relevance of the objectives/effects to the context (the European Union's overall strategy for country assistance, the main actors' themes or sectors and objectives, the activity's scope and objectives of other donors)
  • The appropriateness of the means dedicated to carry out these objectives/effects

The faithfully reconstructed objectives diagram displays the objectives system and provides the evaluator with a first approach to the strategy and policies inner quality. Indeed, an unclear, incomplete and incoherent diagram means a lack of relevance in the forthcoming planning or a lack of faithfulness to the initial objectives system. 

These analyses must be presented in the notes and reports of the evaluation organisation stage. They must also appear as an abstract in the final report.

Definition of the themes for evaluation questions

The previous analyses reveal questions about:

  • The faithfulness of the strategy or the policies to the European Union's overall objectives for its country assistance
  • The relevance of the strategy and policies to the country's situation, the evaluation's sector or theme, and the main actors' policies (governments in particular)
  • The need for complementarities and synergies with other donors
  • The consequences of a lack of coherence in the objectives system
  • The relevance of the means to the objectives

These questions result in the determination of a range of themes which could be investigated during the following stages of the evaluation, particularly through evaluation questions. 

However, evaluation questions cannot be automatically deduced from these analyses. Other questions can emerge during the organisation stage, formulated by the main strategy implementation operators (since they are the most well-informed actors about the problems encountered during the planning and the implementation stages). However, the number of evaluation questions is limited and additional questions would have to be the subject of a selection process.

HOW ARE THE FINDINGS PRESENTED?

Notes and reports of the organisation stage

Objectives diagrams are established during the organisation stage, where reports and notes should be provided. At this stage, the diagram's construction must be precisely described. 

For the faithfully reconstructed objectives diagram, the sources of the objectives/effects (quotations, references to the original documentation) must be provided. References to documentation, interviews and expertise must support the objectives' location in the diagram, and the assumptions developed during the construction of the diagram must be explained. 

The process through which the logically reconstructed objectives diagram has been extracted from the faithfully reconstructed objectives diagram must also be clearly explained.

Final report

Diagrams should be incorporated into the final report because they are not only stages in the development of the wording of evaluation questions, but also an illustration of a possible analysis of the evaluation strategy. 

The report may include a short presentation of the diagrams, in addition to a full explanation included in the annexes.

Verbal presentations

The evaluation team will need to present its work (methodology and findings) to different types of people (the evaluation reference group, participants in the debriefing session). The objectives diagram or the effect diagram are very efficient tools for this purpose, providing that they are readable without being over-simplistic. 

To do so, a main diagram, and several sub-diagrams developing fundamental sections of the main diagram, should be presented, each of them not exceeding 20 items.

WHAT ARE THE PRECONDITIONS FOR THE ESTABLISHMENT OF AN OBJECTIVES DIAGRAM AND AN EFFECT DIAGRAM?

 

Human resources and working arrangement

Type of work required for the design of an objectives diagram

Observation :
The number of working days mainly depends on the scope of the evaluation, and thus on the number of programmes to study and diagrams to construct.

 

Travelling expenses

Strategy papers have been drafted under the responsibility of the head office (DG RELEX or DG DEV). The head office is also responsible for the planning stage. EuropeAid is in charge of the project drafting. This implies that the majority of the useful documentation (the whole baseline documentation) can be found in Brussels. This situation may devolve to Delegations in the future whose responsibilities in this area will grow.

The current needs (2004) in travelling expenses to Brussels

Categories of professionals journeys per diem
Juniors / intermediates 2 3
Experienced evaluators 2 4

These elements (and the corresponding budgetary components) will have to be reviewed when the Delegations take broader responsibilities for strategy development (2007).

Computer devices

Most of the graphic problems can be addressed with software such as MS PowerPoint

IN COUNTRY EVALUATIONS

.

IS THE USE OF AN OBJECTIVES DIAGRAM AND AN EFFECT DIAGRAM COMPULSORY?

Every donor must implicitly or explicitly explain the decision to provide assistance to a country or a region in a given timeframe. For this purpose, the objectives to be achieved or the intended effects which form the basis for the assistance are described in documents which are regularly published and present the overall objectives and priorities. Such documentation can either display orientations, an individual programme, or both.

Such documents may present programmes which are jointly drafted by the donor and beneficiary. In this case, the interventions under study should be that of the European Commission and of the country partner. The outcomes and intended impacts are the outputs of the implementation of these two types of interventions.

In national programmes

National Indicative Programmes (NIP) can be inserted in the logical framework which can be a single document or correspond to each main sector of intervention. In this case, the evaluation uses the logical frameworks (i.e. objectives and implementation indicators) as the reference for efficiency criteria. Its goal will be to check in what extent the objectives and effect systems have been completed during the implementation of the programme.

In country strategies

Country Strategy Papers (CSP) do not usually rely on an overall logical framework because they cover a long timeframe and are implemented through various successive programmes, each of them supported by a logical framework. Consequently, the evaluators should assemble the objectives and effects system, and draft a diagram for each programme.

WHICH OBJECTIVES AND EFFECTS MUST BE TAKEN INTO ACCOUNT?

In strategies and complex bilateral assistance programmes, the European Commission is the donor and the States and national public institutions are usually its partners. The European Commission and the States have their own overall objectives, corresponding to their own issues. Yet, to a certain extent and degree of detail, the objectives overlap.

In a bilateral assistance contract, three objectives or effect systems can be developed:

  • The European Commission's own system, corresponding to strategic issues meant to be resolved with financial assistance (for example, regional instability, weak absorption capacity of the local market, uncontrolled immigration)
  • The country partner's own system, relating to major political issues to which the State is confronted (macro-economic unbalances, social instability, weak growth, social or geographical disparities, etc.)
  • A common system for the European Commission and its country partner, where similar issues are relevant for both of them and to the fields of the assistance scope (for example, health, SME, irrigation, environment)

Except for participatory evaluations, the evaluation team will prioritise the first and third objectives system.

WHAT ROLE DO OBJECTIVES DIAGRAM AND EFFECT DIAGRAMS PLAY IN COUNTRY EVALUATIONS?

Once completed, the logically reconstructed objectives diagram and the effect diagram provide two parallel approaches to the country strategy evaluation:

  • The wording of evaluation questions, which mainly deals with the effectiveness and the effect of programme implementation and indirectly with the efficiency and sustainability of the outputs and their effects
  • The internal coherence of the strategy and its relevance relating to the European Union's overall objectives, State policies and the positions taken by international donors

These approaches are illustrated in the following table. The main branch of the logically reconstructed objectives diagram splits into two: the right branch deals with evaluation questions, the left branch with the overall analysis of the strategy.

Role of the objectives diagram in the conduct of an evaluation structured by evaluation questions

 

Footnotes

Two main aspects of relevance are examined:

  • (a) in relation to the overall objectives of the European Union and the European Commission
  • (b) in relation to the needs and /or demands of the beneficiary(ies)

The evaluation questions connect the intended effects (results, outcomes, secondary impacts) with the specific and general means which are used to achieve the objectives.

The overall analysis studies the strategy building, i.e.:

  • The choice of the intended impact
  • The choice of results, outcomes, and secondary impact
  • Their relevance regarding the country situation and the main actors' policies
  • The efforts to gain complementarity and synergy with other donors

WHICH TYPES OF OBJECTIVES DIAGRAMS OR EFFECT DIAGRAMS CAN BE USED FOR COUNTRY EVALUATIONS?

Two types of diagram, setting the framework for the evaluation, illustrate the objectives of the programme or the strategy to be assessed.

The faithfully reconstructed objectives diagram

The faithfully reconstructed objectives diagram is useful for two reasons:

  • It accurately reproduces the objectives system of the logical framework, when it exists. It also highlights, as closely as possible, the objectives system which is more or less explicit in the documentation without a logical framework. Thus, it represents a crucial stage in the development of the logically reconstructed objectives diagram.
  • It underlines, when required, the limitation of the objectives system's coherence (see the limitation of the faithfully reconstructed objectives diagram). It fills the gap in ex ante evaluation, and thus appears as an information tool, benefiting the authors of strategy and planning papers.
  • It is a schematic illustration of the declarations included in the strategy (CSP, Regional Strategy Papers, RSP) and planning (NIP, RIP) papers relative to the targeted objectives and the various links between them.

The logically reconstructed objectives diagram

The logically reconstructed objectives diagram determines the thematic scope of the evaluation.

It establishes the reconstructed classification of the objectives system as they are presented in logical frameworks or reconstituted in the faithfully reconstructed objectives diagram. It is also the logical reconstitution of the objectives' classification implied in strategy papers (CSP, RSP) and planning papers (NIP, RIP).

It contributes to the assessment of strengths and weaknesses on which the evaluation questions should focus:

  • Relevance of the objectives' selection
  • Quality of the wording
  • Logical links within the hierarchy of objectives

The logically reconstructed objectives diagram is particularly needed because the strategy papers and documentation concerning the planning of geographic assistance do not provide the evaluator with a systematic presentation (with or without a diagram) of the objectives system.

It should be summarised within the final report of the evaluation, and fully presented in the annexes.

Additional observation

Diagrams are not the only tools available to analyse objectives and effects. For example, evaluators frequently use a matrix.

HOW MANY EFFECT DIAGRAMS MUST BE ESTABLISHED?

A single diagram

If the analysis of the situation does not radically evolve during the evaluation and if the activities focus on a single intended overall impact, a single diagram is enough.

Many diagrams, depending on the context

When the strategy evolves, effect diagrams should be constructed for each change in strategy of the European Commission.

When the evaluator cannot determine a single overall impact and when each of the global impact depends on an independent effect system, the evaluator should establish as many diagrams as overall impacts.

Intermediate diagrams

Intermediate diagrams, corresponding to each of the assistance priorities (or sectors) for the period(s) covered by the evaluation, can be useful for the construction of an objectives diagram or effect diagram. They are particularly adapted for programmes with a logical framework provided for each of the priorities.

Intermediate diagrams can be useful for the determination and justification of evaluation questions.

Magnifier diagrams

When overall diagrams are very complex, their readability and information quality may suffer. A simplified overall diagram should be prepared, and detail diagrams should be drawn from the overall diagrams, in order to emphasize some of their specific parts.

WHICH DOCUMENTATION SHOULD BE REFERRED TO IN COUNTRY EVALUATIONS?

Baseline documentation

The baseline documentation includes strategy papers (CSP) and the related indicative programmes (NIP) relevant to the period covered in the evaluation scope.

Bilateral agreements, such as the Association Agreement, Cooperation Agreement and their monitoring documents should also be included, as well as multilateral agreements (the Lome and Cotonou agreements) and their related documents (for example, the Barcelona Declaration).

In most cases, they should be provided to the evaluator at the start of the evaluation.

However, the evaluator should check that:

  • All the successive programmes, including the modifications afterwards, are provided
  • The latest version provided

Additional documentation

Three types of documents must be available :

  • Strategy and planning papers related to European Commission activities at the regional level (RSP and RIP) in the case of a country evaluation
  • Official documentation concerning other non-geographic budgetary resources at the disposal of the European Commission's geographic assistance (such as 'horizontal' budget lines and other contributions)
  • Specific planning papers dealing with the main areas of the assistance ('concentration' or 'assimilated' sectors)

Other informative documentation

Other documentation may be useful, such as successive versions of the baseline documentation (the date of the final version must be checked), including the various contributions and reactions of the Commission's services, their national partners and other donors (particularly Member States). It is valuable, but often difficult to identify and seldom available or accessible.

.

EXAMPLES

.

BIBLIOGRAPHY

.

The logical framework and the objectives tree

  • Manuel de Gestion du cycle du projet, Commission Européenne, EuropeAid, mars 2001. [FR]
  • Guide récapitulatif des Formations - Gestion du Cycle de Projet, Commission Européenne, EuropeAid, Itad Ltd, Particip GmbH. version 1.1, février 2001. [FR]
  • 'Logical framework analysis', BOND Guidance Notes No 4, March 2003.
  • 'The Logical Framework Approach (LFA)', A summary of the theory behind the LFA method, Sida - Swedish International Development Co-operation Agency, Draft, June 2002.
  • 'The Logical Framework Approach', Handbook for Objectives-oriented Planning, NORAD, 4th edition, January 1999.
  • 'The logical framework approach', Australian agency for international aid (AusAid), AusGuide, AusGuidelines, 2000.
  • 'Logical frameworks: a critical assessment', Working paper # 264, Des Gasper, Institute of Social Studies, the Hague, December 1997.

Use of the logical framework in evaluations

  • 'Project appraisal 2 (4)', Logical Framework Approach to the Monitoring and Evaluation of Agricultural and Rural Development Projects, Coleman G., 1987, p. 251-259.

The other approaches

  • Program Theory in Evaluation: Challenges and Opportunities, New Directions for Evaluation, a publication of the American Evaluation Association, Fall 2000.
Evaluation methodological approach > EN: Methodological bases and approach > Evaluation tools > Objectives diagram and the effect diagram > Check lists

Check lists

CHECK-LIST FOR EVALUATORS

.

Questions Answer
Preparation and design  
Has the preliminary analysis of the strategies under evaluation been undertaken?  
Has the preliminary analysis of the institutions participating in the preparation and implementation of the strategy and/or the programmes been undertaken?  
Has the list of the relevant documents been established?  
Has the list been submitted to the group in charge of the monitoring of the evaluation?  
Has the dating of the documents been confirmed by their authors or contributors?  
Implementation  
Has a cross-reading of the documentation been conducted?  
Have the missing elements been sought (?) during the test?  
Are hypotheses and uncertainties about the objectives' links clearly stated?  
Did their authors and/or contributors confirm this classification during the test?  
Was there a triangulation of the perspectives?  
Have specialists been consulted by means of written exchanges, if necessary?  

.

CHECK-LIST FOR MANAGERS

.

Questions Answer
Are documentary references and quotations provided in the report?  
Is the list of the documents consulted to establish the diagram complete?  
Do the diagrams appear coherent?  
Are the relations between objectives and effects explicit?  
Are hypotheses and uncertainties about the objectives' links clearly stated?  
Have the external effects been identified?  
Are the constraints and the risks exposed?  
Evaluation methodological approach > EN: Methodological bases and approach > Evaluation tools > Decision diagram

Decision diagram

SUMMARY

.

Why is this tool used in evaluation?

Strategies result from a procedure, taking into account global objectives, contextual elements as well as the position of the stakeholders. This procedure is usually empirical. The decision diagram sketches the strategy drafting process, particularly for the selection of key information, the participation of stakeholders in the process, and the management of the implementation arrangements. It highlights the choices that have been made when the strategy was elaborated as well as the consequences of the selected objectives and their forecast impact. The decision diagram is a useful complement to the objectives diagram

The documents which present co-operation strategies usually display a range of objectives among which one or occasionally more global objectives, operational objectives relative to development assistance projects, and a range of intermediary objectives at various levels can be identified. The selection carried out by the authors of the strategic and programming documents depends on various sources:

  • Recommendations of the European institutions (Council, Parliament, Commission)
  • Analyses of the external events (major events, country situation)
  • Intervention of non-European Commission actors (partner governments, Member States, other donors)
  • Lessons learned in previous programmes or projects

The aim of the decision diagram is to describe the impacts of such orientations, contextual data and analyses. Indeed, each box of the diagram's central column represents a choice (selected and rejected objectives), while the boxes on each side illustrate the flow of inputs which represents the external justification for these choices. 

What are the possible uses of these diagrams?

The decision diagram highlights:

  • The range of options for the establishment of the objectives system (overall objective, selected and rejected intermediary objectives, etc.)
  • The external events influencing the decision-making

The decision diagram facilitates the analysis of the strategy in terms of internal coherence (logical succession of the choices) and external relevance (contextual elements and position of the stakeholders).

When the terms of reference of an evaluation require an analysis of the partnership, the diagram is used to highlight the intervention of the main partners (governments, Member States and other donors) in the strategy design, the establishment of the programmes and the selection of the projects.

The diagram can perform the same role for the analysis of the 3 Cs (Coherence, Co-ordination, Complementarity).

How is the decision diagram constructed?

Figure 1 - Steps involved in the decision diagram's construction

 

The drafting process of the decision diagram continues in two steps:

  • Determination of the points at which the decision-making took place (selection of the objectives)
  • Identifying, collecting and analysing of the relevant information

What are the preparation stages for the construction of the diagrams?

Stage 1: determination of the points when the decisions were made

Figure 2 - Sketch of the moment of the decision-making: example of first row intermediary objectives

 

Usually, the establishment of the decision diagram follows the construction of the objectives diagram. The objectives diagram is the basic tool for identifying the points at which the decision-making takes place. Apart from the definition of the overall objective, each intersection of the diagram represents a decision-making point. It stimulates the following questions: Why have these objectives been selected? Why have others been rejected?

Stage 2: drafting of questions

The questions focus on the justification of the selection of objectives. The identification of the rejected objectives helps the accuracy of the wording of questions. How are these objectives identified? At each decision-making point, the evaluator may encounter four situations:

  • Strategy and planning papers explain why certain objectives have been rejected and identify reasons to support the decision.
  • Working papers (interim versions, notes and various correspondences) provide elements to pinpoint rejected objectives and may justify the choices made (in principle, they always include the positions of Member States and the response of the European Commission).
  • The documentation available does not explain the choices made, but interviews with the decision-makers provide relevant information.
  • The evaluators have not collected any written or verbal information relevant to the explanation of the choices made.

In the first three situations, the evaluators should investigate which of the objectives were planned but eventually rejected, and the reasons for rejection. In the last situation, would-be objectives should be identified.

Stage 3: collection of useful information

Before the construction of the diagram, the quality of the sources of information should be checked. If this verification reveals insufficient sources, the construction of the diagram should be abandoned.

Usually, written information can be found in four types of documents:

  • Strategy papers and programmes can provide information about the lessons learned from previous implementations, the political, economic and social context, and the interventions of Member States and the main donors.
  • The preparatory documentation (meeting reports, notes at the end of preparatory missions, correspondence, internal notes or notes between services) may provide explanations about the priorities chosen.
  • Evaluations can be valuable in terms of lessons learned.
  • The European Council and Commission's more general documents (conclusions, recommendations, reports), as well as the international treaties and agreements often display the contextual elements, lessons learned and priorities which are known to the designers of strategies and programmes.

The evaluation team should formulate its conclusions about the quality of the sources in terms of quantity, relevance, reliability and accessibility. This judgement is presented to the managers, who take the final decision.

Stage 4: constructing the temporary diagram

Preliminary selection of the relevant information is carried out on the basis of the objectives diagram. It requires:

  • The establishment of one or more timelines, describing the successive events and/or information included in the drafting process of the strategy and the programme
  • The selection of texts from the documentation collected, relative to the choice of the objectives and the scheduled assistance process, or relative to the factors influencing such choices
  • The classification and the construction of an index for the texts

Provisional answers to the evaluation questions are formulated on the basis of the information collected. Some of the questions may not be answered at this stage.

The objectives diagram supports the decisions chain. It identifies four (sometimes five) decision-making points dealing with:

  • The global objective
  • The first row intermediary objectives
  • The secondary row intermediary objectives
  • Operational objectives

The drafting of strategies and programmes is not strictly and exclusively driven by such a rationale. Implicitly or explicitly, the designers of the strategy and programme begin with an overall objective. Thereafter, they examine which means at the disposal of the European Commission are able to achieve this objective. The successive choices can be synthesised in two points:

  • The choice of the global objective
  • The choice of the fields and the intervention processes

At the left of the sketch, a magnifier explains the decision's outcome:

  • Selected objectives
  • Rejected objectives.

A global fishbone shaped diagram is thus completed through detailed diagrams corresponding to each decision-making point.

Figure 3: Decision diagram of the Commission's strategy drafting in Tanzania 2000-2007

Stage 5: testing of the temporary diagram

The decisions and their explanation must be confirmed by the main actors responsible for the drafting of the strategy and the programming, including the European Commission's services (head office and delegations), the representatives of the other stakeholders (Member States, NGOs, etc.), the usual interlocutors in beneficiary countries and/or their government.

Stage 6: constructing the final diagram

The process of testing of the temporary diagram may question some of its parts when the justifications do not illustrate the real strategic and programming drafting process.

In this case, the information should be reviewed and augmented by another consultation round. The final and temporary diagrams have the same shape (a main diagram and the sketch of the point of decision-making). The final diagram includes an explanatory table about the analysis of the information collected.

What are the preconditions for the diagrams use?

Figure 4: the preconditions for its use
The time span Identification, collection of the documentation, examination of the documentation, construction of the diagrams: 3 to 8 working days
Data analysis: 3 to 8 working days
Tests : 3 to 10 working days.
Human resources Knowledge of the European Commission's strategies and development programme procedures
Fair knowledge of computer tools
Logical process of thinking
Experience in the fields covered by the strategies and programmes
Specific knowledge of the country, sector or theme under study
Financial resources A budget of at least €5,000 should be planned.

What are the advantages and limitations of the tool?

Figure 5: The advantages and limitations of the tool
Advantages Clarification of the strategy

The diagram highlights the range of options for the establishment of the objectives system (global objective, selected and rejected intermediary objectives, etc.) and the external events influencing the decision-making. It facilitates the analysis of the strategy in terms of internal coherence and external relevance.

When the terms of reference of an evaluation require an analysis of the partnership, the diagram is used to highlight the intervention of the main partners (governments, Member States and other donors) in the strategy design, the establishment of the programmes and the selection of the projects. The diagram can perform the same role for the analysis of the 3 Cs (Coherence, Co-ordination, Complemen-tarity).

Limitations The major limitations in the use of the diagram are:
 

  • the availability of information
  • uncertainties about causal links

Direct information about the factors influencing the strategic and programming drafting process is sometimes scarce in the official papers. The restricted documentation (studies, notes, correspondences) is in principle more informative, but cannot be consulted, or its access is difficult.

An interpretation is sometimes required, which can be the source of a risk of error, particularly in cases where several causes support the decision-making.

Evaluation methodological approach > EN: Methodological bases and approach > Evaluation tools > Decision diagram > Detailed presentation

Detailed presentation

.

This section is structured as follows:

.

WHAT IS A DECISION DIAGRAM? 

.

WHAT DOES THE DECISION DIAGRAM REPRESENT?

General definition

The decision diagram shows the process during which the strategic objectives and the overall policies of cooperation with developing countries, which are defined by the European Union's assistance agreements, are converted into short-term and medium-term bilateral co-operation decisions.

It illustrates the successive official and informal decisions, resulting in the programming of development assistance.

Decision factors

Two types of decision factors can be found:

  • Factors included in the decision process, each selected objective justifying the objectives at a lower level
  • External factors influencing the decision-making

External factors appear during the decision-making process as a flow of information originating from the European Union's institutions, or other sources (strategic trends, context of the country under consideration, governmental policies, position of the Member States and other donors, etc.). They are called inputs.

WHERE DOES THE DECISION DIAGRAM COME FROM?

Various fields of application

Decision-making is an important theme in management theory. Various tools are proposed for the determination of public policies (such as in the field of defence), medical and managerial decisions. They usually lead to a series of choices for alternative solutions which have to be optimised by rationalisation tools, such as cost-effectiveness analysis.

In the analysis of the strategic decision-making process, the retrospective use of such a tool was suggested for the first time (in a simplified version) for the evaluation of the European Commission's strategic co-operation with Egypt. The tool, however, was not exploited fully during this evaluation.

Reasons for its use

In the management field, the role of the decision diagram is to rationalise the decision-making process and to make it more efficient for the resolution of problems and the achievement of objectives. It provides an answer to the following question:

Which elements must be taken into account to make the most appropriate operational
decision, with regard to the intended objectives and the available means?

Decision diagrams can be supported by a variety of software (computer devices), providing the user with guidance throughout the decision-making process. Thus, decisions diagrams are, by nature, a normative and simplifying tool.

The evaluator needs a tool which describes a complex approach with undefined rationale and several stakeholders with different aims. Such approaches are more focused on consensus than efficiency, and do not necessarily rely on any rationalisation of decision-making.

Can the diagram be combined with other tools?

The decision diagram can be directly combined with the objectives diagram and effect diagram (and indirectly with the problem diagram). It was originally designed to overcome the deficiencies of the objectives diagram, but it is also useful in itself.

Its construction can be supported by a SWOT analysis and/or a socioanthropological analysis, in which information about the context of the societies affected by the strategy is provided.

WHAT SHAPE CAN IT TAKE?

The fishbone shape can be used to illustrate the three main components included in decision-making. Thus, a theoretical decision diagram for the European Commission's country development assistance will present:

  • In the central axis of the diagram, the planning chain of planning decisions which goes from the most general orientations to field decisions, including medium-term strategic and political decisions, and operational decisions
  • On the left side of the diagram, the flow of inputs originating from the European Union's institutions (Council, Parliament, Commission), such as decisions, positions, communications, directives, lessons learned by the implementation of previous strategies and programmes, review of projects in process or in the portfolio
  • On the right side of the diagram, the flow of other inputs, such as events or major turning points in the world situation or a specific region, evolution and need of the country partners, and the position and interventions of other donors

Standard decision diagram (fishbone shape) 

 

The diagram illustrates the decision-making stages and the main external flow of information supporting the drafting of the strategic or political decision.

.

WHY AND WHEN SHOULD THE DECISION DIAGRAM BE USED?

.

WHEN SHOULD THE DECISION DIAGRAM BE USED?

During the design of the strategies

The European Commission's external intervention strategies result from a complex procedure, taking into account the contextual elements as well as the position of the stakeholders.

This procedure is usually empirical and lacks transparency, although more rationalisation and explanation may lead to more appropriate decisions (selection of priorities, adoption of programmes). These decisions could have a positive influence on the country partners' ownership of the strategies.

Using a decision-making tool such as the decision diagram could facilitate the organisation of the design process, particularly for the selection of key information, the participation of stakeholders in the process, and the management of the implementation arrangements.

This methodology, however, focuses on the tool's usage in the particular context of the evaluation (during the organisation stage). Its use in the strategy design stage requires an adaptation of the methodology.

During the evaluation of the strategies

Illustration of justified choices

Co-operation strategies and intervention policies designed by national and multinational donors have explicit and implicit objectives. The documents which present them usually display a range of objectives more or less in order (see the complex objectives diagram), and among which one or occasionally more overall objectives, operational objectives relative to development assistance projects, and a range of intermediary objectives at various levels can be identified.

Thus, the operational items of a programme (planned projects and/or in process) which depend on the strategic or political papers, are supported by the objectives system given in these papers.

In an objectives diagram, the selection of a higher-order objective and its division into several intermediary objectives, down to the operational objectives often results from non-explicit choices.

Unless decisions are deemed arbitrary, the selection carried out by the authors of the strategic and programming documents depends on various sources:

  • Recommendations of the European institutions (Council, Parliament, Commission)
  • Analyses of the external events (major events, country situation)
  • Intervention of non-European Commission actors (partner governments, Member States, other donors)
  • Lessons learned in previous programmes or projects

The decision diagram illustrates the impact of these information flows on the successive choices (i.e. the progressive elimination of alternative options) made by the decision-makers. Indeed, each box of the diagram's central column represents a choice (selected and rejected objectives), while the boxes on each side illustrate the flow of inputs which represents the external justification for these choices.

A strategic sequence and an operational one in the decision

Strategy papers are established for each 7 year budgetary period. They deal with the events, orientations and changes having a significant and long-lasting impact, which can be worldwide (such as the attack on the World Trade Centre in 9/11/2001) or regional (the second Intifada in 2000, or the military invasion of Iraq in 2003). If such events happen during the strategic period, strategy papers can be revised (for example, in the mid-term revision). However, these documents usually remain unchanged until the end of their term.

Programme durations are sometimes shorter than strategy papers (usually 2 to 3 years). They can be affected by substantial changes in the political orientations of the partner government, natural disasters (typhoons, droughts), the signature of association or co-operation agreements, political shifts in the main donors' activities, etc. The revision of existing programmes or the drafting of new programmes is common in the programming activity during the budgetary period.

In addition to a short presentation of the whole decision-making process (in the form of a summary diagram), the evaluator should divide the theoretical diagram (standard decision diagram) in two. This should result in a strategic decision diagram on one hand, and an operational decision diagram on the other hand.

Standard strategic decision diagram

 

 

Standard operational decision diagram

 

Such a separation is interesting because it limits the number of boxes in each diagram and improves their readability.

Evaluation of the decision process

The design of the strategy and programming papers is theoretically collaborative. These documents are the subject of negotiations between the European Commission and the partner governments of the ACP regions.

In other regions, programmes are the subject of an agreement between partners, whereas the communication of strategies is for information only.

Member States must be consulted on all strategy and programming documents. Practice shows, however, that this collaboration is sometimes only formal.

As a consequence, the European Commission insists on an in-depth examination of the decision procedure which could be part of a country evaluation, or stand as a thematic evaluation on its own. In both cases, the decision diagram is a very relevant tool.

WHAT ARE ITS ADVANTAGES AND LIMITATIONS?

Advantages

Clarification of the strategy

The diagram highlights:

  • The range of options for the establishment of the objectives system (overall objective, selected and rejected intermediary objectives, etc.)
  • The external events influencing the decision-making

Complementing the objectives diagram and effect diagram, the decision diagram facilitates the analysis of the strategy in terms of internal coherence (logical succession of the choices) and external relevance (contextual elements and position of the stakeholders). 

Illustration of the main partners' interventions and the analysis of the 3 Cs

When the terms of reference of an evaluation require an analysis of the partnership, the diagram is used to highlight the intervention of the main partners (governments, Member States and other donors) in the strategy design, the establishment of the programmes and the selection of the projects.

The diagram can perform the same role for the analysis of the 3 Cs (Coherence, Co-ordination, Complementarity).

Limitations

The major limitations in the use of the diagram are:

  • The availability of information (stage 3)
  • Uncertainties about causal links

 

Baseline information

Direct information about the factors influencing the strategic and programming drafting process is sometimes scarce in the official papers which present the strategies and programmes. Other public documents, such as evaluations, can provide indications but the information they provide is limited. The restricted documentation (studies, notes, correspondences) is in principle more informative, but often cannot be consulted, or its access is difficult.

Verbal information has the same limitations concerning availability and accessibility. 

Causal interpretation

Even when potentially informative elements are provided (for example, the intervention of other donors, or the priorities of the government), causal relations between decisions are not always explicit. An interpretation is sometimes required, which can be the source of a risk of error, particularly in cases where several causes support the decision-making.

WHAT ARE THE PRECONDITIONS FOR ITS USE?

Existence of an objectives diagram

The presence of an objectives diagram in strategy or programming papers is a precondition for the construction of a decision diagram. If such is not the case, the evaluator will need to construct an objectives diagram.

Existence and relevance of the information

The availability (effective or to be assured) of reliable written or verbal information (stage 3) is also an important precondition. This information needs to be sufficient and allow verification.

Tight collaboration with the European Commission's services

A trustworthy collaboration between the evaluators and the European Commission's services (competent services of the DG Development, DG External Relations, and Delegations) would guarantee the access to the information and its interpretation.

.

WHAT ARE THE DECISION DIAGRAM'S CONSTRUCTION STAGES?

.

Once the preconditions are fulfilled, the drafting process of the decision diagram continues in two steps:

  • Determination of the points at which the decision-making took place (selection of the objectives)
  • Identifying, collecting and analysing of the relevant information

The process cannot start with the collection of documentation because useful information cannot be found in one or more identified documents. It has to begin with the definition of the scope of such a collection.

The determination of the objectives at various levels (overall, intermediate and operational) implies a series of choices which can be translated into a question: why such an objective and not another one?

The question's wording therefore orients the collection of the information judged to bring answers. The relevance of these answers is checked, and may result in seeking new information (written or verbal), and even rewording some of the questions, particularly those concerning the objectives which are rejected.

STAGE 1: DETERMINATION OF THE POINTS WHEN THE DECISIONS WERE MADE

General principle

The decision diagram (existing or to be constructed) identifies the points at which the implicit or explicit selection of the objectives were made.

Whilst the objectives diagram assesses the internal coherence of the objectives system, the decision diagram illustrates the external factors corresponding to each choice in the decision-making and resulting in the selection of one objective over another.

Determination of the period allocated for the selection of objectives

Usually, the establishment of the decision diagram follows the construction of the objectives diagram. This sequence should be respected.

The objectives diagram is the basic tool for identifying the points at which the decision-making takes place. Apart from the definition of the overall objective, each intersection of the diagram represents a decision-making point. For example, in the objectives diagram, 3 intersections are illustrated, which stimulates the following questions:

Why have these objectives been selected? Why have others been rejected?

The decision diagram questions the choices concerning:

  • The initial selection of the overall objective
  • The subsequent selection of the intermediate and operational objectives

The planning process should be both strategic (implementation of the best means to achieve objectives of different levels) and empirical (selection of the means likely to achieve specific objectives). Whatever the planning process, the choices will be made in accordance with the objective diagram.

Each of the decision-making points should be given an identification code (which may include a date), to facilitate its transfer to analytical tables (see stage 3).

STAGE 2: DRAFTING OF QUESTIONS

Identification of the rejected objectives

The questions focus on the justification of the selection of objectives. The identification of the rejected objectives helps the accuracy of the wording of questions. How are these objectives identified? At each decision-making point, the evaluator may encounter four situations:

  • Strategy and planning papers explain why certain objectives have been rejected and identify reasons to support this decision.
  • Working papers (interim versions, notes and various correspondences) provide elements to pinpoint rejected objectives and may justify the choices made (in principle, they always include the positions of Member States and the response of the European Commission).
  • The documentation available does not explain the choices made, but interviews with the decision-makers provide relevant information.
  • The evaluators have not collected any written or verbal information relevant to the explanation of the choices made.

In the first three situations, the evaluators should investigate which of the objectives were planned but eventually rejected, and the reasons for rejection.

In the last situation, the identification of the objectives will have to be made by the evaluation team, or with the help of external expertise. The propositions developed this way can be supported by:

  • Information and analyses stated in the available documentation (for example, about the beneficiary country's situation or other donor commitments)
  • Examples from similar countries

Wording of questions

Whatever the situation, the outcome is the answer to the following question:

Why such an objective has been rejected?

Several questions of this type may be needed at each decision-making point.

STAGE 3: COLLECTION OF USEFUL INFORMATION

Before the construction of the diagram, the evaluation manager or, failing that, the evaluation team should check the quality of the sources of information. If this verification reveals poor or unreliable sources, the construction of the diagram should be abandoned.

What is useful information?

Useful information, written or verbal, should not be identified in a too limited way, nor in a comprehensive way. Indeed, the identification of appropriate information for any type of evaluation can only be stated in general.

Written information

Usually, written information can be found in four types of documents.

  • Strategy papers and programmes can provide information about the lessons learned from previous implementations, the political, economic and social context, and the interventions of Member States and the main donors.
  • The preparatory documentation (meeting reports, notes at the end of preparatory missions, correspondence, internal notes or notes between services) may provide explanations about the priorities chosen.
  • Country/thematic/sector-based and project evaluations can be valuable in terms of lessons learned.
  • The European Council and Commission's more general documents (conclusions, recommendations, reports), as well as the international treaties and agreements often display the contextual elements, lessons learned and priorities which are known to the designers of strategies and programmes.
Verbal information

To complement the written information, the evaluator can ask the authors and contributors of the main documents (strategy papers and programmes) to explain the reasons for their choices.

What specific information is required for country evaluations?

Written information

The following list, relating to the kind of available documents and the nature of their information, is not intended to be comprehensive.

  • The final version of the Country Strategy Paper (CSP) indicates the priorities and the final programming; sometimes, it describes the lessons learned from previous implementations, the economic and social problems of the country, and the priorities of the beneficiary government; it usually includes reference to co-operation agreements of the Member States and main donors. The successive versions of the CSP may reveal the priorities and programming intentions which were rejected.
  • Meeting reports and notes following preparatory missions may provide justification for the determination of the priorities and programming.
  • Documents produced during the drafting of the documentation, correspondences, internal notes, and notes between services may specify and explain the resolution of the priorities and programming.
  • Documents describing the reactions of Member States to the drafts addressing the choices made, and the need for explanation, can highlight the types of questions which were debated.
  • The European Commission's compulsory reply to the Member States' questions shows the justification for the choices.
  • Documents collecting the partner government's reactions to the projects may explain governmental priorities.
  • Thematic, sector-based and project evaluations may include lessons learned and recommendations, which explain the choices made.
  • Mid-term revisions are supposed to review the analyses underpinning the selection of the priorities and programming.
  • Specific bilateral and multilateral agreements (co-operation, association) and reports from joint monitoring yield details about the common or specific priorities of the two partners.

Most of these documents can be found for the strategies established after 2001. Prior to this date, the situation is more challenging because the documents are not routinely made available, even when the relationship between the evaluators and the Commission's services is productive.

Verbal information

In addition to - or in substitution for - written information, the authors and contributors to the main documents (strategy and programming papers), as well as the decision-makers responsible for the drafts in process, may be valued informants, providing that they are available and in post for a significant period in the head office (DG Dev and DG Relex, EuropeAid), or in one of the delegations.

How is useful information collected?

Collection of written information

Written information should be collected during the preliminary stage of the country evaluation from the relevant services of the Commission, such as the DG Relex and DG Dev (country and regions), and in the European Commission's Delegation in the country under consideration.

Collection of verbal information

The main interlocutors are the managers of the services previously described. If they are unavailable, former managers can also be interviewed if they can be easily contacted.

Is the information collected sufficient?

If the evaluation managers have not identified a list of information sources, the evaluation team should formulate its conclusions about the quality of the sources in terms of quantity, relevance, reliability and accessibility. This judgement is presented to the managers, who take the final decision.

STAGE 4: CONSTRUCTING THE TEMPORARY DIAGRAM

Preliminary analysis of the information

Preliminary selection of the relevant information is carried out on the basis of the objectives diagram. It requires:

  • The establishment of one or more timelines, describing the successive events and/or information included in the drafting process of the strategy and the programme
  • The selection of texts from the documentation collected, relative to the choice of the objectives and the scheduled assistance process, or relative to the factors influencing such choices
  • The classification and the construction of an index for the texts

Following the analysis of the texts, additional information resulting from the documentation collected may be required, or new documents could be requested and investigated (this method yields uncertain findings).

Drafts of the explanations

Provisional answers to the evaluation questions are formulated on the basis of the information collected. Some of the questions may not be answered at this stage.

A table of questions can illustrate the results of the information analysis and take the following shape:

Table A: results of the collected information analysis

Code for the decision-making Question
(wording of the choice)
Answer
(justification of the choice)
Identification of the original text
       
       
       

In Table A, the columns gather data concerning:

  • The decision-making point: determination of a code highlighting the decisions chain and the successive decision-making point. This code can be the date when the choice has been made.
  • Questions about the justification of the decision: each decision-making point corresponds to several questions about the rejected and selected objectives
  • Answers to questions: justification of the choices determined by the strategies and programmes
  • The identification of the original texts refers to the texts which have been selected and coded

Construction of a temporary diagram

The objectives diagram supports the decisions chain. It identifies four (sometimes five) decision-making points dealing with:

  • The overall objective
  • The first row intermediary objectives
  • The secondary row intermediary objectives
  • Operational objectives

Most of the time the drafting of strategies and programmes is not strictly and exclusively driven by such a rationale and is, in fact, often empirical. Implicitly or explicitly, the designers of the strategy and programme begin with an overall objective. Thereafter, they examine which means at the disposal of the European Commission are able to achieve this objective. The successive choices can be synthesised in two points:

  • The choice of the overall objective
  • The choice of the fields and the intervention processes

Preceding each point, and on both sides of the decisions chain, flows of inputs identified from the analysis of the written and verbal information collected, can be found.

At the left of the sketch, a magnifier explains the decision's outcome:

  • Selected objectives
  • Rejected objectives

A global fishbone shaped diagram is thus completed through detailed diagrams corresponding to each decision-making point.

Sketch of the moment of the decision-making: example of first row intermediary objectives

 

STAGE 5: TESTING OF THE TEMPORARY DIAGRAM

The decisions and their explanation must be confirmed by the main actors responsible for the drafting of the strategy and the programming, including the European Commission's services (head office and delegations), the representatives of the other stakeholders (Member States, NGOs, etc.), the usual interlocutors in beneficiary countries and/or their government.

The observations of the respondents can be recorded in the following table.

Table B: Results of the testing of the temporary diagram

Date and code for the decision-making Decisions made according to the temporary diagram Justification based on the temporary diagram Observations by the decision-makers consulted
       
       
       

In Table B, the columns gather data concerning:

  • The decision-making point: identification of the decision-making within the decisions chain (identical code to the first column of Table A)
  • Decisions made according to the temporary diagram: determination of the selected and rejected objectives which have been subjected to the questions presented in the second column of Table A
  • Justification based on the temporary diagram: answers to the questions justifying the decisions set out in the strategies and programmes third column of Table A
  • Observations by the decision-makers consulted: approval (with or without reservations) or rejection of the decisions and their justification

STAGE 6: CONSTRUCTING THE FINAL DIAGRAM

If the observations of the decision-makers consulted are not critical, the temporary diagram is corrected incorporating the observations. This operation produces the final diagram.

The process of testing of the temporary diagram may question some of its parts when the justifications do not illustrate the real strategic and programming drafting process.

In this case, the information should be reviewed and augmented by another consultation round. The evaluators and the evaluation managers should decide whether they need to construct a new temporary diagram and test it. Indeed, the observations collected during the test may be sufficient to avoid another consultation and to establish the final diagram directly.

The final and temporary diagrams have the same shape (a main diagram and the sketch of the point of decision-making). The final diagram includes an explanatory table about the analysis of the information collected.

.

HOW SHOULD THE DECISION DIAGRAM BE USED?

.

HOW MANY DIAGRAMS SHOULD BE ESTABLISHED?

 

Diagram of strategic decisions

Usually, the number of the diagrams to be constructed should correspond to the number of the strategy papers drafted during the evaluation period.

Evaluators and evaluation managers may decide to work on the most recent document only, when required. Indeed, some of the early documentation needed for the establishment of the decision diagram may be too old to guarantee its availability and that of its authors.

Whatever the case, it is recommended that a decision diagram for each possible medium-term strategic revision is prepared.

Diagram of operational decisions

Such a diagram is usually prepared for each development assistance programme. As strategy papers may cover several programmes, the evaluators may have to study 4 or 5 programmes for an evaluation in a twelve-years timeframe.

The construction of such a large number of diagrams represents a significant amount of work and cost, and yields uncertain results, being dependant on the availability of the information (documents and informants). Thus, the evaluators and evaluation managers may agree to focus on programmes relating to the most recent strategy paper.

HOW SHOULD THE DECISION DIAGRAM BE USED?

Recommendations for country evaluations

The general recommendations about the use of decision diagrams are derived from the experience drawn from their use in the particular context of country evaluations.

Decision diagrams should be established whenever the evaluation studies the relevance of the objectives with regard to the European Union's goals, the country specificities and the overall context.

Previous evaluations show that, excluding extreme situations, objectives and interventions are usually relatively coherent, although the appropriateness of the choices made is not always indisputable. The decision diagram is likely to alleviate this particular limitation.

The use of decision diagrams in thematic and sector-based evaluations appears possible, but should be tested, and adapted where necessary.

Evaluation of the relevance of the strategy and planning

The explanations and justifications for the choices stated in the documentation and from informants should be evaluated.

Thanks to their experience or with the support of external experts, evaluators should formulate their own views about the relevance of the explanations which are provided for each decision-making choice. They should be able to judge to what extent the rejection of an objective seems justified.

The selection of the overall objective (following the rejection of other possible overall objectives) should be consistent with the strategic goals formulated by the European Union's institutions (Council, Parliament, Commission). The justification of the subordinate objectives is based on coherence with the overall objective or the ones resulting from it, and with the decision chain's inputs.

Evaluation of the quality of the partnership

The partnership between the country's authorities and the representatives of the Member States can be evaluated through the study of specific input flows.

Thus, a judgement can be formulated about the importance given to the following topics during the strategy and planning decision process:

  • Member States' strategies, programmes and agreements
  • Member States' opinions about the European Union's interventions
  • Priorities and agreements of the partner government
  • Partner government's wishes about the European Union's interventions

Evaluation of the 3Cs

The following can also be analysed:

  • The coherence of the development assistance strategy with other European policies (migration, trade, agriculture, etc.)
  • The coordination between the co-operation policies of the Commission and Member States
  • The complementarity of the Commission's programme planning with that of other donors (including the EIB)

Additional tools

These tools can be used when the decision-making factors are not explicit in the documentation collected, or when their interpretation is uncertain.

They are especially useful for evaluations where the terms of reference explicitly include an analysis of the decision-making process (thematic evaluations about the decision-making process).

The evaluators can present their successive judgements in a series of tables completing the diagram. 

Validation of the diagram by the decision-makers consulted during the test

Table C: Evaluation proposed for the selection of objectives

Code for the decision-making Decisions made according to the final diagram Justification based on the final diagram Observations by the decision-makers consulted Judgement of the evaluators
         
         
         

In Table C, the columns gather data concerning:

  • The decision-making point which is identified in the sequence of the decision-making (the code is the same as the one chosen in the first column of Table A (stage 4))
  • Decisions made according to the final diagram: determination of the selected and rejected objectives
  • Justification based on the final diagram: includes the findings of the test (reported in the fourth column of Table B (stage 5))
  • Observations by the decision-makers consulted: approval (with or without reservation) or rejection of the decisions and their justification
  • The judgement of the evaluators: agreement (with or without reservation) or rejection of the observations made by the decision-makers
Validation of the diagram by an expert panel including the Commission's decision-makers and independent experts

The evaluation may be subjected to a panel composed of representatives of the Commission's services (geographic departments, delegations), and independent and recognised experts. The findings could be reported in the following table:

Table D: Final evaluation of the selection of objectives

Code for the decision-making Decisions made according to the final diagram Opinions in favour of the final diagram Opinions against the final diagram Judgment of the evaluators
         
         
         

In Table D, the columns gather data concerning:

  • The decision-making point, which is identified in the sequence of the decision-making (the code is the same one chosen the first column of Table A (stage 4))
  • Decisions made according to the final diagram: determination of the selected and rejected objectives (second column of Table C)
  • Opinions in favour of the final diagram: opinions of panel members relating to the final diagram
  • Opinions against the final diagram: opinions of panel members relating to the final diagram
  • The judgment of the evaluators: agreement (with or without reservation) or rejection of the observations made by the decision-makers

HOW ARE THE FINDINGS PRESENTED?

Intermediary documentation

The intermediary documentation is addressed to the managers of the evaluation and the steering committee. It must include all the graphs and tables in detail.

Final report

The final report is addressed to a wider public, more interested in the findings than the evaluation's methodology.

It is recommended that analytical graphs and tables should be placed in an annex and a synthesis of the findings should be shown in the main report under three headings:

  • Relevance of the strategic and planning decisions
  • Description of the partnership between Member States and the beneficiary government
  • Coherence, Co-ordination and Complementarity (3 Cs)

A simplified diagram of the strategic and operational decisions may usefully be included in the main report.

Presentation of the final study

It may be useful for the evaluators to use the standard decision diagram during the presentation of the evaluation's findings.

WHAT ARE THE PRECONDITIONS FOR THE ESTABLISHMENT OF A DECISION DIAGRAM?

Human resources and working arrangements

Type of work required for the design of a decision diagram

 

Travelling expenses

Strategy papers used to be prepared under the responsibility of the Commission's Directorates-General (Relex or Dev) which are also responsible for the planning stage.

EuropeAid is in charge of the project design. Thus, the majority of the useful documentation (the whole baseline documentation) can be found at the Commission's Headquarters in Brussels. The delegations have taken part in the drafting process of the most recent documents and their responsibilities in this area will grow.

In addition, decision-makers from ACP countries benefiting from European assistance may in principle take part in the drafting process of the strategies and programmes which have been co-signed.

Computer devices

Specialist software can support decision-making. Yet, these devices do not seem relevant to the evaluation context, where spreadsheet programs or PowerPoint should be sufficient.

.

EXAMPLES

.

BIBLIOGRAPHY

.

  • 'Analyse stratégique de la décision', Carluer F. et Richard A., 2002.
  • 'The anatomy of influence: decision making in international organization', Cox et Jacobson, 1973.
  • 'L'analyse des politiques publiques', Muller P. et Y. Surel. 1998.
  • 'Les approches cognitives des politiques publiques', Muller P. et Y. Surel (eds), Revue française de science politique, avril 2000.
  • 'Administration et processus de décision', Simon H. A., 1983.
Evaluation methodological approach > EN: Methodological bases and approach > Evaluation tools > Decision diagram > Check lists

Check lists

.

CHECK-LIST FOR EVALUATORS

.

Questions Answer
Preparation and design  
Has a preliminary assessment of the nature and quality of the information been carried out by the managers of the evaluation?  
If not, has the evaluation team carried out such an assessment, which has been approved by the managers?  
Construction of the diagram  
Have the choices of objectives been clearly identified in the objectives diagram?  
Is the decision-making process clearly established within the decision chain?  
Does the available documentation provide information about rejected objectives and the reasons for these rejections?  
Do the interviews with the authors of the documentation compensate the deficiencies of the written resources?  
Do the questions of interviews logically result from the identification of both the rejected and selected objectives?  
Does the temporary diagram explicitly present the results of choices (objectives selected and rejected)?  
Has the test of the temporary diagram been conducted with the main stakeholders (authors, operators, partners)?  
Have the findings of the test been impartially taken into account in the drawing up of the final diagram?  
Has the diagram been used to properly judge the relevance of the strategy and planning, the quality of the partnership, and the 3Cs?  
Has the reviewed judgement been validated by a panel composed of decisions-makers from the European Commission and independent experts?  
Are the panel's conclusions accurately taken into account?  

.

CHECK-LIST FOR MANAGERS

.

Questions Answer
Does the diagram describe the whole of the objectives and present in an explicit way the rationale of the choices carried out?  
Has the test of the temporary diagram been conducted with the main stakeholders (authors, operators, partners)?  
Has the diagram been used to properly judge the relevance of the strategy and planning, the quality of the partnership, and the 3Cs?  
Has the reviewed judgement been validated by a panel composed of decisions-makers from the European Commission and independent experts? Are the panel's conclusions accurately taken into account?  

.

Evaluation methodological approach > EN: Methodological bases and approach > Evaluation tools > Interview

Interview

SUMMARY

.

Why is this tool used in evaluation?

The interview is an information collection tool which usually takes the shape of a face-to-face discussion between the evaluator and the interviewee. In evaluation, the use of interviews is simple, quick, and affordable, which makes its use inevitable.

Figure 1: Example of the interview's role in evaluation

What use can be made of the interview?

In evaluation, the interview collects different kind of information:

  • Facts and information for the verification of facts
  • Opinions and perspectives
  • Analyses
  • Suggestions
  • Reactions to the evaluator's hypotheses and conclusions

The interview may be used as a quantitative collection tool; however, it is mostly a qualitative device. Information, including facts that can be checked, points of view, analyses and opinions should be clearly distinguished. Three types of interviews can be carried out:

Unstructured interviews
The interviewee expresses himself/herself freely and can discuss unplanned topics, because there is no predetermined set of questions. The evaluator intervenes only to generate and develop questions relating to the interviewee's comments.

This type of interview is particularly interesting at the start of an evaluation, in order to get a global view of the subject, and identify the major topics and issues.

Semi-structured interviews
This type of interviews collects the respondents' testimonies using an interview guideline (flexible framework of topics derived from the evaluation questions). The evaluator modifies the interview guide's instructions with additional questions, in order to develop useful areas of inquiry during the interview.

This type of interview is the most frequently used, particularly when the evaluator knows sufficient about the aims and the main questions to pose during the evaluation.

Structured interviews
The evaluator follows strictly the interview guideline instructions. He/She asks different interviewees the same set of questions and the interviewee is not given the opportunity to express himself/herself freely. The evaluator avoids generating and developing additional questions. Answers to each question tend to be short.

This type of interview is useful when a large number of interviews must be carried out and when the evaluator wants to minimise the risk of bias from the interviewer.

Establish homogeneous interview grids when several teams are responsible for conducting the interviews. Semi-structured interviews are the most commonly used tool in evaluation and are the subject of further guidance.

How is the interview carried out?

How is the interview prepared?

The evaluator should first prepare the list of questions to be asked during the interviews.

The schedule of questions indicates the categories of respondent to be interviewed, within which the evaluator chooses those most capable of providing the information. The evaluator must determine:

  • The real beneficiaries of the policy implemented
  • The people who have played a strategic role
  • The people concerned with programme implementation
  • The people who might have been behind the programme's limitations or unscheduled impacts (such as actors with diverging interests, intervening during the programme's operation, or target groups of the policy or the programme)

Once the categories of respondent are defined, the evaluator can schedule the interviews and try to find a balance between the rational and optimal use of his/her own time and a flexible and "human" vision of the other's time.

Questionnaire grids (the evaluation's strategic questions), and interview guidelines derived from them (questions asked during the interview), vary with the categories of respondent, the latter's links with the evaluated issue and the type of interview (unstructured, semi-structured or structured interviews).

Grids should include all themes and questions which the evaluator wants to discuss with the respondents. The questionnaire grid is an intermediary between the evaluation study's design and its implementation

Interview guidelines provide the interview with a framework which is not binding on the evaluator.

How is the interview conducted?
 

Stage 1: Establish a rapport

  • To be aware of and respect local habits and customs
  • To anticipate any language difficulties
  • To explain the purpose of the interview
  • To establish the rules, such as the interview's length, the recording of the interview and anonymity

Stage 2: Adjust the respondent's answers to the interview subject

  • The evaluator must adjust to his interlocutor's role and hierarchical rank in the institution and be aware of the specificities of the respondent's answers
  • The evaluator must be flexible while controlling the interview's progress

Stage 3: follow the interview guide and deepen the questioning

  • Show reactivity through the use of contradiction, notification, etc.
  • Make direct observations during the interview, even when it is not planned in the interview guideline

Stage 4: conclude the interview

  • Keep track of all the information: the evaluator should read his notes shortly after the interview, structure them and add, if necessary, non-verbal details
  • Protect the confidentiality of the interview
  • If necessary, validate the content of the interview report with the respondent

What are the preconditions for its use?

Figure 2: the preconditions for its use
The time span The preparation for the interview does not take long.

The number of interviews which can be carried out during the day is limited. In practice, at the interviewee's request, the expert may conduct an interview with several respondents at the same time. This particular use of the interview increases the opportunity of collecting the information required in a relatively short time.

Human resources Interviews must be conducted by a trained professional. The necessary skills are:
 

  • Thorough knowledge of the major topics and issues addressed in the evaluation
  • Excellent interviewing skills (ability to listen and maintain the momentum of the interview, to be reactive and get to the point quickly, to control the course of the interview)
  • The ability to quickly understand the respondent's perspective in order to be interactive
Financial resources Possible transportation costs

Costs depend on the number of interviews; however, the interview itself does not lead to substantial costs

What are the advantages and limitations of the tool?

Figure 3: The advantages and limitations of the tool
Advantages Quick and easy to use.

Short delays and low costs.

Appropriate tool to meet a limited number of key respondents.

Essential tool to develop analyses and understand the stakeholders' perceptions of the programme.

Limitations At a reasonable cost, only few people can be interviewed.

Problems relating to the respondent's 'representativeness' particularly for social groups and beneficiaries.

The information must be checked and interviews are generally combined with other analytical tools.

Evaluation methodological approach > EN: Methodological bases and approach > Evaluation tools > Interview > Detailed presentation

Detailed presentation

This section is structured as follows:

 

WHY AND WHEN?

 

WHAT ARE THE REASONS FOR USING INTERVIEWS?

What is the purpose of an interview?

The use of interviews in evaluation is inevitable. This tool collects information and points of view, and analyses them at each stage of the evaluation.
The interview usually takes the form of a face-to-face discussion between the evaluator and the interviewee.

Where does this tool come from?

Interviews are used in many fields, such as psychology, ethnology, sociology, anthropology, and philosophy.
In psychology, this tool focuses on motivation, or the reasons for a person's behaviour. The interview is thus used to study human actions and to help with the understanding of the human psyche.
In ethnology, the interview aims mainly at collecting direct observations. In sociology, the interview is used in all activities.

Three types of interviews and their contribution to the evaluation

Unstructured interviews

The interviewee expresses himself/herself freely and can discuss unplanned topics, because there is no predetermined set of questions. The evaluator intervenes only to generate and develop questions relating to the interviewee's comments.
This type of interview is particularly interesting at the start of an evaluation, in order to get a global view of the subject, and identify the major topics and issues.

Semi-structured interviews

The evaluator modifies the interview guide's instructions with additional questions, in order to develop useful areas of inquiry during the interview.
This type of interview is the most frequently used, particularly when the evaluator knows sufficient about the aims and the main questions to pose during the evaluation.

Structured interviews

The evaluator follows strictly the interview guide's instructions. He asks different interviewees the same set of questions, in the same order, and using the same words. The evaluator avoids generating and developing additional questions, and the interviewee is not given the opportunity to express himself/herself freely. Answers to each question tend to be short. Structured interviews are seldom used in evaluation, where the evaluator needs to adapt to the situation. However, they can be used to classify points of view and information about the impact of a project/programme by categories. Thereafter, the evaluator can use the results of these interviews to design a questionnaire, with a view to analysing the impact of the project/programme.

Semi-structured interviews are the most commonly used tool in evaluation and are the subject of further guidance.

 

WHEN SHOULD THE INTERVIEW BE USED?

What kind of information does the interview collect?

The interview may be used as a quantitative collection tool; however, it is mostly a qualitative device.
In evaluation, the use of interviews is simple, quick, and affordable.
The interview collects:

  • Facts and information for the verification of facts
  • Opinions and perspectives
  • Analyses
  • Suggestions
  • Reactions to the evaluator's hypotheses and conclusions

However, a series of interviews is not sufficient to quantify an event, because the answers are not standardised. This is the main difference between the interview and the questionnaire.

The added value of the interview

Among other advantages, this tool is essential for the development of analyses because it collects information taken directly from the context. Thus, it provides a good indication of what motivates stakeholders to act, their various perceptions of the programme's aims, problems encountered and effective outcomes.

Can the interview be combined with other collection tools?

The interview is a useful device for developing hypotheses and analyses. It can highlight the programme's aims and dynamics, the stakeholders' rationale, and the organisation of the various opinions and perceptions of the programme.
Depending on the type of observation tool used and on the stage of the evaluation, the interview can be used in combination with other tools:

  • To test the main questions to address at the beginning of the evaluation with a view to preparing a series of focus groups.
  • To be the main observation tool, and be supported by a questionnaire (if the country's general context allows it) or, for example, a focus group of beneficiaries.

How should the interview be conducted?

Usually, the interview takes the form of a face-to-face discussion. This arrangement is particularly effective, as the relationship builds on trust and interviews supplement written information.
However, beyond verbal information, other elements should be taken into account, such as the context, the general mood of the session, the people likely to influence the interviewee, etc, as well as the interviewee's reactions: hesitation, silence, eye contact, etc.

From whom does the interview collect information?

The interview is a suitable tool for collecting information, analysing and forming conclusions from a limited (but essential) number of respondents, such as:

  • Partners and people in charge of the evaluated policy or programme
  • Strategic institutional stakeholders
  • Main operators and people in charge of the programme implementation
  • Representatives of beneficiaries

 

AT WHAT STAGE OF THE EVALUATION SHOULD THE INTERVIEW BE CONDUCTED?

Types of interview appropriate to various stages of the evaluation

Stages of the evaluation Type of interview The interview's contribution to the evaluation
Desk phase:
implementation of the methodology and preparation of the mission to the country
Preparatory interview

used for the design of questionnaire grids and the selection of respondents. At this stage, interview guides should be flexible and aimed at highlighting the topics on which the interview is based.

It strengthens the basis for the choice of major topics and issues, and completes the questionnaire grid.

Its structure should be flexible.

Collection of information in the country and from the European Commission The interview is

designed to collect information and perspectives. Several interview guides should be developed to correspond to the different categories of respondents, and to the major topics and issues.

It collects information on the programme's objectives and outcome from the people in charge of the programme, operators, stakeholders and beneficiaries.
Analysis and preparation of judgements In-depth interviews for presenting and investigating the issues, used to collect reactions to the evaluator's findings and analyses. At this stage, the evaluator can test the relevance and the feasibility of his/her conclusions. It collects feedback from respondents. It gives in-depth information and reformulates old questions.

 

WHAT ARE THE ADVANTAGES AND THE LIMITATIONS OF THE INTERVIEW?

The advantages

The interview is appropriate for project evaluation, as well as for more complex evaluations, such as sector evaluations and country/region evaluations, where it should be combined with other tools.

The limitations

 

WHAT ARE THE PRE-CONDITIONS FOR THE USE OF THE INTERVIEW IN EVALUATION?

The time span

The preparation for the interview does not take long.

One expert will not be able to conduct many interviews per day and, therefore, the number of interviews which can be carried out during the mission is limited. In practice, at the interviewee's request, the expert may conduct an interview with several respondents at the same time. Thus, this particular usage of the interview increases the opportunity for collecting the information required in a relatively short time.

Human resources

Interviews must be conducted by a trained professional. The necessary skills are:

  • Thorough knowledge of the major topics and issues addressed in the evaluation
  • Excellent interviewing skills:
  • The ability to quickly understand the respondent's perspective (his/her interest in the interview, whether he/she has expressed himself freely, whether he/she has committed himself) in order to be interactive and, where appropriate, modify the questions

Financial resources

Costs depend on the number of interviews and their physical location in the country. However, apart from professional fees and transportation costs, the interview itself does not lead to substantial costs.

 

HOW THE INTERVIEW IS CARRIED OUT?

 

HOW IS THE INTERVIEW PREPARED?

Stage 1: list the categories of stakeholders and interviewees

The evaluator should define the categories of stakeholders he needs to meet, and then select a sample group for each category.

Examples of stakeholders' groups
  • Agents in charge of the operational strategy / the implementation
  • Actors specialised in a sector of intervention
  • Operators working at a local level / at a national level
  • Technical operators: representatives / on-site operators
  • Public sector agents (working for the state, the administration, etc.) / private sector agents (working for the civil society, NGOs, etc.),
  • Beneficiaries

The list of the stakeholder categories will evolve as the evaluation progresses. Thus, additional interviews should be anticipated.
Some choices are obvious (for example, the agent in charge of the implementation of a nationwide programme), but others are more complex.
Moreover, the evaluator must consider how he will meet the interviewee, which depends on:

  • The respondent's accessibility (will the evaluator meet him in person or only a representative)
  • The respondent's availability
What are useful questions to bear in mind for the selection of interviewees?
  • Within the relevant institution, who knows the programme/the politics well?
  • Who is in charge?
  • Who is in charge of implementation, of monitoring?
  • Of which aspects of the programme is the respondent in charge?
  • Who will be able to trace the programme's evolution, in the context of a rapid turnover of the implementation team?

During the interviewees' selection, the evaluator should be careful to distinguish between direct witness' testimonies and second-hand testimonies.

Stage 2: design the questionnaire grid

Questionnaire grids, and interview guides derived from them, vary with the categories of respondent and the latter's links with the evaluated issue.
Grids should include all instructions, themes and questions which the evaluator wants to discuss with the respondents.

The stages required to organise the grid

 

The questionnaire grid is an intermediary between the evaluation study's design and its implementation.

Design of the interview guide

The evaluator should allow the interview to remain as close to an open discussion as possible and accept discursive answers, inconsistencies, pauses, hesitations, etc.
The questions can be written in the questionnaire grid, or be asked spontaneously, in order to complete, to probe, to give new insights and to challenge the respondent's answers.

 

Stage 3: schedule the interviews

This organisational stage depends on the evaluation stage within which the interview is implemented, and on the respondent's availability.
The selection criteria must clearly indicate the priority of the people to be met. Furthermore, the evaluator must be careful about:

  • The risk of key officials being unavailable
  • His/Her short mission span
  • Travelling time
  • Scheduling too many interviews, leading to the cancellations of appointments due to poor planning or unexpected events
  • Respecting his/her appointment, such as the time, although conventions vary from one country to another
  • Conducting the interview's introductory stage slowly, in order to establish rapport and respect traditions and customs
  • Unplanned and additional interviews which have to be organised because a respondent has suggested an "interesting" informant who may be interviewed, or because a key interviewee is now available
  • Tiredness: too many interviews conducted in a day may affect the capacity of the interviewer to listen

The evaluator must find a balance between:

 

Scheduling 4 to 6 interviews a day seems to be a good compromise between the two visions.

Make the appointments

This stage constitutes the first contact between the interviewee and the evaluator. Therefore, the choice of the person in charge of this task is very important.
It is often better to schedule appointments in advance. However, time should also be allocated for unplanned interviews, particularly in the context of country/region evaluation.
The evaluator must know how to resist pressure to meet as many respondents as possible. Organising focus groups can solve this problem.

Send interview guides prior to the appointment

This stage is optional, however, it often eases the interview's course by giving the respondent the opportunity to:

  • Understand the topics on which he/she has to talk
  • Collect all the necessary documentation
  • Ask the evaluator whether the participation of other respondents during the interview is possible

It will save time for both the respondent and the evaluator, and help them feel at ease. Most of the time, the quality of the interview depends on this.

Conduct additional interviews

They are often necessary after the first elements are collected and during the progress of the evaluation. Thus, they must be programmed into the planned timetable and budget.

HOW IS THE INTERVIEW CONDUCTED?

Stage 1: establish a rapport

The interview is easy to conduct but its findings and value vary greatly with the way it is conducted.

The evaluator's duty

  • To be aware of and respect local habits and customs (such as rules of politeness, silences, wording of questions, time devoted to conversation prior to core questions, etc.)
  • To anticipate any language difficulties (codes, interpreter's competence, meaning of terms such as "development", etc.)
  • To adjust himself/herself to the interviewee, his/her role and his/her hierarchical rank in the institution (for example, the evaluator may be asked not to reveal some information)
  • To take into account the interviewee's material, social and hierarchical environment which may greatly influence his/her attitude
  • To explain the purpose of the interview, how the respondents have been chosen, and the intended use of the information
  • To establish (and sometimes to negotiate) the rules, such as the interview's length, the recording of the interview, etc.
  • To inform the interviewee at the start whether his answers will be used anonymously
  • To ensure that the interviewee has understood the aims of the interview and is willing to respond

To find out more:

  • What are the tasks to be completed during the preparation for the interview in country/region evaluation?

Stage 2: Adjust the respondent's answers to the interview subject

The evaluator must adjust to his interlocutor's role and hierarchical rank in the institution. As a consequence, he/she must be aware of the specificities of the respondent's answers, such as:

  • The way he/she understands the questions and deals with them
  • The possible difficulty in expressing points of view, or criticisms
  • Ideas he/she might want to point out with the evaluator's mediation
  • Considering the evaluator as an auditor

The evaluator must also adjust to the interlocutor's attitude: his/her personal perspective, points of interest not planned in the interview guide, etc.

As a consequence, the evaluator's flexibility is the key to successful data collection in an interview. During the interview, however, he/she must control its progress by staying within the bounds of the subject and avoiding dwelling too much on one topic.

Stage 3: follow the interview guide and deepen the questioning

Flexibility and control are two elementary rules in an interview. The respondent's information is being analysed simultaneously through the evaluator's capacity to listen. Thus, the evaluator should not express himself/herself in a way which is detrimental to the interview, nor let the interviewee talk without limit. In practice, the evaluator's ability to react should provide him/her a balance between flexibility and control.

Three types of reactions

  • Contradiction (the evaluator highlights the respondent's contradictions or expresses the contrary views of other respondents, whose identity may have to be concealed)
  • Notification (the evaluator informs the respondent that he/she is about to address a new theme or a new question)
  • Clarification (the evaluator asks the respondent to develop a specific point)

Types of clarification

Questions Repeat verbatim the respondent' s comments in the interrogative form
Repetition Return to items which have already been discussed
Interpretation Summarise the content of what has been said, check its interpretation with the respondent (and correct it, if necessary), and move on to the next point

 

Intervene directly in the interview

Although the interview guide provides a useful structure for the interview, the evaluator must be free to generate and develop questions beyond it.

Making direct observations during the interview enables the evaluator to collect specific information about the respondent's attitude, behaviour and customs, and / or about the people around them. With these observations, the evaluator can detect discrepancies between the respondents' attitude and their words.

To find out more:

  • Three types of interviews and their use in evaluation

The nature of the collected information

The evaluator must distinguish reliable facts from points of view, personal analyses and opinions. To do so, he/she should press the respondent to support his/her allegations with facts or actual examples which the evaluator can check, and which reinforce the respondent's comments.

Avoid asking difficult questions

During an interview, questions should be answerable and should avoid discouraging respondents. Thus, interviewees should be spared from having to provide information stemming from archives or voluminous report.

Control the information

In an interview, the first level of control occurs with the triangulation of questions. This methodology is based on the principle that three different sources are needed to validate the reliability of the information. Triangulation is used in the interview in two ways:

  • In interviews, the evaluator asks systematic and similar questions to at least three different respondents. Prior to this triangulation, the information is not considered to be reliable.
  • In a single interview with one respondent, the evaluator asks a question in three different ways, in order to check the information and observe the possible fluctuation in the respondent's analyses and interpretations of an event.

Stage 4: Conclude the interview

Conclude the interview

The evaluator should close the interview with positive reflections or an open conversation, and maintain a polite approach. He/She should also decide whether another appointment with the respondent is necessary.

Keep track of all the information

The evaluator should read his notes shortly after the interview, structure them and add, if necessary, non-verbal details such as the respondent's behaviour, trouble, silences, interruptions, his relationship with him, the atmosphere, and especially the respondent's suggestion of other people to be interviewed (who will have to be contacted) and read the specified documentation, etc.

These elements are invaluable; they are not detectable in audio tapes, nor short notes. Thus, the evaluator's impressions at the end of the interview must be considered a valid source of information.

Protect the confidentiality of the interview

If the respondent's willingness to respond depends on the evaluator's assurance of confidentiality, this principle must be respected, along with respect for the respondent's private life.

If necessary, validate the content of the interview report with the respondent

Depending on the intended use of the respondent's answers, this validation can be important if, for example:

  • the respondent's verbatim comments are cited in the report
  • They stand as evidence on their own
  • The evaluator finds them ambiguous and is afraid of making a poor interpretation, or worse, a misinterpretation
  • The respondent has an official or sensitive duty
  • The respondent's willingness to respond depends on this validation

Anticipate what will have to be done after the interview

  • The intended use of the information collected
  • Its analyses
  • The types of debriefing, which should meet the client's and respondents' expectations (sometimes, interviewees request feedback about the use of their answer)

 

IN COUNTRY/REGION EVALUATION

 

WHAT ARE THE TASKS TO BE COMPLETED DURING THE PREPARATION FOR THE INTERVIEW?

The hypotheses and questionnaire grid development

In country/region evaluation, the evaluator needs to define the strategic questions quickly, in order to be efficient during the information collection.

 

The evaluator's tasks during this first stage

Select the types of information to collect

The evaluator should:

  • Organise the information to be collected on strategic questions between key and secondary interest
  • Devise a methodology to limit the collection of documentation. As country/region evaluation covers so many diversified fields, the evaluator risks dissipating his efforts if this stage is neglected
Develop and organise the questions

The evaluator should classify key questions by their aims and categories of interviewees.

Determine the list of the respondents
Settle the order in which respondents will be met

THE TYPES OF INTERVIEWS?

Different types of interviews can be distinguished during the two main stages of country/region evaluation

During the first stage

Preliminary interviews setting the programme's purpose and boundaries

These consist of meeting people who have a global vision of the policies implemented, their objectives and impact, and who belong to relevant institutions. These interviews can avoid bias in the process of an evaluation arising from reliance on officials in charge of policy definition and monitoring.

Interviews with different categories of stakeholders

During these interviews, the evaluator meets the stakeholders who have been participating in the process of drafting, definition, negotiation and the operational choices for the implementation of policies.
Thus, the relevance and impact of geographical strategies rely on what different stakeholders have done in the process of drafting and negotiation.

During the second stage

Interviews with operators and beneficiaries

The use of interviews to analyse the impact of a programme on beneficiaries in country/region evaluation is not usually advisable because of the large number of people involved, representative problems, etc. However, interviews can be conducted prior to the implementation of other data collection tools, such as focus groups and questionnaires.
Thus, interviews should focus on particular categories of beneficiaries and/or precise questions dealing with various hypotheses.

HOW ARE RESPONDENTS CHOSEN?

In country/region evaluation, this issue is crucial. It is less a question of knowing how to collect information, than of determining what the information's content should be, and from whom it can be obtained, in the context of short missions and wide fields of inquiry.
Choosing the categories of respondents is particularly important:

  • To identify people who have benefited from the implemented policies
  • To identify those who have played a strategic role in the programme
  • To identify those who have played an intermediary role
  • To identify those who could have been behind obstructions or unplanned effects (such as groups with diverging interests, intermediary groups intervening in the middle of the process, target groups of the policies / the programme, etc.)
  • To analyse current strategies and the various rationales considered

Distinguish between the different roles of officials from institutions

The list of officials who should be met can be long because of the nature of the country/region evaluation, which can cover numerous programmes and projects.
Thus, a selection must be made consistent with the budget allocated to the evaluation, and especially when the mission is very short. The evaluator should make a distinction between key interviews and "etiquette" interviews; he/she should also think of the quickest and cost-effective way to conduct them.
Once the roles of officials are identified, the evaluator must locate the people who have agreed to be interviewed, corresponding to this typology.
This selection belongs to the evaluation process and becomes clearer as the first hypotheses are being developed.

Go beyond formal statements

In country/region evaluation, the evaluation's political dimension and economic implications tend to complicate some interviews with institutional authorities and respondents concerned with their hierarchy. The evaluator should be able to "decode" the information he/she is given, by taking into account the context in which the interview is conducted.
A way to avoid formal statements implied by this situation is to carry out an informal interview (for example, away from the institution to which the respondent belongs).

Balance the interviews between officials and beneficiaries

The evaluator must be careful to avoid devoting all the time in his/her mission to interviews with officials, to the detriment of beneficiaries. He/She must check the interviewee's "representativeness", which is always relative and sometimes self-proclaimed. Several questions may help find the respondent's interest in the evaluation and his/her "representativeness":

  • To which group does the respondent belong?
  • What is his/her professional background?
  • What is his/her influence on the programme's strategy?
  • Does he/she have direct or indirect interests in the programme? Are they explicit or implicit?
  • What is his/her opinion about the policies / the programme?
  • How did he/she become a representative (has he/she been elected, designated)?
  • Is he/she representative of the entire group he/she stands for?
  • How many respondents should be interviewed to collect representative information and perspectives?

HOW DOES THE EVALUATOR SCHEDULE THE APPOINTMENTS?

The pre-conditions for appointment making

During the preparation for his/her interviews, the evaluator should be practical and:

  • Fix an order of importance for the collection of data and, after developing the first set of questions, select who should be interviewed
  • See: What specific tasks have to be completed during the preparation of the interview in country/region evaluation?
  • Identify quickly on-site informants and stakeholders. They often hold a position at the centre of many sources of information. Thus, they can advise the evaluator, and suggest "good" respondents, who will be able to give accurate answers.
  • Identify respondents who would be better interviewed in a group
  • Consider the possibility of interviewing people on the telephone, which could save time
  • Avoid overestimating the number of interviewees that can be met during a short mission

How to schedule the appointments?

The particular context of country/region evaluation implies taking additional precautions, in addition to the usual precautions, in the making of appointments.
The evaluator should:

  • Schedule in advance at least part of the interviews that he/she needs to conduct on-site
  • Ensure that the evaluation's clients (for example, the European Commission Delegation or the national authorities), or the local consultants organise the first set of interviews on his/her arrival
  • Make time available for unscheduled appointments made necessary because of new perspectives identified during his mission. As a consequence, the evaluator's hypotheses can be extended or modified.

With a view to being more efficient, the evaluator could:

  • Use the appointment making process as an opportunity to communicate to the respondents all the information they need to know: the purpose/ of the evaluation, what questions they should prepare for, and the interview/ duration. This is also the appropriate moment to check the respondent's profile against the planned content of the data collection.
  • Distribute an interview guide prior to the interview. This is appreciated by all parties because it is time-saving and directs the respondent's information gathering to specific documentations.
Evaluation methodological approach > EN: Methodological bases and approach > Evaluation tools > Interview > Check lists

Check lists

CHECK-LIST FOR EVALUATORS

.

Questions Answer
Preparation and design  
Does the list of respondents meet the needs of the evaluation's methodology?  
Have alternatives been planned by the evaluators in case of cancellations of appointments with the actors?  
If any, has the issue of "representativeness" been solved?  
In interviews with representative stakeholders belonging to the evaluation's spotted category, has the respondent's " representativeness " been checked?  
Do the interview grids cover all the evaluation issues?  
Does the design of the interview guides vary sufficiently to meet the needs of different categories of stakeholders?  
Implementation  
Have the evaluators controlled and checked the information collected?  
Does the intended format designed for the debriefing highlight the differences between reliable information and opinions?  
Is the diversity of perspectives, expressed by the various categories of stakeholders, explicitly exposed?  

.

CHECK-LIST FOR MANAGERS

.

Questions Answer
Does the list of respondents meet the needs of the evaluation's methodology?  
Does the design of the interview guides vary sufficiently to meet the needs of different categories of stakeholders?  
Is the homogeneity of the interviews assured?  
Does the intended format designed for the debriefing highlight the differences between reliable information and opinions?  
Is the diversity of perspectives, expressed by the various categories of stakeholders, explicitly exposed?  

Evaluation methodological approach > EN: Methodological bases and approach > Evaluation tools > Focus group

Focus group

SUMMARY

.

Why is this tool used in evaluation?

A focus group is a form of group interviewing which comprises individuals involved in a development policy or intervention. It is set up to get information concerning the people's opinions, behaviours, or to explain their expectations from the said policy or intervention. In that sense, a focus group is a fast result-driven qualitative survey.

The focus group is useful in evaluations of projects or programmes, and particularly for field studies with beneficiaries and intermediary stakeholders. When a focus group is organised after the implementation of a programme with a view to assess its impact, it helps understanding, analysing and identifying the reasons beneath the opinions expressed by the participants.

What use can be made of the focus group?

Figure 1: Example of the focus group's role in evaluation

 

The focus group is a mean to collect information and points of view quickly. When it involves stakeholders with different points of views, it eases the expression and explanation of the discrepancies within those points of view, as well as enabling an in-depth study of the stakeholders' opinions.

Such is the case for a health sector focus group involving doctors from the private and public sectors.

Less frequently, the focus group can stand as a restitution tool at a local scale. In this case, the tool focuses on the observations and the field analyses' first conclusions.

Through the presentation of the survey's first outcomes, this type of focus group collects the reactions of the stakeholders targeted by the intervention.

In an impact analysis, the focus group can identify the various groups of stakeholders involved in the intervention, and check their reactions towards a given problem. The objective is to detect diverging opinions within a group composed of allegedly homogeneous opinions.

Using a focus group in the impact analysis of the construction of a dam, regrouping people in favour of the project for economic reasons may reveal more precise diverging opinions within the group.

The focus group is the only tool with which the evaluator can analyse and test the information given. It helps grasping the participants' behaviours, their understanding and perception of an intervention, which would not be possible with an interview. Group interviewing can collect a variety of points of view and perceptions stimulated by the interaction between participants. Each of the focus group's members is permanently driven to prove one's statement.

How is a focus group investigation carried out?

What are the conditions for use of the tool?

Before organising a focus group, the evaluator should first define the stakes and goals of the evaluation, and determine a theme to which the tool will provide answers.

The resources allocated to this task indicate the number of focus groups that the evaluator can forecast.

The categories of stakeholders targeted by the evaluation are another component to have in mind while choosing the type of focus groups, knowing that its composition depends on the objectives : an in-depth objective requires a socially homogeneous group, whereas the testing of a theme can only be realised with a group of diverging points of view.

Figure 2: Steps involved in the use of focus group

 

Two types of focus groups can be organised to assess the impact of a policy on a new school course: a first one involving the course's teachers, and a second one gathering inspectors, and the school's directors and teachers.

What are the stages for the setting up of the focus group?

  • Choose the type(s) of focus group needed for the different steps of the evaluation
  • Select one or more local moderator
  • Identify the various interest groups among categories of the targeted stakeholders which are concerned by the assessed policy
  • Select the participants
  • Construct the moderator's guide
  • Plan the focus group meeting(s)

How is the focus group implemented?

Who implements focus groups?

The moderator implements the focus g roup. He/she should be well informed of the evaluation's topics and goals, be familiar with all the techniques relating to group interaction and speak the language of the participants.

If the evaluator does not qualify for one of these skills, he/she must be assisted by a local moderator. The latter should be introduced to the tool's goals within the context of the evaluation, and trained to foster group interaction.

Key informants are often helpful to the evaluator during the selection of participants and the identification of active participants who can foster the debate.

An observer can be invaluable to the evaluation by keeping track of the opinions expressed during the session. 

How is the group interaction fostered?

Prior to the session, the moderator should meet the participants in order to motivate them in becoming actively involved in the focus group. The participants should understand the principles underlying the session's process, and think about the topic before the session. This is particularly recommended for focus groups with users and beneficiaries.

A focus group investigation is organised with countrywomen and the chief of the village has the responsibility to recruit them. In this context, the moderator should benefit from a quick meeting with the women to make himself/herself known, establish a relaxed atmosphere and suggest the first topics to reflect upon.

Focus groups are not just the sum of individual interviews. Thus, the moderator must always initiate and maintain a dynamic interaction between the participants.

A session can be organised in a reactive way, where participants react to the evaluator's analysis, information, etc. as well as in a pro-active way (the information and testimonies of the participants support the development of collective analyses and suggestions).

The moderator should organise the session into stages, including a mid-term debriefing to the group. 

How to keep track of the information?

This stage should not be under-estimated for focus groups conducted in a local language. It consists of the transcription of a session verbatim from the notes which have been taken during the session and its recording (if it has been scheduled).

In the absence of recording, it can be interesting to organise a debriefing session, in order to validate the content of the focus group's transcription.

Example of the focus group investigation in a country evaluation: the Benin mission

Four focus groups were set up to bring elements of explanation to one of the evaluation questions (relative to the decrease of the number of patients going to the health centres which have benefited from the EC's assistance). The evaluation team decided to compose and conduct them differently, in order to check the conditions in which focus group investigations should be prepared and carried out in country evaluations.

Two focus groups with beneficiaries were conducted by local moderators ; 2 members of the evaluation team were responsible for the moderation of 2 focus groups: one involving doctors from the public and private sector of the Cotonou district, and another involving midwives and nurses from Cotonou.

Conclusions from the course of the focus groups and their outcomes can be drawn: the focus group is an efficient collection tool when it gathers socially homogeneous groups within the same socio-professional category. Yet, caution should be taken when participants share too close experiences about the questions under consideration. Their testimonies may turn out to be too identical (for that reason, the focus group with the doctors worked better than the one involving the midwives and the nurses) .

The different course taken by the two focus groups with beneficiaries points out that a particular care should be taken for the selection (when recruiting the) of participants. For example, the moderator should ensure the presence of active participants within the group, in order to foster participation (the moderator should also check that leaders do not impose their points of view on the rest of the group). The moderator should also motivate the participants by meeting them a day before the meeting

What are the preconditions for its use?

Figure 3: the preconditions for its use
The time span
  • If focus groups must be conducted by local moderators, their selection must be organised before the arrival of the evaluation team on site. 
  • Schedule a presentation of the moderator's guide to the moderator. 
  • Meet the participants a day before the focus group's session. 
  • The session should last at least half a day. 
  • Plan for a day dedicated to the transcription of the session's verbatim in case of a focus group conducted in case the focus group was conducted in local language.
Human resources
  • Recruit local moderators in the case of several focus group investigations on a large scale territory or conducted in the local language. 
  • Select key informants for the selection of participants.
Financial resources
  • Remuneration of the moderators and the possible interpreters. 
  • Possible per diem to the participants. 
  • Costs relative to the catering and logistical expenses

What are the advantages and limitations of the tool?

Figure 4: The advantages and limitations of the tool
Advantages
  • It enlarges the reference sample. 
  • It is useful with groups of beneficiaries and especially for impact analysis. 
  • Group interaction fosters the participants' explanation, specification and justification of their testimonies. 
  • It has limited implementation costs. 
  • It is time-saving.
Limitations
  • The collected information are qualitative. 
  • In certain contexts, organising focus groups may prove to be difficult, because of the determination of the beneficiaries groups, far-reaching locations, and the finding of national and local competences. 
  • Public expression could be limited by political and social weights, or impaired by the participant's position within the group.
Evaluation methodological approach > EN: Methodological bases and approach > Evaluation tools > Focus group > Detailed presentation

Detailed presentation

.

This section is structured as follows:

.

WHY AND WHEN IS A FOCUS GROUP USED?

.

WHAT ARE THE REASONS FOR USING A FOCUS GROUP?

The evaluators use focus groups for two different reasons:

  • This tool collects information from a group of participants
  • It facilitates the collective expression of analyses, perspectives and suggestions

In the evaluation field, focus groups are used to achieve different objectives, at different stage of the evaluation and with various participants.

For what purpose should a focus group be used?

A focus group can be used as:

  • A means to collect information and points of view from a group of stakeholders, operators or beneficiaries
  • A means to investigate hypotheses and analyses with groups of stakeholders, operators and beneficiaries
  • A means for various groups of stakeholders (whether or not they belong to the same social and institutional background) to express divergent points of view and their analyses
  • An analytical tool focusing on the programme's effects, particularly at a local level
  • A debriefing tool, at the local level, disclosing the evaluator's observations and his/her first conclusions of the on-site analyses
  • A means to prompt and validate various suggestions and recommendations at an institutional and local level

Les fonctions de l'entretien collectif

For what outcomes?

Focus groups are used in order to:

  • Collect qualitative data, and more rarely quantitative information
  • Examine analyses and perspectives (the participants interact during the session, which facilitates a spontaneous exchange of views)
  • Provide the participants with insights into perpectives from other groups, and examine the diversity of analyses developed by different groups of stakeholders. For example, using a focus group for a sector-based evaluation with the various operators in the sector usually yields good results.
  • Develop and test working hypotheses at the beginning or during the evaluation
  • Develop and test conclusions, recommendations and suggestions with stakeholders and beneficiaries for a mid-term evaluation

WHERE DOES THIS TOOL COME FROM?

The focus group comes from sociology, anthropology and the marketing field.

A controversial issue

Some schools of thoughts refuse to include focus groups in the "family" of group interviewing. As there are several techniques and methodologies available to design a group and conduct sessions, each member of the "family" has its own scientific basis. The choice between these various methodologies depends on the focus group's purpose.

Specific examples

  • In the marketing field, focus groups are frequently used out to develop and validate advertisements or analyse new products.
  • In developing countries, the social communication field uses focus groups to design understandable messages (for example, in awareness campaigns, dealing with aids prevention).

WHAT IS THE ROLE OF A FOCUS GROUP IN EVALUATION?

In what kind of evaluation?

Focus groups are frequently used in project or programme evaluations, and especially in field studies focusing on the beneficiaries' situation.
This tool could also be successfully used in sector and country/region evaluations, where the selection of objectives and types of participants is broader.

What type of focus group for which stage in the evaluation?

Preparatory focus groups

They provide grounds for the selection of the major topics and issues, and complete the questionnaire grid.

Focus groups for information collection

At the heart of the evaluation, these focus groups collect information and analyses:

  • From the people in charge of the project/programme/policies, about the objectives and outcomes
  • From stakeholders and operators, about the policies implemented
  • From beneficiaries, about the effects
In-depth focus groups

During the evaluation, in-depth focus groups gather operators and beneficiaries together to analyse, collect and examine their reactions to the first draft of conclusions and recommendations.
In-depth focus groups should not be confused with sessions between the evaluator and clients which are designed for the validation or presentation of the final conclusions.

With which tools can the focus group be combined?

Focus groups can complement interviews, and be a useful alternative to surveys (which often appear to be difficult to plan and manage, and expensive).

With case studies

Focus groups can be used to conclude case studies in sector and country/region evaluations. For example, the evaluator can set up focus groups to compare opinions, and focus groups which encourage reactions to the evaluator's suggestions.
These types of focus groups concentrate on geographical sites, sectors or economic activities outside the scope of the case studies. Using the information and opinions collected by these focus groups, the evaluator will be able to assess the specificity or the general view of particular findings from case studies.

With interviews

In this case, focus groups aim at examining and challenging elements of individual analyses and opinions which have been expressed in earlier interviews.

WHAT ARE ITS ADVANTAGES AND LIMITATIONS?

The advantages

A means to interview numerous respondents

By definition, the number of respondents from whom information, analyses and opinions are to be collected, is greater in focus groups than in individual interviews. Consequently, the focus group's reference sample is bigger.

  • With focus groups, the evaluator can investigate and develop information and analyses provided by interviews.
  • The evaluator can also ensure that a point of view individually expressed can represent the opinion of a group of stakeholders.

However, participants in a focus group do not constitute a representative sample of the programme's reach, and findings cannot be interpreted statistically because the information collected is qualitative.

A tool which facilitates the study of the programme's effects

This tool is cost effective and can also provide the evaluator with valuable information about the effects of policies / programmes on specific groups of beneficiaries or stakeholders. Considering effect analyses, the evaluator can use this tool to collect the opinions of strategic interest groups about the aims of the policies / programmes under evaluation.
The more precise the preliminary identification and the grouping of strategic interests, the better the quality of the information collected.

A source of creativity, if the group dynamic is controlled

Focus groups facilitating the expression of divergent points of view provide opportunities:

  • To analyse the stakeholders' reactions to other perspectives
  • To examine the argumentation, perceptions and analyses of each group of stakeholders, through exercises involving confrontation and debate

Whilst not designed for this purpose, these sessions, can overcome obstacles and contradictions which are often linked to misunderstandings rather than contradictory interests.

The limitations

Complex organisation and preparation

Prior to the organisation of focus groups, the evaluator usually needs to conduct preliminary analyses on strategic groups which have been formed within the various stakeholders.
In developing countries:

  • When the physical location of beneficiaries is scattered, sessions are difficult to organise and are time-consuming
  • Local authorities may have difficulty in appointing national and local professionals to prepare and organise focus group sessions with the evaluator.

 

Restricted participant expression

Sometimes, the evaluator fails to elicit the participant's views due to:

  • The public nature of the session
  • The inhibiting presence of certain participants
  • Political and social issues, which are more significant in a group situation than in interviews
  • The natural tendency within a group to conform with the majority view and to avoid expressing minority opinions
Time allocation and costs (which can easily increase)

Conducting several focus groups can be problematic in terms of time allocation and costs, especially due to:

  • The preparation, which is often time-consuming
  • Compensation paid to participants
  • Transportation costs
  • Facilitators' salaries
  • Information processing, which is often time-consuming

.

HOW IS A FOCUS GROUP INVESTIGATION CARRIED OUT?

.

HOW IS A FOCUS GROUP SET UP?

Choose the type(s) of focus group

For each type of focus group, the evaluator must specify:

  • The objectives of the focus group (to collect information, to analyse, to challenge opinions, to suggest, etc.)
  • The categories of participants to be gathered
  • The preliminary tasks to be completed in preparation for the focus group
  • The most appropriate way to conduct the sessions

Schedule the implementation of the focus group

Two elements must be taken into account:

  • Preparing focus groups may be time-consuming, especially for the selection of participants, the planning, and the organisation of the session. This is particularly true for focus groups of beneficiaries.
  • Focus groups relating to an impact study of a policy or a programme should be implemented only after the evaluator has developed his/her initial hypotheses and analyses, which implies significant preparation.

Within the allocated time, the evaluator may have difficulty in dealing concurrently with the methodological quality of the focus group and implementation delays imposed on him/her.

Identify the strategic interests groups

Preliminary documentation analyses and interviews are necessary before the focus groups can be set up.

This work helps the evaluator to determine the various strategic interests groups among categories of stakeholders and beneficiaries. These groups usually identify themselves by their expectations, their points of view, their social perceptions and their specific strategies relating to the policy or the programme under evaluation.

Select the participants

Once the strategic groups are identified, the evaluator must select from them the people who will be invited to take part in the focus group. To do so, the evaluator needs a mediator or a local key informant, particularly when representatives of populations living in remote areas, or ones who only speak in dialects are invited.

The selection of the participants depends on their availability, the equipment and human resources (an interpreter, for example) at the disposal of the evaluator.

In addition to the moderator's skills, which play a crucial role in the overall conduct of the session, the group interaction must also be considered. It can be fostered by one or more leaders who often intervene in the debate and stimulate the other participants. The moderator and the key informant should thus ensure that such leaders participate in the session.

The participants' motivation to become actively involved in the focus group should have been stimulated by an in-depth presentation of the topic of the session. They should understand the principles underlying the session's process, and think about the topic before the session. This is particularly recommended for focus groups with users and beneficiaries, who are not familiar with this type of approach.

Construct the moderator's guide

The moderator's guide should be designed sufficient in advance to allow the moderator time to study it. The evaluator should specify the topics under study and the themes to be developed during the session, although participants are free to develop other questions linked to the suggested themes. Suggestions should therefore be highlighted in the text (including the information which is to be collected by the moderator).

Advice:

  • Summarise the questions: the list of questions should be short. The moderator's guide should be a memento more than a questionnaire. The moderator should adapt the formulation of the questions to the reactions and statements of the participants. Thus, some questions prepared in advance by the evaluator will not need to be submitted to the participants specifically, because they will have discussed these questions spontaneously.
  • Avoid questions expressed in an alternative form. Experience has shown that only the first part of the question is taken into account.

Plan the focus groups meetings

This stage mainly depends on the availability of:

  • The people invited to join the focus groups
  • The evaluation staff
  • The key informants

Planning the focus groups also requires sufficient progress in the preparatory work relating to the focus group implementation.

WHO IMPLEMENTS FOCUS GROUPS?

Usually, the moderator (or the facilitator) implements the focus groups. He/she should be selected for his/her capacity to create and maintain the group interaction. The quality of the focus group depends on this capacity, as well as a focused and relevant exchange between participants. He/she should therefore be well informed of the evaluation's topics and goals.

What are the necessary skills?

Focus groups must be conducted by a skilled professional who:

  • Thoroughly comprehend the evaluation's major topics and issues
  • Is familiar with all the techniques relating to group interaction
  • Speaks the language of the participants
  • Is sufficiently aware of the participants' social and cultural characteristics, in order to:
    • respect their traditions and customs,
    • understand any implicit chains of command among them,
  • Avoids power games or social in-fighting between the participants

How should the moderator be trained?

Although moderators are selected for their abilities to moderate a group, few of them will really know how to moderate a focus group. They should be introduced to the topics and goals of the evaluation, and be trained in the type of techniques which will ensure that the focus group process is successful. A training day should be scheduled.

This training day includes the presentation of the moderator's guide, indications about the goals of the focus group and how the evaluator wants it to be conducted. Before the session, the evaluator should check with the moderator that all the terms and expressions used in the moderator's guide are understood.

Advice to the moderators:

Example (taken from the tool's testing mission in Benin)
Prior to the focus group session, the moderator indicated that he did not understand the following question: 'Is the reason which stops women from going to health services the cost of the treatment or the available financial means?'. After a discussion with the evaluator, the final wording became: 'Is the reason which stops women from going to health services the price of the treatment or the availability of money?'. This example underlines that the differences between international English and the local English language should be known, and abstract words should be avoided or at least properly explained to the moderator.
  • React to statements, in order to obtain more information
  • Ensure that all participants have sufficient time to express their opinions
  • Present a summary for each topic developed, and ask the participants to react to the summary
  • Reformulate the questions in case the participants have not properly understood them
  • Take body language into account (nod of the head, laughs, etc.)

What are the moderator's tasks?

Guide the sessions
  • Describe precisely the issue(s) to be discussed without suggesting solutions
  • Provide details about the original situation
  • List all the hypotheses
  • Focus the discussions on the major points from the interview guide
Supervise session progress
  • Make a general review of the group's findings
  • List the issues raised by the discussion by order of importance
  • Rephrase ideas which are not clear
  • Maintain the momentum of the debate
Encourage the discussion
  • Establish the reasons for any impediment to free expression within the group
  • Avoid tension
  • Control "leaders" and encourage "shy" participants
  • Take a full account of group silences and hesitations

What are the tasks of the local key informant or intermediary?

The participation of key informants during the selection of participants is often helpful to the evaluator.

The interaction within the group depends on its composition, and of the presence of one or two very active participants, who can foster the debate. The key informant can ensure that such individuals are included in the group.

The key informant can also prepare the participants by explaining the organisation of the session, the topics and the objectives of the focus group.

What are the observer's tasks?

In addition to the moderator's activities, an observer can be invaluable to the evaluation by keeping track of the opinions expressed (shared and contradictory) during the session.
The observer can also replace the moderator in his note-taking task and allow him more time to concentrate on his core duties.

HOW ARE THE PARAMETERS OF THE GROUP DETERMINED?

How are the groups designed to match the evaluation's purpose?

Determining the group's composition in support of the evaluation's purpose

 

When this tool is used to help a group articulate its of points of view, two possibilities can be considered for the composition of the group:

Groups constituted by people sharing similar opinions and professional activities

When the group comprises socially homogeneous members, participants can speak freely. The debate, however, can be stimulated when participants do not have identical views on the questions. Differing analyses, viewpoints and experiences can be challenged on the basis of a common understanding.

Example (from the Benin mission)
During the mission, four focus groups were organised. The most efficient one gathered a group of doctors belonging to the private and public sector of the administrative zone of Cotonou. The group was socially homogeneous, but the experiences of each doctor differed because of the location of their activities and the institution for which they work.

A focus group with a homogeneous composition is used to probe information and points of view expressed by stakeholders sharing the same strategic interests (in terms of social and professional status). Thus, the focus group provides the evaluator with an overall view of the experiences, the opinions, the behaviour and the needs of participants.

Groups constituted by people having different opinions and professional activities

In this case, the evaluator expects that the participants' analyses and points of view about the issue will be divergent. Consequently, the focus group is used as a means to express these divergences publicly. Each category of participants is given the opportunity to explain and defend his/her opinions. The focus group is a means to collect statements about perspectives on an issue. The evaluator can draw the participants' attention to the similarities, the differences and the complementarities of their various points of view. The momentum of this divergent discussion provides the evaluator with elements for new analyses and insights relating to the causes of these divergences.

Specific cases

The composition of the groups may sometimes alter the content of the information and the analyses under discussion.

Example

  • In hierarchical societies, categories of people may have difficulties in expressing themselves in front of their superiors, not to mention challenging the general opinion. This aspect accounts for the potential artificiality of village sessions about impact analysis. Yet, the evaluators tend to under-estimate this social context.

The evaluators can also set up groups of agents and officials working for the same institution. In this case, they should be careful about the impact of a point of view expressed in public.

Example

  • In poorly democratised societies, participants expressing themselves publicly at the request of the evaluator may take risks.

The participants can intentionally redirect the focus of the discussion for different reasons. They can speak solely amongst themselves, or they can defend the strategic interests of their group and attempt to manipulate the evaluator, or even accuse him/her of bias.
Thus, the evaluator must always be wary and keep control of the focus group's process. He/She must particularly avoid letting an individual participant dominate the debate, or allowing a coalition to emerge which dominates the group. He/She must also monitor the substance of comment and be fair in the allocation of time for expression.

How is the size of the group determined?

The size of a focus group varies greatly. However, it should not exceed 10 or 15 people, in order to let everybody talk effectively.
In the case of numerous groups, the evaluator should organise the participants into sub-groups for at least part of the evaluation's course. Consequently, he/she must have enough moderators at his disposal for each sub-group. Conversely, a focus group of less than 4 or 5 participants is difficult to manage, due to the lack of group interaction which constitutes the added value of this tool.

HOW IS GROUP INTERACTION FOSTERED?

Presenting the team in charge of the focus group to the participants

Prior to the focus group session, the context of the evaluation and its objectives should be reviewed and the team should be introduced. This will help avoid misunderstandings which otherwise may change the content of the participants' statements.

Example (from the Benin mission)
During the focus group with the women of a village in Porto-Novo, participants strongly criticised the public health utilities, thinking that the focus group was organised by representatives of the health centre. The moderator had to present the objective of the focus group and the position held by each member of the moderating team again, so as to collect properly focused statements.

The evaluator must therefore remember that when he/she organises a focus group with beneficiaries, he/she is perceived as the representative of international donors, which makes him/her an agent for dysfunctions highlighted by participants. The evaluator should be careful about participants' natural tendency to consider the evaluator as a means of communicating with other groups (of civil society or the political arena). This is not a handicap for the group interaction, however, because the exchange between participants is livelier than a collection of personal experience statements.

Which methodologies should be selected to conduct a session?

Several techniques for conducting focus group sessions are at the disposal of the moderator, to avoid boredom and retain the attention of the participants, and to develop the progress of group interaction.

Focus groups are not just the sum of individual interviews. Thus, the moderator must always initiate and maintain a dynamic interaction between the participants. The quality of the interaction depends on the moderator's skills to conduct a focus group.

  • Focus groups organised in a reactive way: the evaluator provides the participants with information, analyses and suggestions, which support the collective process of thinking and debating. The purpose of this method is to collect information and validate the conclusions and recommendations of the evaluator.
  • Focus groups organised in a pro-active way: the information and testimonies of the participants support the development of collective analyses and suggestions.

Recommendations about the moderator's behaviour

What the moderator should encourage and avoid during a focus group
 

Do Don't
Be careful about his/her general attitude towards the participants: be relaxed and focussed at the same time, stay concentrated Interrupt the participants abruptly
Show his/her interest in the participants' comments with expressions, such as "I see", "I understand" Show his/her surprise, impatience or disapproval
Avoid taking the opinion of a single person as representative of the whole group Express his/her own opinion
Prompt shy participants to express themselves Let a few individuals dominate the discussion, turning it into succession of long monologues
Let the participants interact freely, on condition that they discuss the relevant topic and that everybody can hear what is said Allow the group dynamic to run out of control
Ensure that the participants speak one after the other Always fill the silence (silence may reveal a particular group effect which should be carefully noted)
Control the general course of the discussion and ensure the free expression of divergent opinions Let the participants express themselves aggressively, make unsupported allegations, etc.
Address a question to the focus group first, and then to a participant (willing to speak or not). After each topic has been developed, ensure that each participant has had the time to express himself/herself with questions such as "does somebody have something more to say?" When the group does not react, take turns at questioning each participant, without trying to change the rhythm of the session during the focus group
React positively to a statement to obtain more information, and systematically sum up what has just been said Feel limited by the order of the moderator's guide questions
Reformulate the questions when they do not yield the outcomes expected by the objectives of the focus group Let questions go unanswered which are relevant to the objectives of the focus group
Take into account the body language and ask the participants to explain it Ignore the various types of expressions of the participants

Focus groups do not yield quantitative data and are not representative of the whole target group (beneficiaries, for example). Thus, the evaluator should remember that the expression of personal opinion is susceptible to a group bias (need for belonging and recognition, consequences of leadership, exaggerated assertions, artificial conflict originating from the debate on various opinions, etc.).

How to manage the tempo of the focus group?

Several elements must be taken into account:

  • Schedule the focus group's duration: it should last half a day as a minimum, because a group needs time before yielding collective results
  • Organise the session into stages, including a mid-term debriefing to the group. Regular debriefings are necessary to record and confirm the various assertions and points of view from the participants
  • Quickly create group interaction in order to go beyond the logic of "sum of individuals". The evaluator must therefore develop a confident and relaxed mood for the session, a balanced expression of opinions among the participants, etc.
  • Maintain the momentum of the group's interaction by frequently changing the dynamic of the focus group

The same session can be sometimes reactive to the outline of an analysis, information, etc. and sometimes pro-active, i.e. the group is able to yield rational information and analyses by itself.

Which type of interaction should be encouraged?

Proactive participation needs to be very structured. To this end, the moderator can use visual or audio displays, and introduce practical exercises, simulations, and time dedicated to syntheses and summing up, etc.
The moderator can plan the progress of the focus group in advance. However, anticipating the quality of the interaction and the effective tempo of the group is very difficult.

HOW TO KEEP TRACK OF INFORMATION?

Taking notes on a board, or using other collection devices during a session can help the group to keep track of the information debated and the subsequent synthesis.

The evaluator must conclude the session with a debriefing of what has been said, in order to check for any disagreements or misunderstandings. This stage should be kept informal, in order to avoid time-consuming formal validation.

The moderator or the observer should categorise participants: people who often express themselves and people who seldom do; sub-groups conversing with other sub-groups and sub-groups who seldom do, etc. These notes are important to put the verbatim comments in its proper context.

After the session, the evaluation team compares notes, determines and fills any missing points. The evaluators should also study the group interaction: if it has been impeded, they will have to restructure future focus groups.

A good way to keep track of the comments concerning the organisation of the session and the content of the discussion itself is to record the session. The observer can collect all the verbal expressions, while focusing on the most important aspects (which will be developed in the report of the session) and intervening in the session when the moderator omits questions or when the questions are not properly answered.

This is particularly recommended for focus group conducted in a local language, and where notes are generally taken in another language (French, English, Spanish, etc.). Indeed, the session's verbatim record is often lost during this instant translation, and the recording fills the missing parts of the notes. The evaluator should plan an additional working day in the focus group schedule, dedicated to the transcription of the notes.

HOW ARE THE FINDINGS ANALYSED?

This stage is highly dependent on the quality of the focus group's organisation.

In structured focus groups

The best way to analyse findings from structured focus groups is to gather all the answers and identify the people in the group who have defended certain opinions on a particular question, and the people who have opposed to these points of view.

Thereafter, the findings can be easily organised into sub-groups and the evaluator can use them directly.

In less structured focus groups

These focus groups tend to be a conversation between several people. The participants express many views but can agree on only a few conclusions. Thus, the analysis of the findings is similar to the one carried out for interviews.
The findings of the focus groups represent the sum of individual opinions and can be used as such by the evaluator. But focus groups have more advantages than interviews: they provide individual opinions expressed publicly, and consequently, they are more substantial because other participants can immediately challenge them.

WHAT ARE THE FINANCIAL REQUIREMENTS FOR ITS USE?

The full cost of a focus group can be fairly small. However, cost varies with the number of sessions and the difficulties of organisation (distances, etc.).
As a whole, a focus group is less expensive than a survey.

Elements of cost to be taken into account

In addition to the remuneration of the evaluators' working days and their travel expenses, the budget must also include:

  • The remuneration of the moderator, if the evaluator is not able to conduct focus groups himself/herself
  • The local moderator's training time and its cost
  • The remuneration of the mediators, if long-distance sessions are scheduled
  • The remuneration of the interpreters
  • The participants transportation expenses
  • The catering and logistical expenses (room rental, etc.)

Should the evaluator pay for participant's time?

Some evaluators provide the participants with a per diem, in addition to the transportation and catering expenses. The award of a per diem may lead to bias in the responses of participants, which can undermine the validity of the findings.

However, depending on the participant's occupation (for example, a shopkeeper or a farmer), the corresponding loss of income of a session lasting half-a-day or a full day can be financially damaging.

Thus, the evaluator should be pragmatic and provide only a limited number of participants with financial compensation. The compensation should be exceptional and the amount low, because the respondents should not participate in focus groups for money, or get the impression that their opinion is being "bought".

.

IN COUNTRY/REGION EVALUATION

.

WHY USE A FOCUS GROUP IN THIS CONTEXT?

In country/region evaluation, the main challenge is not how to collect information, but to determine what information to look for, from whom, about a wide range of questions, and during a limited period of time.

What are the specific difficulties in country/region evaluation?

The selection of beneficiaries and the understanding of their strategic goals

One of the country/region evaluation's purposes is to determine precisely the categories of stakeholders who have benefited from the implemented policies in some way. Thereafter, the evaluation must provide an analysis of the various stakeholders' strategies, and rationales considered, within the context of the evaluated policies.

Numerous stakeholders

In country/region evaluation, setting up meetings with numerous and varied stakeholders (such as institutional officials, implementation operators and agents, beneficiaries, etc.) can be challenging.

Sample preparation

The sampling of individuals and institutions from whom the evaluator collects information and points of view can yield insufficient results. Indeed, for reasons of timetable management, this selection is often carried out at the beginning of the evaluation process, and may not be adapted to the evaluation's purpose and strategic problems.

Lack of time

Usually, the evaluators spend most of their time interviewing officials, to the detriment of impact studies with beneficiaries which are particularly difficult to carry out in country/region evaluation.
In addition, time and budgets allocated to the evaluator are usually insufficient to conduct a series of focus groups with all categories of stakeholders potentially or actually involved in the evaluated policies.

To what extent is the focus group adapted to these difficulties?

Historically, focus groups have seldom been used in country/region evaluation.
However, evaluators should consider using focus groups more often at each stage of the evaluation, and particularly in the definition of strategic topics and issues, and for the selection of target groups.
In a country/region evaluation context, focus groups can be used to:

  • Identify and analyse major topics and issues
  • Provide a wealth of information, quickly collected
  • Conduct an impact study

Dependant on the nature of the evaluation (ex-ante, mid-term or ex-post), the evaluators deal with these three tasks in a different way. For example, in an ex-ante evaluation, a focus group is conducted to highlight the needs and priorities of the stakeholders who may be affected by a programme or a policy. In this case, impact studies are not a priority. They may be useful, though, to assess past policies or the impact of funding institution's activities, with a view to improving the drafting of new policies.
In a mid-term evaluation, a focus group is conducted to collect opinions and suggestions from operators and beneficiaries. In this context, its purpose is to detail any positive elements or dysfunctions, with a view to supporting recommendations for the policies in process.
As focus groups are flexible and multifaceted, they can be adapted to the various types and stages of the evaluation. However, their use should be clearly defined, in order to meet the evaluation's requirements.

HOW TO TAKE ADVANTAGE OF THE MAIN FEATURES OF THE FOCUS GROUP?

In response to the particular difficulties of country/region evaluation, focus groups concentrate on three objectives.

Atool for defining and analysing major topics and issues

In this context, focus groups can be combined with interviews.

First stage: interviews

The evaluator carries out highly focused interviews, in order to better comprehend the purpose of the evaluation. They are organised with people:

  • Who have an overall perspective of the policies, objectives and impacts to be evaluated
  • Whose interest and activities are diverse, in order to avoid bias in the evaluation process arising from over-reliance on officials who are responsible for policy implementation and monitoring
Second stage: focus groups

On the basis of the findings of these preliminary interviews, the evaluator conducts one or more focus groups with:

  • Stakeholders affected by and officials in charge of policy definition and implementation
  • Stakeholders sharing an overall view of the policy impacts

As a consequence, the categories of people who have been interviewed individually at the first stage are increased.
The objective of this second stage is to investigate collectively the interview findings and to explore a first draft of the strategic analyses which will support the evaluative questioning.
This type of focus group can be conducted at different geographical locations and on a variety of sectors. It provides the evaluator with a speedy means of interacting with a large number of people.

A broad and rapid information collection tool

This tool obtains information:

  • From different key stakeholders about the policy context
  • From programme operators about programme implementation

A tool for the analysis of impact

Its major objective is to determine which categories of stakeholders have been affected by the policy (positively or negatively).
These categories of stakeholders should not be limited to the programmes' final beneficiaries. The evaluator should also consider stakeholders who have held various responsibilities for the policy, those who have benefited from it, and those who are concerned with it in other ways.

Beneficiaries not identified at the policy definition stage may also be included in the evaluation process. Consequently, in country/region evaluation, impact analysis also deals with a series of intermediary stakeholders, from decision-makers to final beneficiaries.
Designed as a tool to analyse the impact of implemented policies and to complement series of interviews, the focus group can be conducted in three ways:

Three types of focus groups
 

Types of focus groups Types of participants' gatherings
Focus groups designed to identify groups and their strategic interests Heterogeneous groups of stakeholders and/or beneficiaries, whose interests are not yet identified
In-depth focus groups with homogeneous strategic interests Distinctive groups of stakeholders and/or beneficiaries, sharing identical strategic interests, but different (sometimes opposing) interests between one group and another
Focus groups designed to challenge opinions Groups artificially constituted, gathering representatives from the various strategic interests groups

.

EXAMPLES

.

Evaluation methodological approach > EN: Methodological bases and approach > Evaluation tools > Focus group > Check lists

Check lists

CHECK-LIST FOR EVALUATORS

.

Questions Answer
Preparation and design  
Was the use of the focus group fully justified?  
Have the topics under study been clearly determined before the setting up of the focus group?  
Has reference documentation been at the disposal of participants?  
Have local speaker animators experienced in techniques relating to group interaction been selected?  
Were participants informed prior to the focus group of the objectives and the topics under study?  
Implementation  
Were the animators informed of the context in which the focus group is organised?  
Were they trained for the topic and goals of the focus group?  
Has the neutrality of the animators concerning the issues of the focus group's topics been checked?  
Has the verbatim of the participants been collected?  
Does the debriefing clearly distinguish the factual information from opinions?  
Does the debriefing accurately describe the diversity of points of view and opinions developed by the various stakeholders?  

.

CHECK-LIST FOR MANAGERS

.

Questions Answer
Was the use of the focus group fully justified?  
Have the topics under study been clearly determined before the setting of the focus group?  
Does the debriefing clearly distinguish the factual information from opinions?  
Does the debriefing accurately describe the diversity of points of view and opinions developed by the various stakeholders?  
Is the presentation of the various stakeholders' points of view thorough and explicit?  

Evaluation methodological approach > EN: Methodological bases and approach > Evaluation tools > Survey

Survey

SUMMARY

.

Why is this tool used in evaluation?

A survey is an observation tool which quantifies and compares information. Data are collected from a sample of the population targeted by the evaluation.

A survey questionnaire is a schedule of questions collecting information needed for the evaluation. Respondents are not supposed to answer the essential issue under investigation: in a good questionnaire, questions derive from the evaluation questions and are sufficiently basic for the respondent to answer them.

Questionnaires often combine both types, with a preference for structured items and a few open-ended questions (yielding information which is more diverse and/or precise, but less amenable to statistical analysis). 

Structured questionnaires

Structured items are questions which respondents must answer in a specific way by choosing from a limited and predetermined set of answers. The questionnaire format is designed to obtain information about facts, to find out whether respondents agree to a suggestion, to record their opinions on a set of assertions, etc. 

Open-ended questionnaires

In open-ended questionnaires, respondents answer a precise question and interviewers take notes. Thus, open-ended questionnaires are similar to structured interviews, as open-ended items allow a variety of approaches and depth in response.

What use can be made of the survey in evaluation country?

A survey is the best tool for collecting information from the population, and to compare and quantify the various opinions expressed. Its use is particularly relevant to know the final beneficiaries' degree of satisfaction concerning a policy.

Structured questionnaires have the advantage of allowing a cost-effective statistical analysis. Yet, they can be unfitted during the survey's implementation course when the evaluator needs to refine some questions.

Through a daily check with the interviewers, the evaluator can decide to develop or add questions during the interviews, with a view to undertake a more precise analysis.

How is a survey carried out?

Figure 1 : Steps involved in the use of survey
Figure 2 : What are the conditions for use of the tool in country evaluation?
Concerning the question wording Be familiar with the context and the programme's stakes for the beneficiaries prior to the evaluation.

Be provided with any relevant information to cover the survey's scope.

Concerning the sample design Be provided with the minimum statistical data.

Be sure of the physical and logistical access of respondents.

Concerning the survey's organisation Identify a solid local partner, able to provide for human and logistical resources.

Spend enough time to carry out all the preliminary stages dedicated to the questionnaire and sample design without which the findings may be disappointing.

An efficient time management is a prerequisite for the survey. During the tool's testing mission in Benin, the local partner was identified a month and a half prior to the field mission; the sample and questionnaire design, and interviewers training were organised on-site 2 weeks before the arrival of the testing team.

How is the questionnaire developed?

Relevant questions for the evaluators
  • What is required?
  • Is each question strictly necessary?
  • Will a single question be sufficient to obtain this information?
  • Is the respondent in a position to answer the question?
  • Will the respondent provide the evaluator with accurate information?
Structuration of the questionnaire
  • Define the investigation's topic and design the questionnaire precisely
  • Ask overlapping questions in order to check the relevance and coherence of the answers
  • Formulate clear and short questions for the respondents; incite them to take sides

How is the survey carried out?

Design the samples

The methodology selected depends on the determination of the population that constitutes the target group of the survey. This determination is linked to:

  • The purpose of the survey
  • The working hypotheses selected
  • The nature of the available documentation
  • The evaluation constraints
  • The degree of homogeneity of the population
  • The scope of the area to be surveyed

Various types of sampling can be developed: simple (random sampling), stratified, cluster sampling, probability proportional sampling, progressive, etc.

Questionnaires among households during the testing mission in Benin:

" The method to collect data is the itinerary method with which the interviewer can identify the survey's targets by counting the households while covering each street of the scoped area, register them in a household counting slip, proceed to the selection of the household samples and interview the head of family an/or his wife.

The counting should start from the chief of the village's house and progress clockwise, segment after segment, so as to cover the whole village and reach the number of households to be surveyed.

The number determining the first household to be surveyed is random and given by the survey's centre. The numbers of the other households to survey will be determined by the drawing number established by the survey's centre.

For example: if the first number is 3 and the drawing number is 5, the first household to be interviewed will be the third of the survey's list. The other households will therefore have the number 8, 13, 18, 23, etc. "

Conduct a pilot questionnaire

A good quality survey relies on the clarity of the question wording, the ease of response, the questionnaire's length and flow, problems encountered by the interviewers, etc.

How is the questionnaire conducted?

The choice of a specific type of survey depends mainly on the context:

  • In developed countries, questionnaires by telephone, email or Internet are increasingly conducted
  • In developing countries, face-to-face questionnaires remain the surest tool to obtain the information required. However, other types of questionnaire or tools may be used, dependant on the stakeholders' categories (for example, local agents for development)

The evaluator should keep in mind that, because of linguistic reasons (mostly with final beneficiaries) and time spent in the preparation of the survey, a local partner is strongly advised. The latter should be able to provide the evaluator with human resources - interviewers, statisticians, demographers, etc. - and material resources - transportation, IT, demographic data, etc. - all of which the evaluator may not have available on-site.

How are the findings analysed and treated?
 

  • Coding questionnaires: assign a code value to each response, which forms the input for data processing. Responses are gathered in relevant categories.
  • Counting findings and correlations: always refer to the working hypotheses.
  • Analysing findings: in the evaluation field, simple treatments are often sufficient for the analysis stage, even though establishing correlations between items enriches the analysis.

The analysis of structured questionnaires may not yield the expected results. During the testing mission in Benin, although the local partner was equipped for statistical analysis, the analysis was disappointing and less thorough than if it would have been directly managed by the evaluator.

What are the preconditions for its use?

Figure 3 : the preconditions for its use
The time span Carrying out a survey requires great care at the preparation stage, and an allocation of time in proportion to the importance of the survey, the extent of sampling and field difficulties.

The elaboration of the questionnaire dedicated to specific groups requires sufficient data and hypotheses, which means that the survey cannot take place at the start of the evaluation.

Human resources Where cultural and linguistic specificities are important, it is better to have locally recruited interviewers.

Specialist organisations may sometimes be able to support the evaluator's recruitment process.

The evaluator should organise one or more training/debriefing days for the interviewers.

Financial resources Remuneration of the interviewers.

Transportation expenses.

.

What are the advantages and limitations of the tool?

Figure 4 : The advantages and limitations of the tool
Advantages Yields quantified and reliable data 

Useful to identify changes and make comparisons between opinions 

Enables the evaluator to survey a large number of final beneficiaries 

Enables the evaluator to work on a target population and on a limited scale 

Identifies the outcomes of programmes and policies

Limitations Requires implementation delays exceeding the average time scale of an evaluation mission 

Requires important resources and logistics provided by a reliable local partner 

Requires pre-existing data on the initial situation 

Requires a large number of staff to conduct the survey and analyse the findings 

May present difficulties during the development of representative sampling

Evaluation methodological approach > EN: Methodological bases and approach > Evaluation tools > Survey > Detailed presentation

Detailed presentation

.

This section is structured as follows:

.

DEFINITION

.

WHAT IS A SURVEY?

What are the reasons for using a survey?

A survey collects data over broad populations, by means of a standardised set of questions.
A survey questionnaire is a schedule of questions collecting information needed for the study. It standardises the information collected in survey. Thereafter, this information is used to assess a change in a given situation. The selection of respondents depends on various methodologies aimed at developing a representative sample of the whole segment of the population considered.

Are there different types of questionnaire?

There are two main types of questionnaire:

  • Structured questionnaires
  • Open-ended questionnaires

Usually, questionnaires combine both types, with a preference for structured items and few open-ended questions (yielding information which is more diverse and/or precise, but less amenable to statistical analysis).

Structured questionnaires

Structured items are questions which respondents must answer in a specific way by choosing from a limited and predetermined set of responses. The questionnaire format is designed to obtain information about facts, to find out whether respondents agree to a suggestion, to record their opinions on a set of assertions, etc. Structured items do not collect detailed information, but as the categories of response are predetermined, they provide the basis for efficient statistical analyses.

 

Types of questions in structured questionnaires

Close-ended question options Number of suggested responses Number of expected responses Comment
Dichotomies 2 1 Easy to implement
Multiple choice with one response Set of responses 1 Differences of meaning between answers must be obvious
Multiple choice with multiple responses Set of responses Set of responses Statistical analyses are difficult to conduct. Each choice of questions should be considered as a dichotomy system in which respondents answer yes or no

 

In structured questionnaires, the respondent may be inclined to respond in a way which does not accurately reflect his opinion, because:

  • He/She thinks that his/her answer will please the interviewer or put him/her in a favourable light
  • He/She has chosen the 'least worse' answer at his/her disposal
  • He/She has been subconsciously attracted by an answer, for example, expressing an average opinion
Open-ended questionnaires

In open-ended questionnaires, respondents respond to a precise question and interviewers take notes. Thus, open-ended questionnaires are similar to structured interviews, as open-ended items allow a variety of approaches and depth in response.

 

Types of responses in open-ended questionnaires

Type of expected responses Response collection Analysis
Responses with figures (amounts, volume, etc.) Easy Easy, because the response is quantitative
Narrative responses The interviewer should allow space in the questionnaire or prepare a list of responses which can be ticked off as the respondent mentions them Difficult: coding the responses into keyword, or group of keywords could facilitate the analysis

Limitations of its use

The survey is an outstanding observation tool for the collection of information from a large number of final beneficiaries.

Although structured questionnaires can measure the relative weight of the opinions expressed, they must be conducted with a carefully constructed sample so as to be statistically valid. Thus, facilities and means are required, such as:

  • A sufficient number of interviewers to survey a large number of people without whom the collected data would loose its representativeness
  • Technical means, such as computer devices to undertake statistic analyses
  • Enough time to carry out the survey's three stages (drafting, implementation and analysis). The less the evaluation team knows the context (topic under study and surveyed population), the longer it takes to draft a relevant and useful questionnaire
  • Financial resources allocated to the organisation of a long field mission, the recruitment of a interviewer and a team responsible for the data analysis

The use of open-ended questionnaires can be an alternative to structured questionnaires when the evaluation does not need a statistical analysis of the data collected. Past experiences have shown that the information collected by this type of questionnaire eases the understanding and analysis of the population's reactions, and provides answers to the evaluation's requirements.

WHERE DOES THIS TOOL COME FROM?

Its origins

The use of surveys and sampling techniques emerged after the First World War, during electoral forecasting. Polls institutes were the first to use surveys. Nowadays, surveys and sampling techniques represent the social sciences' reference tools. The demands from quantitative sociologists have led to the development of surveys and sampling techniques to be able to benefit from the statistical data.

Its use in other fields

In addition to opinion polls institutes, the marketing field is a significant user of surveys. In anthropology and ethnology, observation surveys are designed as semi-structured interviews.

This type of survey can be used in some case studies, but is seldom used in evaluations due to constraints on time and budget. Indeed, these surveys are more useful for researchers, although their findings may provide the evaluator with insights into the general context and the socio-cultural dynamic of the region under evaluation.

.

WHY AND WHEN IS THE SURVEY USED?

.

WHEN AND HOW SHOULD A SURVEY BE USED?

In which situations is this tool appropriate?

An observation tool collecting quantitative and qualitative information

The survey is an observation tool which collects:

  • Stakeholders' needs, opinions, etc., especially in ex ante evaluations
  • Information about expected changes or those which have occurred during the programme. This type of survey is carried out in mid-term or final evaluations.

In developed countries, opinion pollsters and market researchers often conduct surveys because they yield quantitative data, whereas in developing countries, they are less common because of constraints of costs and difficulties in implementation. As a general rule however, surveys remain an appropriate tool to collect information in studies and evaluations.

Surveys require good definitions, and careful management of the information under investigation and of the people from whom the information is to be collected. Surveys should be implemented after a piloting stage.

Theoretically, surveys can provide good indications of change, if carried out with the same sample and within a period of time sufficient to yield information about the impact of a policy or a programme. In practice, however, these conditions seldom prevail in developing countries.

Warning about its use in evaluation

The evaluator will experience fewer difficulties designing a precise set of questions in project and programme evaluation than in country/region evaluation, where the process of identifying real beneficiaries and stakeholders can be difficult.

What type of information does a survey yield?

In evaluation, surveys are mainly used to assess the impact of policies and programmes on various categories of people.
Surveys collect:

  • Information about the socio-economic situation of various groups of people, which may support the design of typologies
  • Information about changes of situation and practice (provided by economic and social indicators)
  • Opinions, judgement and perceptions of a policy or a programme (provided by precise assessment criteria)
  • Analyses dealing with the causes of change, and with the impact of policies and programme upon these changes. The analyses start with questions supporting the preliminary hypotheses.

Surveys are carried out:

  • Face-to-face with the respondent
  • By telephone
  • By mail
  • By email and on the Internet

What are the advantages and limitations of a survey?

Advantages Limitations
Yields quantified and reliable data Requires important resources and logistics
Enables the evaluator to survey a large number of final beneficiaries Requires a large number of staff to conduct the survey and analyse the findings
Useful to identify changes Requires pre-existing data on the initial situation
Enables the evaluator to work on a target population and on a limited scale May present difficulties during the development of representative sampling
Identifies the outcomes of programmes and policies Yields only a simplistic description of reality

Despite the tool's advantages (especially comparing with other observation tools), limitations in the time span, the financial resources and the technical means can limit its use in evaluations.

Can a survey be combined with other tools?

Structured questionnaires bring complementary quantitative information to other information collection tools (which only yield qualitative data), such as:

  • Interviews
  • Focus groups

These tools can be used during the exploratory stage of a survey (to identify or detail working hypotheses) to ease the drafting of the questionnaire with:

  • The listing of themes to be studied
  • The determination of relevant questions

A series of interviews or focus groups can be organised prior to a structured questionnaire, in order to highlight the context of the outcomes, and to support the understanding of the data and its interpretation (for example, to describe a custom, or explain an individual perception).

Surveys' findings also support other tools which are more complex, such as:

  • Case studies
  • Socio-anthropological observations
  • Benchmark studies

WHAT ARE THE PRE-CONDITIONS FOR ITS USE IN EVALUATION?

The time span

Carrying out a survey requires great care at the preparation stage, and an allocation of time in proportion to the importance of the survey, the extent of sampling and field difficulties.
Generally, surveys are conducted after the start of the evaluation, when the evaluator has enough information and when his working hypotheses are sufficiently developed to support a questionnaire dedicated to specific groups of the population.

Human resources

Setting up a survey necessitates the appointment of interviewers.

Required skills for interviewers
  • Ability to quickly become familiar with the purpose of the survey
  • Ability to establish a relationship based on trust
  • Ability to detect possible misunderstandings and impediments
  • Ability to complete the questionnaire in a clear and reliable way
  • If necessary, speak a dialect and be aware of local customs
The recruitment
  • Where cultural and linguistic specificities are important, it is better to have locally recruited interviewers (such as development operators, teachers, technicians, etc.)
  • Specialist organisations may sometimes be able to support the evaluator's recruitment process
  • The evaluator should organise one or more training/debriefing days for the interviewers. Before the start of the survey, this training should ensure that the survey's objectives and the questionnaire are clearly understood, and a "team spirit" has been established. After implementation, a debriefing on the survey's process, the difficulties encountered, possible bias, respondents comments, etc., should be held. The output of the training will provide the evaluator with insights which will improve the analyses of the findings.

Financial resources

Most of the costs of surveys are remuneration of the interviewers and transportation expenses, if the survey is carried out using widely dispersed interviewees. It is not possible to specify exact amounts because expenses vary greatly with the type of country in which the survey is implemented.

However, the transportation budget is critical in countries where transport is scarce. Generally, in assessing the financial requirements, the following should be taken into account:

  • The type of survey which is being implemented (face-to-face, by mail, by telephone, by email)
  • The size of the samples
  • The remoteness of the area surveyed
  • The number of interviewers to be recruited and the duration of their contract, which depends on the elements noted above

Surveys belong to the category of tools which are the easiest to use, allowing the evaluator to obtain information from a large number of people. Surveys by telephone are cheaper than face-to-face surveys. Surveys by mail are the least expensive, although the response rate is usually low and limits information about the respondents profile, especially in countries where computer networks are poorly developed.

WHAT IS THE ROLE OF A SURVEY IN EVALUATION?

Why should it be used in evaluations?

Although open-ended questionnaires require skilled resources for drafting, the organisation of implementation and their analysis, the time spent in data collection on site is limited in most evaluations for financial reasons.

Yet, in certain cases, the evaluation questions require the collection of information from final beneficiaries. In this context, the questionnaire is crucial because it collects information about the viewpoints and practices from a large range of people.

When the completed structured questionnaire is analysed statistically, it quantifies the opinions surveyed, which can be supportive of an impact analysis, or the drafting of evaluation or programme indicators. This type of analysis is however difficult to undertake and requires more resources than in any other analyses.

What are the requirements for the use of a survey?

 

  • The evaluator's knowledge of the context and the programme's impact on the beneficiaries, so as to formulate relevant questions for the questionnaire.
  • An available statistical database on the social structure of the population, which should be sufficiently reliable to support the design of samples with a minimum risk of errors.
  • The identification of a reliable partner for the recruitment of interviewers, statisticians, and demographers, and the procurement of material resources (transport, computer devices, demographic data, etc.).
  • A context allowing for the easy selection and interviewing of a sample of respondents. The evaluator should also have access to respondents, suitable meeting locations, and means of grouping respondents together to maximise the use of time.

WHY IS THE SURVEY USED IN COUNTRY EVALUATIONS?

As surveys need specific conditions for their implementation in country/region evaluations, they have proved to be more appropriate for countries at a high level of development.

In developing countries, however, the evaluator may survey target groups to obtain information (such as quantitative data) which cannot be collected by other means.

In country/region evaluations, surveys have a selective purpose and focus on geographically defined categories of stakeholders. Consequently, the evaluator needs to closely supervise the interviewers, or delegate the supervision to a reliable local partner.

The evaluator can be confronted with continuing problems relating to the collection and speedy selection of the required information from abundant sources. In addition, it may be difficult to reach final beneficiaries. But, by interviewing a large number of beneficiaries, surveys can provide the evaluator with part of the information on the policy under evaluation, such as:

  • The progress of programme implementation
  • How the needs are taken into account
  • Lessons learned
  • Impediments, etc.

Currently, surveys and questionnaires are rarely used in country/region evaluations but as surveys can provide useful analyses of impact relating to various groups (and particularly beneficiary groups), the number carried out in the future should increase.

.

HOW IS A SURVEY CARRIED OUT?

.

HOW IS A QUESTIONNAIRE DEVELOPED?

What are the relevant questions for the evaluator?

What is required?

As the design of the questions depends on the target population, and on the information, the analyses and opinions which are sought by the evaluator, the questionnaire should start with a list of data about the study's topic, including:

  • The purpose
  • Working hypotheses
  • Target groups
  • The type of information which is required
  • The geographical setting of this information
  • The period of time in which the survey is implemented

This list organises the stages of the survey and the drafting of the implementation timetable.

Is this question strictly necessary?

Asking this implies that the purpose of a study can only be fulfilled if the question is posed. Therefore, if the answer is 'no', the question should not be included in the questionnaire.

Will a single question be sufficient to obtain this information?

Depending on the complexity of the information required, several questions may be necessary, which is particularly important for elements that seek to examine causality.
The evaluator must also be careful about asking questions with two parts. In these cases, the question should be broken into two separate and straightforward constituent parts.

Is the respondent in a position to answer the question?

Three main reasons may account for a lack of response:

  • He/She does not understand: the vocabulary is not appropriate, the question is too theoretical, etc.
  • He/She does not know: the questionnaire is not adapted to the target group, or the respondent has been poorly selected
  • He/She has forgotten: the evaluator should distinguish between the situation where the respondent can find the information and give it later, from the situation where the respondent does not have access to this information
Will the respondent provide the evaluator with accurate information?

This issue mainly deals with information whose nature may be sensitive, such as:

  • Socially sensitive topics, in which certain attitudes are likely to be rejected or changed (for example, this can lead to an under/overestimation in the survey)
  • Topics closely related to the respondent's private life
  • Topics demanding extra effort and time consuming inquiries

In these cases, respondents may be tempted to alter or avoid a topic. The evaluator may anticipate or avoid this reaction by the skilful choice of questions or by structuring the questionnaire.

How should a questionnaire be structured?

The questionnaire's basics
  • Determine the topic under investigation and focus the questionnaire's elements on the subject
  • Do not be over-ambitious about the questionnaire's capacity to obtain information: it should be short and straightforward, and the wording of questions should be simple and precise
  • Allow for overlapping questions, in order to check the reliability and the coherence of the responses
Introductory and transitional documents

These documents are necessary for the clarity and the natural flow of the questionnaire. They must be adapted to the survey's type, its topic, and the socio-cultural characteristics of the sample.
These documents must include:

  • "Indications relating to the topic under evaluation: 'the questionnaire's purpose is to determine…, to identify…, we will now study the issue of…, now that we have listed all the relevant points of…'
  • Technical indications: 'please tick from the list below any reasons that apply to you, check one box for each row, please write short answers …'
  • Polite phrases: 'your opinion is critical…, thank you for your kind collaboration'
  • Confidentiality/usage statement: 'this questionnaire is strictly confidential…, this survey will only be used for scientific purpose'
  • Short and understandable comments: they should be introduced only when necessary" (extracts from "Surveys, a handbook for practitioners". Jauveau C. 1992. Brussels University Edition)
Sequence of questions
  • Place the most sensitive questions at the centre of the questionnaire, to prevent avoidance, fatigue or loss of attention by the respondent if placed at the end of the questionnaire
  • Regroup attribute questions to the beginning or the end of the questionnaire. If they are very personal (dealing with income, religious practice or political opinion), place them in the core of the questionnaire.
  • Start with easy questions which give respondent confidence
  • Insert easier questions between difficult or sensitive questions, in order to make the respondents more comfortable. Their purpose is to facilitate progress through the questionnaire.
  • Plan transitional statements between topics which are unrelated
  • Group questions dealing with the same topic together
  • Rank questions:
At the beginning Towards the end
General questions Specific questions
Questions dealing with the respondent's acquaintances Questions dealing with the respondent's personal situation or opinion
Factual questions Attitude questions

 

  • Incorporate non-responses into categories such as "do not know", "no opinion", "refuse to answer". Beware: the respondent may favour these options to avoid responding to the question. The evaluator should determine a coding scheme for this type of response which must be classified as a regular answer.
  • Balance open-ended questions and close-ended questions. Open-ended questions provide developed answers, but are difficult to analyse statistically. The evaluator must consider when open-ended questions should be replaced by several close-ended questions.
  • Prior to its drafting, plan how the questionnaire will be used: its wording influences its structure (open-ended questions, close-ended questions, correlations between questions, etc.)

What is a typical format for a questionnaire?

Presentation of the interviewer and the survey

This stage, sometimes overlooked, is useful to introduce the questionnaire to respondents.
The evaluator may conceal or slightly change the description of the study's topic if he/she considers that it would otherwise provoke the respondent's negative reaction (such as bias, superficial response, concealment, fear, refusal, etc.). The topic of the study may need to be rephrased into simpler wording, and presented in a less "institutional" format.

Introductory questions

Introductory questions aim at arousing the interest of respondents and establishing a rapport in the case of a face-to-face survey. They convey the general scope of the topic and are formulated to obtain a positive response, even though this response may not be directly useful for the survey.

Background questions

Background questions are used to check whether a respondent has the required information or, if necessary, give the interviewer contact details of an informed respondent. Background questions often take the format of dichotomies.

Warming-up questions

Warming-up questions contribute to progressively focusing the questionnaire on the topic of the study. Usually, they take the form of straightforward questions (for example, behaviour questions).

Specific questions

Specific questions represent the core of the questionnaire: responses to these questions must provide the study with critical and focused information. At this stage, the respondent should be completely at ease with the interviewer and concentrate on the topic. The interviewer may ask him/her personal questions, or press him/her to reflect upon his/her motivation, opinions and judgements.

Attribute questions

With the help of responses to attribute questions, the evaluator describes respondents and incorporates them into sub-groups. These questions yield two types of information:

  • Socio-demographical variables (gender, age, profession, etc.)
  • psychological information (values, opinions, etc.)

How should a questionnaire be written and presented?

The questionnaire's basics
  • Short, straightforward, clear and unambiguous questions
  • Question wording must not influence the response, for example by implying a positive or negative answer
  • Question wording should take into account possible translation, which requires the quality of the wording to be high
The vocabulary
Do use Don't use
Usual vocabulary Technical words
Clear wording avoiding double meanings Ambiguous, subjective or confusing words
Precise temporal adverbs Vague adverbs: often, seldom, from time to time, occasionally, etc.
Vocabulary adapted to the target population Words which can be understood differently, depending on the region, the social status, the age, etc.

 

The style
Do use Don't use
Simple and concise style: the question should not exceed a line and a half Negative or double negative questions: they influence the answer
Neutral terms Leading questions: they influence respondents who answer "yes" when they are usually not so assertive
Variation in direct and indirect style: if the topic of the question is too personal, indirect style should be favoured Personal questions which are phrased too abruptly

 

Ways of introducing the questionnaire to respondents
This stage influences the quality of the survey's findings and should not be overlooked. The most effective way to introduce the questionnaire to each respondent of the sample is to send a letter or an email which should state:

  • The nature of the body and the department/service conducting the survey
  • The disclosed purpose of the study
  • Inducement to participate: interest for the community, research, etc.
  • Instructions about how to return the questionnaire if it has been mailed (closing date, contacts, etc.)
  • The signature of an official of the body conducting the survey

The cover letter should be carefully prepared, especially its general appearance.

HOW ARE DESK-BASED SURVEY TECHNIQUES CARRIED OUT?

Why are desk-based survey techniques carried out?

The evaluator develops a line of questioning supported by his working hypotheses, with a view to designing the questionnaire. In many cases, pre-field techniques are used to validate these hypotheses and the resulting questions. More generally, they are carried out to improve the questionnaire, prior to the start of the survey.

Techniques for the validation of the questionnaire

Complementary techniques may be used to validate the questionnaire and the whole drafting process. These techniques include:

  • Studying the available documentation about the subject under evaluation
  • Setting-up a group discussion (a brainstorming meeting) with field test interviewers and, if necessary, people having insights into the subject but not involved in the survey process
  • Conducting a focus group, with a view to interviewing competent people about the line of questioning
  • Conducting a debriefing session, with a view to collecting critical information for the questionnaire wording

These techniques enable the evaluation team to agree on the same definition of the study's key concepts. This stage is necessary for the questionnaire wording process.

HOW ARE SAMPLES DESIGNED?

The constraints of the population selection

The selected methodology depends on the determination of the population that constitutes the target group of the survey.
This determination is linked to:

  • The purpose of the survey
  • The selected working hypotheses
  • The nature of the available documentation, such as lists, updated or not, etc.
  • Constraints such as financial and human resources, delays, need for translation, transportation, etc.
  • The degree of homogeneity of the population
  • The scope of the area to be surveyed
Basics for the sampling technique

Sample quality is key to the reliability of the findings. The sample should reflect as closely as possible the whole population. It should be a scale model of the population.
To be analysed statistically, a sample should gather at least 100 people. The less the evaluator knows about the population under study, the more numerous the sample should be in order to be representative.

The evaluator's dilemma

In programme or co-operation policy evaluation, samples are often hard to design: registry documents, professional index, etc., relating to the population under study are seldom available, and information about the population's characteristics are often non-existent, unavailable or unreliable. The situations in which the survey is to be conducted are often very difficult.
In this context, the evaluator must implement surveys based on numerous samples, which can be very expensive. Otherwise, he must restrict the survey's field of inquiry to very specific categories of people, which leads to problems of reliability.
The evaluator may conduct several very focused surveys, providing that the analysis of the group of stakeholders has been correctly carried out during the pilot stage.

The various types of sampling

Simple sampling
  • Respondents are randomly selected among a pre-determined population during a single selection. Everyone has the same probability of being part of the sample, and each respondent can be selected only once.
Stratified sampling
  • Stratified sampling is used when the survey deals with a heterogeneous population. The process consists of dividing the population in homogeneous sub-groups, and of drawing from each of them a simple sample to carry out the survey.
Cluster sampling
  • Cluster sampling could be useful if the survey covers a wide area. The area is divided in similar segments (clusters). The evaluator selects some of them and questions all their inhabitants. This technique may also be used to survey people belonging to a representative sub-group, for which the evaluator does not have systematic data.
Probability proportional sampling
  • Probability proportional sampling consists of developing a scale model of the population under study, which should represent its characteristics. Specific quotas of respondents for each category of characteristics are determined. Thereafter, the evaluator identifies the people illustrating these characteristics, and selects an appropriate number of respondents.
Progressive sampling
  • The evaluator designs the samples progressively. First, he/she identifies people matching precise characteristics. Second, he/she investigates them. Finally, he/she asks them to name other possible respondents. Progressive sampling is used in social network surveys, as well as in situations where the evaluator does not have a list of people conforming to specific characteristics.

How is the size of the samples determined?

Basics: the findings' accuracy increases with the number of respondents.
The size of the sample is a compromise between the required precision and the available budget.
An equilibrium must be found between:

Therefore, more extensive sampling than the survey's requirements must be scheduled, so as to anticipate possible refusals, non-attendance of a session, disruptions, etc., which often occur during the implementation of a survey.

The respondent list

This list should include the respondent's name, profession, contacts, and, if required, his selection characteristics.

HOW IS THE PILOT QUESTIONNAIRE CONDUCTED?

Why is the questionnaire piloted?

A good quality survey relies on the piloting of the questionnaire and of the survey's methodology, in order to identify problems relating to the clarity of the question wording, the ease of response, the questionnaire's length, problems encountered by the interviewers, etc.
A limited number of people (usually 10 to 30) is selected to be the "sample for the sampling" and to test the questionnaire before its use.

The content of the pilot questionnaire

  • Understanding of the terms used: if sentences have to be explained, the evaluator should rephrase them
  • Complexity of questions: should they be split up? Should repetitive or investigatory questions be included in the questionnaire?
  • Sequence of questions: does it provoke negative reactions (such as surprise, suspicion, boredom, avoidance, refusal, etc.)?
  • Question wording: does it lead to the expected answers?
  • The questionnaire's length: is the questionnaire tedious, irritating?
  • The questionnaire's flow: are introductory and transitional texts sufficient and efficient?

Because surveys are expensive and difficult to administer (constraints of time), they are not usually systematically piloted during an evaluation, although this could add a qualitative dimension.

HOW IS THE SURVEY CARRIED OUT?

The different types of survey

The choice of a specific type of survey depends mainly on the context:

  • In developed countries, questionnaires by telephone, email or Internet are increasingly conducted
  • In developing countries, face-to-face questionnaires remain the surest tool for obtaining the information required. However, other types of questionnaire or tools may be used, dependant on the stakeholders' category (for example, local agents for development).
The various types of questionnaire and their particular characteristics

Type of questionnaire Comments
Face-to-face Face-to-face questionnaires ensure the best control over the sample. Some contexts only allow this type of questionnaire: absence of means of communication, linguistic impediments, need for direct contact to ease mutual understanding, etc.
By mail The questionnaire must be carefully designed: respondents read and answer the questionnaire alone and may give up quickly if they encounter any difficulties. Thus, the survey's presentation and the cover letter must be attractive.
By telephone Advantage: can quickly yield information, but difficult to retain the respondent's attention. Sending a cover letter prior to the interview, informing respondents about the questionnaire's content may reduce this difficulty.
By email Specificity: the use of the electronic communication is available only to a selection of a specific population. The questionnaire is sent through the Internet and is completed on-line. This technique is time-consuming for respondents, but helps the evaluator by facilitating data processing.

 

Advice

The carrying out of surveys with beneficiaries may pose logistical problems, especially in far-reaching areas. Among other distortions, the evaluator must be careful to avoid an over-representation of the most accessible areas (for example, nearby main roads), to the detriment of remote areas which may never be selected in field inquiries. If these elements are overlooked, the evaluator may miss specific elements of social organisation, such as access to equipment and utilities.

HOW ARE THE FINDINGS ANALYSED AND TREATED?

Coding questionnaires

After the receipt of the completed questionnaires, the code development stage can begin. This stage is necessary for open-ended questions which call for statistical analysis. Codes should reflect the type of analysis which has been selected.
The coding of findings consists of assigning a code value to each response, which forms the input for data processing. Codes present a straightforward interpretation of a free response (to an open-ended question) in numerical terms, and the responses may be a fact or a judgement.
The treatment of open-ended questions requires a very careful work. Responses must be grouped in relevant categories.
The determination of the way to code responses falls to the person or the team responsible for the analysis of findings.

Counting findings and correlations

After coding the questionnaires and entering the codes in a data processor, the evaluator can process the responses which should yield findings corresponding to the working hypotheses.

 

The treatment of responses

Counting Simple counting of the responses to a question
Dependency or correlation The evaluator examines how two or more sets of responses, a response and an attribute (gender, age) are linked together. If the correlation coefficient is high, some explanatory variables are determined. However, the evidence of a correlation should not be interpreted as causality.

 

Analysing findings

The analysis of the findings relates to the working hypotheses. Pre-field techniques and documentation collected are also used during this analysis stage. Going back to the original responses may sometimes be useful to investigate certain elements (for example, to make additional "overlapping" of responses).

 

The different types of analysis

Basic analysis Frequency tables which present the collection, synthesis and selection (flat or crossed) of the data.
Statistical calculations: percentage, average, variance or standard deviation, variation coefficient, etc.
Statistical test Study which determines whether a relation between items reflects reality or has appeared by chance
Association and causal analyses Study of simultaneous variations from several variables
Multivariate analysis Descriptive methodology which reduces, simplifies and summarises data (variables are analysed altogether).
Explanatory methodology which detects variables to be studied and explanatory variables

 

Numerous statistical methodologies can be developed, whose usage is more or less complex. In the evaluation field, simple treatments are often sufficient for the analysis stage, even though establishing correlations between items enriches the analysis.

Margin of errors

Errors almost always occur in the treatment of findings (the survey's conditions, sampling errors, constraints of time, costs, etc.) and a margin of errors should not be overlooked during the analysis of findings.
Thus, findings should be interpreted as indicators of a tendency rather than precise statistical measurements.

Developing the survey report

The survey report must include an analysis of basic findings. It may contain, for example:

  • A title page
  • An index
  • An executive summary
  • An introduction
  • An overview of the methodology
  • The analysis of the findings
  • The conclusions
  • Recommendations interpreting the conclusions in terms of strategic choice, and providing advice about the types of action to implement
  • The annexes

The evaluator must ensure as a minimum that the survey report has:

  • Clarity
  • Readability
  • Accuracy
  • Simple presentation with tables and diagrams
  • A summary of the information ranked in accordance with its importance

.

GLOSSARY

.

Sample/sampling

Samples are designed to gather people who are to be effectively surveyed. People are selected from the evaluation's target population, and this selection depends on the methodology chosen for the design of the samples (see How are samples designed?).

Working hypotheses

Apart from assuming an empirical approach for the evaluation (which is almost impossible), scientific researches usually: (1) develop hypotheses, and (2) check them. This principle supports the survey's design. The survey's findings enable the evaluator to check the hypotheses at the core of the survey's development, to reinforce them and/or to correct them.

Target group

The target group represents the part of the population on which the survey focuses. A typology or descriptive variables determine it.

Typology

The typology is a methodology dividing a population into types or homogeneous sub-groups with specific criteria (economical, social, demographic, etc.). Within each sub-group of the typology, respondents share similar characteristics (corresponding to the typology's criteria), or even the same values for certain predetermined variables.

 

EXAMPLES

 

Evaluation methodological approach > EN: Methodological bases and approach > Evaluation tools > Survey > Check lists

Check lists

CHECK-LIST FOR EVALUATORS

.

Questions Answer
Preparation and design  
Is the implementation of a structured questionnaire with a representative sample justified by the need of statistical indicators?  
Is the survey carried out with a representative sample?  
Were the questions asked and the answers collected understandable and straightforward?  
Were the length of the questionnaire appropriate?  
Does the questionnaire have check questions?  
Implementation  
Has the questionnaire been checked?  
Were the techniques used to conduct the questionnaire with the various categories of respondents (face-to-face questionnaire, by telephone, etc.) coherent?  
Has a monitoring and control process been organised for the interviewers?  
Have training or guidance sessions been set up?  
Were the interviewers independent from the policy / programme under evaluation?  
Was the number of respondents high enough to be representative?  
Is the degree of accuracy required for quantitative data related to the purpose of the evaluation?  
Were the findings proposed and explained to the various categories of stakeholders and beneficiaries?  
Were the findings combined with other tools of information and analyses used by the evaluators?  

.

CHECK-LIST FOR MANAGERS

.

Questions Answer
Is the implementation of a questionnaire with a representative sample justified by the need of statistical indicators?  
Is the survey carried out with a representative sample?  
Has the questionnaire been checked?  
Is the degree of accuracy required for quantitative data related to the purpose of the evaluation?  
Were the findings combined with other tools of information and analyses used by the evaluators?  

Evaluation methodological approach > EN: Methodological bases and approach > Evaluation tools > Expert panel

Expert panel

SUMMARY

 

Why is this tool used in evaluation?

An expert panel usually comprises independent specialists, recognised in at least one of the fields addressed by the programme under evaluation. The panel specialists arrive at conclusions and recommendations through consensus. Depending on the project proposal, these recommendations deal with the implementation or the impact of a programme, or part of it.

The expert panel is specifically appointed for the evaluation, and in conformity with standard procedures. The panel holds meetings and provides conclusions and recommendations in accordance with a precise and replicable working plan.

The use of an expert panel in country/region evaluations can be helpful in several situations, such as:

  • Studying very specific fields requiring a high level of competence (such as research support and assistance for high technologies)
  • Studying subjects for which other tools are difficult to implement at a reasonable cost
  • Carrying out limited-scope evaluations (such as assistance to small countries)
  • Assisting the evaluators in their conclusions on a subject in complex evaluations
  • Providing assistance in the drafting of final conclusions relating to the possible impacts of a programme in ex ante evaluations

How is a panel expert carried out?

What criteria should be used to appoint the panel?

The pre-requisite for the expert's selection is his/her professional experience. He/She should have specialised in the field under evaluation, and be recognised and respected by his/her peers.

Experts must be independent of the programme under evaluation, because they should not be judge and judged. Independence regarding the programme under evaluation is very important since the expert cannot be the judge and the judged.

Figure 1: Core criteria of the expert panel's composition

The ability to work in a group, listen to other experts and be open-minded is an essential criterion. Otherwise, working conditions may quickly turn out to be unmanageable, which would impede the panel process.

How are experts selected?

In a straightforward selection, the evaluation managers have access to a list of acknowledged experts in specific fields, and limit their selection process to ensuring the expert's independence regarding the programme under evaluation.

In gradual selections, preferred profiles of experts are developed with respect to the topics under scrutiny in the evaluation.

Elements to be taken into account in developing the panel profile are as follows:

  • Project scope
  • Degree of controversy (do the problems to be addressed have alternative resolutions which are controversial?)
  • Available data
  • Uncertainties (will the panel's conclusions discuss the uncertainties?)
  • Number of required disciplines

What are the procedures for the management of the panel?

There is no unique working process, and the expert panel should be encouraged to plan and implement its own workplan. Experts can focus their work on documentation and sessions, or broaden it to include meetings with project managers, field visits, implementation of surveys, etc.

The first panel session must result in the experts having a full understanding of their role in the evaluation.

During this session, the applicable methodology for the management of the panel's work must be discussed and validated. The discussion usually focuses on:

  • The panel's organisation and the role of each member
  • The type of investigation, the data collection methodology, and details of each panellist's task (such as field visits)
  • The intervention work programme, the organisation of future sessions and their contents

The next sessions (ranging from 3 to 5) will be directly linked to the panel's work. They will systematically deal with:

  • The work carried out since the previous session
  • Findings from investigations which are completed or in process
  • Problems encountered
  • Progress in editing the various documents, the review process, and quality control over these documents
  • The tasks to be achieved before the next session and its envisaged content
  • Confidentiality of the panel's debates and intermediate findings is another rule impacting on the panel's working arrangements.

What is the role of the panel chairman?

The panel chairman plays a crucial role. He/She guides the study panel, proposes the working arrangements, records findings, encourages contributions, facilitates debates and is the chief spokesperson for the panel. The quality of the working arrangements often depends on the chairman's leadership.

Figure 2: The various roles of the panel chairman

 

The chairman as Panel facilitator schedules the work of the panel and its production, and steers the panel's progress toward consensus.

The chairman as Report Architect and Integrator ensures a critical overview to the panellists' outputs, so as to improve the debate.

The chairman as Project Manager ensures that the available resources are sufficient and properly employed throughout the study. He/She ensures that the panel's sessions have been properly prepared by the technical writer.

The chairman as Spokesperson represents the panel in various bodies (such as monitoring committee and meetings with the commissioning agency and the press).

How does the expert panel report on its work?

The report, which supports the experts' contribution to the evaluation, is the only output from the panel which is made available to the commissioning agency. The report's structure should include: an executive summary, the mission's terms of reference, the composition of the panel, the evidence gathered and reviewed, the analyses carried out, The conclusion of the experts in the context of the report's consensus findings.

What are the preconditions for its use?

Figure 3: the preconditions for its use
The time span One of the advantages of an expert panel is in its speedy assembly process. For an evaluation, only 3 to 6 months work needs to be scheduled, and even less time for panel advice on a technical field within an evaluation.
Human resources Experts must have recognised expertise in the field under evaluation, be independent of the programme being assessed, be able to work in a group and be available for a continuous work throughout the evaluation.
Financial resources Budget line items normally taken into account while preparing estimates are as follows:
 

  • Salaries for the experts and the technical writer. If necessary, estimates may include salaries for subcontractors in charge of the panel's external studies
  • Communication and travel costs, publication, and dissemination costs related to the reports
  • Translation costs, if required

What are the advantages and limitations of the tool?

Figure 4 : The advantages and limitations of the tool
Advantages The experts' knowledge of the subjects under evaluation is the principal advantage of this tool. It fosters:
 

  • Significant reductions in time allocations
  • Cost effectiveness
  • Credibility of the conclusions
  • Adaptability to a variety of situations encountered in evaluation
Limitations The tool's limitations which should be minimised essentially derive from a series of risks:
 

  • Because the panel must come up with consensus-based conclusions, its organisation tends to eliminate minority points of view and tone down conclusions
  • The point of view of a 'dominant' expert can be over-influential within the panel
  • Experts have a tendency to go beyond their field of competence
Evaluation methodological approach > EN: Methodological bases and approach > Evaluation tools > Expert panel > Detailed presentation

Detailed presentation

.

This section is structured as follows:

.

WHY AND WHEN IS AN EXPERT PANEL ESTABLISHED?

.

What is the role of an expert panel in an evaluation?

What is an expert panel?

Definition

An expert panel usually comprises independent specialists, recognised in at least one of the fields addressed by the programme under evaluation. The panel specialists arrive at conclusions and recommendations through consensus. Depending on the project proposal, these recommendations deal with the implementation or the impact of a programme, or part of it. When consensus is not reached for particular questions, the panel must report on the various perspectives of the experts.

The expert panel is specifically appointed for the evaluation, and in conformity with standard procedures. The panel holds meetings and provides conclusions and recommendations in accordance with a precise and replicable working plan, which accounts for its reliability.

This tool is therefore designed to take advantage of the experts' knowledge in assessing policies, programmes and projects implemented in the field of their expertise.

Various types of expert panels in evaluation

International funding institutions such as the World Bank, the European Commission, Scandinavian countries, Canada and the United States regularly appoint expert panels to evaluate programmes.

In environmental evaluations, funding institutions such as the World Bank use expert panels to assess the quality of the service providers' work and suggest improvements. These panels examine the evolution of the project studies, from their preliminary stages to completion, including the first years of implementation.

The Delphi Method is another type of expert panel used for evaluation, based on an anonymous and repeated postal survey with experts (see Means Documents).

Where does the expert panel come from and how has it evolved?

The panel's origin

The concept of the expert panel originates in the research field. It derives from the peer review of scientific work processes or programmes, where expert panels have developed as an evaluation tool.

Its evolution

Two main developments over the past few years are noteworthy.

  • The panel's composition continues to diversify. Whereas the expert panel's missions were originally focused on limited areas of inquiry, their content and the panel's composition have been progressively diversified. Thus, in addition to programme specialists, economists, evaluation specialists and representatives of programme users are now appointed to panels, with a view to assessing all aspects of the evaluation.
  • The design of the panel's tasks has become more professional. The emergence of rigorous methodologies (and additional studies conducted by independent consultants verifying the panel's conclusions) has increased expert panel's credibility and changed expert panels into a reliable evaluation tool. These improvements have progressively given the expert panel the characteristics of a professional evaluative tool.

When is an expert panel appropriate for an evaluation?

What are the pre-conditions for its use?

Expert panels are appropriate for many evaluation situations, and particularly for:

  • Subjects which are well defined and require the advice of highly specialised experts. This is common in the research field, where the European Commission often uses this tool during evaluations.
  • Highly focused subjects (such as assistance to a small country evaluation) which do not require expensive allocations of resources.
  • Highly complex subjects (such as budget support), which are difficult to assess at a reasonable cost using other tools. In these cases, the point of view of experts specialised in the subject and/or in the country can constitute credible information and evidence for the evaluation.

Various tasks of an expert panel

 

With which tools can the expert panel be combined?

The expert panel can be combined with almost all the usual evaluation tools.

As well as combined with other tools, the expert panel's work is sometimes complemented by external studies. This usage has become widespread in fields such as research evaluations because it eases the task of the experts and provides them with information about the programme under evaluation. External studies include preliminary studies, surveys, database analyses.

.

What are the advantages and limitations of an expert panel?

.

The advantages

The experts' knowledge of the subjects under evaluation is the principal advantage of this tool. It fosters:

  • Significant reductions in time allocations, when compared with other evaluation tools
  • Cost effectiveness compared with other tools, due to reduced time allocation
  • Credibility of the conclusions, due to the experts' reputations
  • Adaptability to a variety of situations encountered in evaluation

The limitations

The tool's limitations which should be minimised essentially derive from a series of risks:

  • Because the panel must come up with consensus-based conclusions, its organisation tends to eliminate minority points of view and tone down conclusions.
  • The point of view of a 'dominant' expert can be over-influential within the panel, to the detriment of other perspectives and the report.
  • When the panel includes members who are the only specialists in the fields to be studied, an inappropriate empathic bias may occur. This could be exacerbated when the experts are acquainted with the actors of the field under evaluation. To mitigate this situation, appointments member of the panel must be reminded of the need for independence.
  • Experts have a tendency to go beyond their field of competence and the credibility of the conclusions can be adversely affected. Therefore, the panel's work should be strictly focused.

Even when these risks are controlled, social science detractors remain sceptical about the reliability of expert panel's conclusions.

.

What are the pre-conditions for an useful panel contribution to an evaluation?

.

The time span

One of the advantages of an expert panel is in its speedy assembly process. For an evaluation, only 3 to 6 months work needs to be scheduled, and even less time for panel advice on a technical field within an evaluation.

Human resources

The core aspect of an expert panel is the issue of human resources. Experts must have recognised expertise in the field under evaluation, be independent of the programme being assessed, be able to work in a group and be available for a continuous work throughout the evaluation.

Financial resources

The expert panel is known for its cost-effectiveness, but in the case of travel into remote countries and field visits, the expenditure allocated for experts' salaries and expenses will need to be increased.

Budget line items normally taken into account while preparing estimates are as follows:

  • Salaries for the experts and the technical writer. If necessary, estimates may include salaries for subcontractors responsible for the panel's external studies.
  • Communication and travel costs, publication, and dissemination costs related to the reports
  • Translation costs, if required

.

Why and how is an expert panel established in country/region evaluations?

.

Reasons for establishing an expert panel

The use of an expert panel in country/region evaluations can be helpful in several situations, such as:

  • Studying very specific fields requiring a high level of competence (such as assistance for the research field and high technologies)
  • Studying subjects for which other tools are difficult to implement (for example, budget support)
  • Carrying out limited-scope evaluations (such as assistance to small countries)
  • Assisting the evaluators in their conclusions on a subject in complex evaluations
  • Providing assistance in the drafting of final conclusions relating to the possible impacts of a programme in ex ante evaluations

Conditions for its use in country/region evaluations

No specific conditions are required for the use of expert panels in country/region evaluations. However, experts must be familiar with the context of the country assistance under study.

Examples of its contribution to country/region evaluations

No example of its contribution to country assistance has been found yet, but this tool is frequently used in programme evaluations in Western countries. A series of examples highlight the use of expert panels in the development of terms of reference documentation, recruitment profile processes, samples of reports.

.

HOW IS THE PANEL PROCESS MANAGED?

.

What criteria should be used to appoint the panel?

Core criteria of the expert panel's composition

 

 

Professional experience

The pre-requisite for the expert's selection is his/her professional experience. He/She should have specialised in the field under evaluation, and be recognised and respected by his/her peers. The credibility of his/her conclusions is highly dependent on these elements.

Independence

Experts must be independent of the programme under evaluation, because they should not be judge and judged. Thus, experts having a direct conflict of interest (such as experts participating in the programme or belonging to a body which benefits from it) should not be appointed.

However, securing sufficient independence is difficult in fields where there are few acknowledged experts. In this case, the panel, and not the experts, must be independent, and differing points of view must be dealt with in an even-handed way.

In addition, the experts' appointment to the panel is personal, which means that they do not represent their home institution and cannot be substituted on the panel.

Ability to work in a group

The ability to work in a group, listen to other experts and be open-minded is an essential criterion. Otherwise, working conditions may quickly turn out to be unmanageable, which would impede the panel process.

Other criteria

In addition to these core criteria and specific requirements relating to the expert's profile, some commissioning agencies may have their own views on the panel's composition (such as the expert's nationality, a balanced representation of differing points of view, the participation of specific categories of panellists such as beneficiaries, consumers).

Ideally, experts should also speak the language used in the area subjected to evaluation.

How are experts selected ? 

Selection process for the panel members

 

Procedures for the recruitment of experts

Varying with the importance of the task and the complexity of the themes, the recruitment of panel members can be relatively straightforward and speedy, or require a time-consuming selection process.

Straightforward selection

In a straightforward selection, the evaluation managers have access to a list of acknowledged experts in specific fields, and limit their selection process to ensuring the expert's independence regarding the programme under evaluation.

Gradual selection

Gradual selections have become a common procedure. Preferred profiles of experts are developed with respect to the topics under scrutiny in the evaluation. These profiles are critical to satisfactory recruitment to meet the evaluation's needs.

If the panel's intervention only focuses on technical issues, the expert's lack of independence within the fields under consideration will usually have minor impact on the evaluation, because the evaluator is responsible for the main tasks.

However, if the panel has to carry out a significant part of the evaluation, it should include an expert experienced in evaluation processes and a socio-economist, in addition to experts specialised in the topics under review. Depending on the nature of the mission assigned to the experts, additional fields of competence may need to be proposed.

Development of the panel profile

The Royal Society of Canada details elements to be taken into account in developing the panel profile, as follows:

  • Project scope: will the study be limited to technical problems, or will it address broad issues of public policy?
  • Degree of controversy: do the problems to be addressed have alternative resolutions which are controversial? Do they affect parties who have strong emotional, political or financial stakes in the outcomes?
  • Technical support: will the panel's conclusions and recommendations be based on data analysis or on the panel's expert judgement?
  • Uncertainties: will the panel's conclusions discuss the uncertainties?
  • Number of required disciplines: do the issues to be evaluated involve a single discipline or are they interdisciplinary?

The selection process

Once these profiles have been developed, the institution managing the evaluation should establish a "long list" of experts and remove experts with possible conflicts of interest. Thereafter, the institution contacts the selected experts.

Depending on the number of experts required (5 to 10 in most cases), the institution should pre-select a larger number of potential experts (2 to 3 times more than the final list), in order to ensure the availability of experts. Within this "long list", some funding institutions may determine nominees and alternates. This procedure is also recommended for the nomination of the chairman and key experts.

Once the selection process is over and the experts are recruited, the commissioning agency (or the experts) proceeds with the appointment of the chairman. As the role of the chairman is essential for the efficient working environment of the panel, the importance of this appointment is paramount. He/She will be responsible for setting the tone and rhythm of the panel's work.

Secretarial work should be the responsibility of a technical writer, whose ongoing availability should compensate the pressurised timetables of the experts. The tasks of the technical writer include the production of working reports, the incorporation of suggestions, and, if required, the monitoring of external studies.

What are the pre-conditions for the expert panel's work? 

A precise definition of the panel's field of work

Evaluators must take time to explain to the experts the context of their work and provide them with information about the programme, the procedures, and evaluation methodologies. In doing so, evaluators help the experts formulate their conclusions with a full knowledge of the environment of area assessed.

Usually, the content of the panel's working sessions is confidential.

Availability of documentation for the experts

Terms of reference

Terms of reference are often provided to ease the panel's working arrangements. They should specify:

  • The nature of the evaluation
  • The function of the panel within the evaluation and how this work is to be organised
  • The questions submitted to the experts or the nature of their investigations (such as points of view about the relevance, the coherence, and the efficiency of the programme)
  • The available data (for example, relating to the implementation of the programme) and the means at the disposal of the panel (including the possibility to go on-site)
  • The content and timetable for the reports

 

Reference interview guides

In evaluations where calls for expert panels are systematic, developing procedures for the management of the panel process, as well as interview guides may ease the panel's work (for example, during interviews with officials, field study, interviews with beneficiaries). These procedures can also benefit repetitive evaluations, through the standardisation of reports, which allow for comparisons.

Interview guides may be provided to evaluation teams unfamiliar with evaluation process, with a view to assisting them in their tasks.

What are the procedures for the management of the panel?

Fundamentals for the expert panel's work

There is no unique working process, and the expert panel should be encouraged to plan and implement its own workplan. Experts can focus their work on documentation and sessions, or broaden it to include meetings with project managers, field visits, implementation of surveys, etc. The choice of working process is highly dependent on the area studied, the experts mission, and the information and resources at the disposal of the experts.

Experts are expected to investigate and analyse the assigned topics and present their conclusions in a written report. The quality of the drafting and writing of the report is crucial and must be given careful attention.

Expert panels are usually expected:

  • To provide studies incorporating the scientific and technical standards of the relevant fields
  • To show complete independence from the programmes under review, which avoids any conflict of interest
  • To reach a consensus in their conclusions and recommendations

Some commissioning agencies believe that the search for consensus usually results in the experts producing an anodyne and unrealistic report. They require a report highlighting the experts' different points of view and the reasons for these differences.

Confidentiality of the panel's debates and intermediate findings is another rule impacting on the panel's working arrangements.

Guidelines for the panel sessions

The first panel session

The content of the first panel session derives from the terms of reference. This session must result in the experts having a full understanding of their role in the evaluation.

During this session, the applicable methodology for the management of the panel's work must be discussed and validated. The discussion usually focuses on:

  • The panel's organisation and the role of each member
  • The type of investigation, the data collection methodology, and details of each panellist's task (such as field visits)
  • The intervention work programme, the organisation of future sessions and their contents

During this session, panellists should be reminded of the general rules (such as independence and consensus) because although experts are generally familiar with the topics under study, they are often less well informed about evaluation principles.

The possibilities of conflict of interest with the programme to be studied should be closely examined, discussed and resolved during this session.

The next sessions

The following sessions (ranging from 3 to 5) will be directly linked to the panel's work. They will systematically deal with:

  1. The work carried out since the previous session
  2. Findings from investigations which are completed or in process
  3. Problems encountered (such as difficulty of collecting data, problems concerning the intervention timetable, the budget)
  4. Progress in editing the various documents, the review process, and quality control over these documents
  5. The tasks to be achieved before the next session and its envisaged content

With a view to ensuring the confidentiality of the panel's work, certain commissioning agencies recommend that records, summaries and intermediary reports of the sessions are destroyed. Only the final report is kept as the formal output required from the panel.

The organisation of the mission

The organisation of the mission depends on the panel's tasks. Most of the experts will be unfamiliar with evaluation techniques and may live far from the session's location. Thus, the production of the expert panel work programme should be scheduled well in advance, preferably as soon as the mission starts. This work programme should be adhered to whenever possible.

Experts responsible for tasks between two sessions, such as field visits, will be expected to work at least in pairs, in order to avoid bias of interpretation or empathy (the limitations). The formation of small groups should reflect the various points of view represented on the panel.

What is the role of the panel chairman?

The panel chairman plays a crucial role. He/She guides the study panel, proposes the working arrangements, records findings, encourages contributions, facilitates debates and is the chief spokesperson for the panel. The quality of the working arrangements often depends on the chairman's leadership.

The various roles of the panel chairman

 

The chairman as Panel Facilitator

  • At the first session, the chairman guides the panel to an agreement on a workplan and report architecture (as a working outline)
  • Thereafter, the chairman schedules the work of the panel, the production of the documentation and its revision
  • Throughout the study, the chairman ensures that each expert takes part in the working process and understands the evaluation's content.
  • Given the essentially diverse composition of a panel, and the often considerable differences in initial views, the chairman must steer the panel's progress toward consensus on the range of issues involved. Fairness and flexibility should be employed.

The chairman as Report Architect and Integrator

The chairman guides the study, defines methodologies, reviews outputs, ensures that timetables are respected, and records the findings of the panellists, which includes:

  • The provision of a critical overview to the panellists' outputs, designed to improve the debate rather than to control it
  • The drafting of the successive versions of the report with the assistance of the technical writer and the commissioning agency

The chairman as Project Manager

The chairman ensures that the available resources are sufficient and properly employed throughout the study. He/She is in permanent contact with the commissioning agency on financial and technical issues. If sub-contractors work for the panel, the chairman is responsible for the management of their studies, the supervision of their progress and their successful completion of their work.

He/She ensures that the panel's sessions have been properly prepared by the technical writer, and that all documentation and means required for their participation in the sessions are provided to the experts in a timely fashion.

If the mission requires a readjustment of its budget, its time allocation or its objective, the chairman presents this to the commissioning agency in order to reach an agreement.

The chairman as Spokesperson

The panel will need to be represented in various bodies (such as monitoring committees) and possibly in meetings with the commissioning agency and the press. As it is impracticable to gather all the experts for these meetings, the chairman serves as the spokesperson for the panel. He/She may delegate certain tasks to other panel members, but he/she should conduct the most important meetings.

How does the expert panel report on its work?

Synthesis of the panel's study

At the end of their mission, the experts report on their work. The report, which supports the experts' contribution to the evaluation, is the only output from the panel which is made available to the commissioning agency. Consequently, the report should be carefully prepared.

Guidelines for the final report

The report's structure depends on the nature of the mission. In technical or scientific missions, the report should at least include:

  • An executive summary, which is not too technical if possible
  • The mission's terms of reference
  • The composition of the panel describing relevant information about the background of the experts (in an appendix if required)
  • The evidence gathered and reviewed. The assumptions made and their impact on any conclusions should also be stated.
  • The analyses carried out
  • The conclusion of the experts in the context of the report's consensus findings. If consensus has not been reached, the report should state the consensus points, the elements of disagreement, and then explain them.

Reaching a consensus is the most challenging task for managers of expert panels, because a consensus strengthens the value of the panel's conclusions. In this context, the role of the chairman is crucial in seeking consensus, or formulating the final position of the panel, even if it include some dissenting views.

.

EXAMPLES

 

BIBLIOGRAPHY

General references

  • Expert panels: manual of procedural guidelines. The Royal Society of Canada. 1998.
  • "Strategic management of the research and scientific field", Chap. 1 Editor Economia. 1995.
  • Means Documents.

Specific references for the use of expert panels in country/region evaluations

  • Expert panel report, dealing with the Megascience forum evaluation, conducted by the OECD, 21 October, 1998.
  • Expert panels report on pesticides, New Zealand Radio, 11 September, 2000.
Evaluation methodological approach > EN: Methodological bases and approach > Evaluation tools > Expert panel > Check lists

Check lists

CHECK-LIST FOR EVALUATORS

.

Questions Answer
Preparation and design  
Was the use of an expert panel fully justified by the impossibility of obtaining judgements by any other way?  
Has the possibility of conflicts of interest with the programme under evaluation been addressed?  
If experts with conflicts of interest remained on the panel, were the reasons for this exposed (such as a lack of available experts for the topic, a balance of points of view within the panel)?  
Did competences of the panel members allow for the coverage of all the topics to be studied?Did the skills of the panel member enable the coverage of all the topics to be studied ?  
Did the global balance of the panel composition favour a balanced approach to the topic under evaluation?  
Implementation  
Is the time allocated for the study sufficient to cover all the issues?  
Were the experts properly informed about their assignments and the panel's working procedures?  
Were the experts provided with sufficient interview guides, procedures, etc., to enable them to produce work which is homogeneous with the work of other expert panels?  
Were the possibility of empathic bias been taken into consideration?  
Was each expert effectively involved in the panel's work, throughout his membership?  
Has confidentiality of the study panel been achieved throughout the contract (for example, with systematic information about the need for confidentiality, systematic destruction of intermediary documents)?  
Does the report elaborate on the analyses carried out?
Does the report shed lights on elements of consensus?
Are reasons for dissenting views explained?

.

CHECK-LIST FOR MANAGERS

.

Questions Answer
Was the use of an expert panel fully justified?  
Is the choice of the experts justified and relevant?  
Does the report elaborate on the analyses carried out?  
Are the elements of consensus clarified in the report?  
Are reasons for dissenting views explained and analysed?  
Evaluation methodological approach > EN: Methodological bases and approach > Evaluation tools > Expert panel > Detailed presentation of the tool

Detailed presentation of the tool

WHAT IS THE ROLE OF AN EXPERT PANEL IN AN EVALUATION?

What is an expert panel?

Definition
An expert panel usually comprises independent specialists, recognised in at least one of the fields addressed by the programme under evaluation. The panel specialists arrive at conclusions and recommendations through consensus. Depending on the project proposal, these recommendations deal with the implementation or the impact of a programme, or part of it. When consensus is not reached for particular questions, the panel must report on the various perspectives of the experts.

The expert panel is specifically appointed for the evaluation, and in conformity with standard procedures. The panel holds meetings and provides conclusions and recommendations in accordance with a precise and replicable working plan, which accounts for its reliability.

This tool is therefore designed to take advantage of the experts' knowledge in assessing policies, programmes and projects implemented in the field of their expertise.

Various types of expert panels in evaluation
International funding institutions such as the World Bank, the European Commission, Scandinavian countries, Canada and the United States regularly appoint expert panels to evaluate programmes.

In environmental evaluations, funding institutions such as the World Bank use expert panels to assess the quality of the service providers' work and suggest improvements. These panels examine the evolution of the project studies, from their preliminary stages to completion, including the first years of implementation.

The Delphi Method is another type of expert panel used for evaluation, based on an anonymous and repeated postal survey with experts (see Means Documents).

Where does the expert panel come from and how has it evolved?

The panel's origin
The concept of the expert panel originates in the research field. It derives from the peer review of scientific work processes or programmes, where expert panels have developed as an evaluation tool.

Its evolution
Two main developments over the past few years are noteworthy.

  • The panel's composition continues to diversify. Whereas the expert panel's missions were originally focused on limited areas of inquiry, their content and the panel's composition have been progressively diversified. Thus, in addition to programme specialists, economists, evaluation specialists and representatives of programme users are now appointed to panels, with a view to assessing all aspects of the evaluation.
  • The design of the panel's tasks has become more professional. The emergence of rigorous methodologies (and additional studies conducted by independent consultants verifying the panel's conclusions) has increased expert panel's credibility and changed expert panels into a reliable evaluation tool. These improvements have progressively given the expert panel the characteristics of a professional evaluative tool.

To find out more:

  • Examples: specific references on the contribution of expert panels to country/region evaluations
Evaluation methodological approach > EN: Methodological bases and approach > Evaluation tools > Case study

Case study

SUMMARY

.

Why is this tool used in evaluation?

The implementation of case study reviews of one or more actual examples, in order to gain an in-depth knowledge of the topic and, if possible, to learn about the entire evaluation programme.

In complex situations, case studies are the preferred evaluation tool when "how" and "why" questions are being posed, They allow a detailed examination of the actual elements in line with the evaluation goals. The purpose of the case study is to provide a picture, which is often more reliable than the outputs from other tools in context of the scarcity of basic data (which is often the case in country evaluations).

Figure 1- Case study's components

 

If case studies include the analysis of documents, statistical and implementing data, they are mostly known as a field observation tool and a means to interview people directly involved in the programme, such as the officials and stakeholders.

How is a case study carried out?

What are the conditions for the use of this tool?

To ensure that a case study is credible and yields satisfactory results the geographic evaluations specific context needs to:

  • find an effective local partner, who should be neutral to the topic under evaluation and competent both in the theme to be studied and in evaluation techniques
  • plan for supervision procedures of the international and local working teams
  • keep control of the selection of participants
  • have a fair distribution between interviews with officials and beneficiary representatives
Figure 2 : How is a case study carried out?

Steps involved in case study implementation

How is the instance selection undertaken?

This selection is crucial because an incorrect basis for selecting an instance can lead to a flawed evaluation outcome and can jeopardise its generalisation.

The United States General Accounting Office suggests 3 possible keys for instance selection:

  • Convenience
  • Purposive samples
  • Probability

Example of the country's selection criteria for the European assistance evaluation in the water sector:
Be included in the list of the country's main beneficiaries of the watsan European assistance
2. Have a representative in every region of the world
3. Have water sanitation as a priority intervention sector
4. Not being influenced by the last evaluations conducted by the European evaluation Unit

How is the case study organised and planned?

A modus operandi defining how to carry out one or more case studies is always useful.

Figure 3 : A modus operandi defining how to carry out one or more case studies is always useful.

Among other advantages the carrying out a pilot case study:

  • validates the methodology with a field test
  • determines more precisely the categories of respondents and the basic bibliography
  • completes the interview guidelines and the questionnaires
  • reviews the questions asked to the evaluators, criteria and indicators, in the light of what is available on-site
  • designs a standard report to be followed by the other evaluators

How are the data collected?
To ensure that the case study findings are reliable, a number of fundamental elements should be carefully taken into account:

  • The data collection should include adequate longitudinal data, i.e. data covering a sufficiently long period of time in order to avoid taking an anomalous situation as a reference point.
  • The data collection should be based on a principle systematically adopted in the evaluation: information verification through triangulation.
  • The evaluator must ensure that the information collected is thoroughly used, and that nothing important has been overlooked. It is essential to obtain as much information as possible, especially when opinions differ among the people interviewed.
  • By definition, the case study is open to any possible discoveries throughout the course of its implementation. Thus, the evaluator must know how to identify key features during the case study implementation and focus on them, even if they were not expected or scheduled in advance.
  • The field stage leads to first-hand observations of what is occurring. The evaluator must also note them down carefully.

How are the results analysed and interpreted?
This is the most challenging stage of the case study. Its goal is to analyse the data that have been collected during the fieldwork and to link as far as possible the effects of the observed facts to their causes. This analysis is difficult to conduct because it is less structured than at the conception and the collection stages.

The analysis overlaps with the data collection stage, and this is particularly true for case studies in which:

  • The data collection stage includes a pre-established hypothesis test that may partly modify the study content during its implementation.
  • The study is large enough to allow the evaluator to review and refine his criteria for the next data collection as a result of the initial findings.

What are the preconditions for its use?

Figure 4 : the preconditions for its use
The time span Preparation: 15 to 20 working days
Field mission: 3 to 10 working days
Data analysis: 3 to 10 working days
Evaluators' training (in case of multiple sites case studies): 2/3 working days
Analysis and conclusions from multiple sites case studies: 10 to 30 working days.
Human resources Qualified people who fully understand the problems associated with the evaluation, have sufficient experience of interview techniques and speak the language of the people interviewed.
Financial resources A minimum budget of €15,000 should be fixed and allocated to the multiple case studies preparation stage.

A budget of at least €5,000 to €7,000 should be planned for each case study, not including long-distance transportation.

What are the advantages and limitations of the tool?

Figure 5 : The advantages and limitations of the tool
Advantages Its richness, made possible by detailed qualitative information and the context of implementation precisely described.
Its relatively straightforward use.
Its flexibility, making continuous adaptation to various situations possible.
Its implementation stage is compatible with that of a country evaluation.
The opportunity to obtain and understand information at a sufficiently deep level. This tool allows evaluators to become familiar with the logic of action of the various actors.
Limitations The difficulty of identifying the appropriate targets.
The difficulty of identifying cases, setting boundaries and linking them to problems as broad as those commonly addressed in a country evaluation.
The difficulty arising from generalisation to a global level (for example, a country) of themes that were studied at a local level.
The tool's cost.
The fact that this tool rarely allows statistical interpretation of data.
The fact that this tool relies on the judgement of one or more evaluators can lead to partiality, even with the most careful use of case study methods.
Evaluation methodological approach > EN: Methodological bases and approach > Evaluation tools > Case study > Detailed presentation

Detailed presentation

 

This section is structured as follows:

 

WHAT IS A CASE STUDY?

.

What is meant by a "case study"?

The manager familiarises him/herself with the strategy, and particularly with the sectors and cross-cutting issues concerned. He/she identifies the key actors and composes the reference group in which the delegation is involved. The head of unit sends a note to the services concerned (example).

The case study is probably one of the most diversified evaluation tools, whose goals and content can vary greatly. So much so, that it is sometimes difficult for experts to be sure whether or not they are dealing with case studies.

To illustrate how case studies can benefit evaluations, two complementary definitions from the various published works can be cited:

According to the United States General Accounting Office (GAO 1970), one of the main evaluation institutions which has used and rationalised case studies in evaluation tasks, “a case study is a method for learning about a complex instance, based on a comprehensive understanding of that instance obtained by extensive description and analysis of that instance taken as a whole and in its context”.

According to R. Yin (YIN 2003), whose interest focuses on applied research in social sciences, “a case study is an empirical inquiry that investigates a contemporary phenomenon within its real-life context, especially when the boundaries between phenomenon and context are not clearly evident”.
The principal difference between research methodology and evaluation methodology is that in the former, a hypothesis is often chosen as a starting point that the case study will confirm or invalidate, whereas in the latter, the case study is a tool to study an example in detail, and generally with no predetermined views.

Thus, the implementation of case studies should start with a review of one or more actual examples, in order to gain an in-depth knowledge of the topic and, if possible, to obtain lessons for the entire evaluation programme.
In complex situations, case studies are the preferred evaluation tool when “how” and “why” questions are being posed, because they allow a detailed examination of the actual elements in line with the evaluation goals. The purpose of the case study is to provide a picture, which is often more reliable than the outputs from other tools in context of the scarcity of basic data (which is often the case in country/region evaluations).

The case study may include the examination of documents, of statistical data, or of programme implementation data, but, in particular, it includes the direct observation of an intervention and interviews with people directly concerned in its implementation and effects. This necessitates fieldwork by the evaluators, and this aspect is one of the strengths of the case study.

With its wide variety of use, the case study is a tool that can be used concurrently with almost all the stages in the evaluation process: data collection, observation, analysis and even in judgement.

Where does the tool come from?

This tool comes from the research field and has been incorporated subsequently into evaluation practice. Its use has expanded since the 1950s, and among research methods in social sciences (survey, experimentation, archives analysis…), the case study has established itself as a tool that can be used in specific situations where other tools have proved to be less effective.

Nonetheless, its general use has not developed easily, for the use of tools producing quantitative data (survey, mathematic tools, etc…) often compete with case studies. In this context, its advocates and users have established rigorous methods that allow them to obtain reliable results.

Fields in which this tool has been successfully used include social and political sciences, psychology, medicine, management, international relations, etc…

In the evaluation field, the use of the case study has covered a wide range of subjects. The first case studies undertaken were similar to research tools, and often carried out by a sole evaluator and essentially descriptive. Their main objective was to illustrate the implementation of a programme and its results through actual examples. They were often based on empirical methods.

Since then, the methodology has been progressively formalised and the approach more structured. However, the case study still remains a tool that allows the evaluator a degree of "freedom". The evaluator can place more emphasis on one aspect of the study, even though it was not designed that way (for example, the evaluator can decide to investigate the reasons why two groups of people have opposite views on a subject).

This progressive building of methodology and experience has led, inter alia, to the identification of various types of case study.

.

WHAT ARE THE VARIOUS TYPES OF CASE STUDY?

.

There are six main types of case study. It is possible to use them in combination, if the context allows it.

Actual implementation varies a little from type to type, however, it follows common conventions. These conventions are described in the section "How is a case study carried out?"

The evaluator can choose to implement more than one study within a single evaluation. These are called multiple case studies, whereas when case studies are conducted in several sites, they are called multiple sites case studies.

The illustrative (or descriptive) case study

The illustrative case study primarily describes one or a limited number of instances. It can be depicted as a tool which allows the evaluator to start from a general perspective and then highlight a specific element. Because it is a very descriptive tool, it can go deeper into an evaluation task and record field elements, increasing the credibility of the argumentation.

This field approach, based on the observation of actual events enables the reader of an evaluation report to fully understand the context of implementation, and any gap between what should have happened and what actually happened. Its principal role is to highlight this gap and explain why it has appeared.

Based on an analysis of available data and a series of interviews, its goal is to collect:

  • facts, and verification of facts,
  • opinions and points of view,
  • suggestions,
  • and/or reactions to the evaluator's hypotheses and conclusions.

From this information, the illustrative case study helps with the understanding of events, and facilitates and supports the analysis made in the evaluation report. As with other case studies, the illustrative case study focuses on the "How" and "Why" questions. However, the limited number of case studies conducted may not be sufficient to draw generalisations from what has been illustrated.

The exploring case study

This type of case study will probably be the most commonly used in country/region evaluation in the future. The exploratory case study mainly contributes to clarifying a situation in which information is scarce, and is undertaken before starting the field stage of an evaluation. This situation is often encountered in country/region evaluation.

The exploratory case study helps the evaluators assess the facts relating to programme implementation, the local context and, if possible, the findings, or the difficulties in identifying and measuring them. By giving this key information, the exploratory case study provides the basis for the establishment of the evaluation questions and the investigation methods that will have to be used. It should offer practical advice as to what questions should be asked, how and to whom.

This type of exploratory study can include one or several studies, depending on the variety of contexts to be taken into account. As exploratory case studies are not designed to provide conclusions, but only to prepare the ground for them, the level of investigation is less rigorous than in a descriptive case study, for example, and the chain of evidence is less well developed.

The reporting of such studies can take many forms but their execution needs less work than in other case study types, because their purpose is principally to ease the work that follows. Thus, they contribute more to the evaluation methodology than the accumulation of evidence.

This aspect is a potential pitfall for this type of study: it can lead to premature findings released as conclusions, in spite of a lack of depth in the study. Evaluators must proceed carefully and thoroughly evaluate their subsequent case studies once they are completed, in order to avoid confirming initial findings instead of testing them.

The critical instance case study

The critical instance case study examines a limited number of sites for one or two specific purposes (i.e. to investigate problematic projects). It analyses in detail cause-and-effect questions relating to the issues of concern, and confirms or invalidates hypotheses that have been formulated before the beginning of the case study. This aspect makes the critical instance case study methodology inappropriate for a country/region evaluation, because it is too focused on situations of extreme specificity: it is only mentioned here for completeness.

Some critical instance case studies may reveal broader issues. In this case, they can be used in country/region evaluation, but illustrative case studies should be used if the objective is to include in the evaluation a fair and balanced picture of the situation.

The programme implementation case study

Along with programme effects case studies, programme implementation case studies are probably be the most appropriate tool in country/region evaluation. Their goal is to examine whether programme implementation:

  • is consistent with the rules (utilisation of reference texts, which ones to select, how often have they changed; rules of monitoring-and-control, etc…),
  • meets expectations (how the beneficiaries are informed; beneficiaries' interest in the programme; difficulties in understanding the documents; delays in implementation, etc…),
  • leads to substantial or significant variations between sites (rules applied differently; differences in implementation between one site and another, etc…),
  • causes problems for the programme outcome, and why.

Programme implementation case studies are designed to answer evaluation questions relating to implementation (and particularly to effectiveness and efficiency). They should be designed to make generalisations from their findings possible. In country/region assistance evaluation, multiple sites studies are therefore almost a necessity. The instance selection is consequently very important because it will be the basis for future investigations.

The programme effects case study 

This is the most appropriate case study for country/region evaluation because it analyses the effects of programmes and strategies. It is designed to study the observed outcomes (positive or negative, expected or not) and to check whether or not they are the result of the implementation of programmes or strategies. The programme effects case study often includes an initial stage of implementation analysis, which corresponds to the coverage of programme implementation case studies, and is essential for a full understanding of the context. Thereafter, it focuses on the results of implementation and attempts to demonstrate how and why the programme's outcomes are positively linked to the observed changes.

As with programme implementation case studies, programme effects case studies also frequently aim at obtaining information that can easily be generalised. Therefore, multiple sites studies are almost inevitable. In order to give more weight to the studies' findings, it is useful to associate them with beneficiary surveys, which can give an interesting perspective on the evaluation, even though they do not always have statistical validity.

These studies concentrate on observed effects by relating differences between sites to explain links and causes between the observed changes and the evaluated programmes.

The cumulative case study

This type of case study brings together the findings from case studies or more general studies undertaken at different times. It aggregates information from one or more sites collected over extended period of time. Clearly, such studies are of a great interest and relevance but there are substantial difficulties in finding sufficient historical material to constitute the starting point for cumulative case studies.

The opportunity to implement these case studies during country/region evaluations is very limited, and the generalisation of the findings may not be a straightforward task. Thus, they are only mentioned here for the record.

.

WHY AND WHEN?

.

In which situation is this tool appropriate?

The case study is an outstanding tool for addressing complex situations, mostly:

  • when quantitative data are scarce or unavailable,
  • when assessing the answers to 'Why' and 'How' questions is as important, or more important, than the data analysis itself.

Case studies can be used in all types of evaluation (ex-ante, intermediary and ex-post). However, even if they can be designed for ex-ante evaluations, in practice implementation is difficult because ex-ante evaluations have to be conducted rapidly. Usually, only exploratory case study can be adapted to ex-ante evaluations.

Multiple case studies provide the evaluators with the opportunity to observe and analyse situations that are of special interest for the evaluation (for example, aspects that are the most problematic, success stories, etc…). Therefore, this tool can be applied using cases as close to the evaluation questions as required.

What are its advantages and its limitations?

The advantages of this tool

Paradoxically, some case study advantages are cited as drawbacks by opponents to this tool, for example:

  • its richness, made possible by detailed qualitative information. The context of implementation is precisely described and it is agreed that, on occasions, qualitative data are easier to understand than quantitative information.
  • its relatively straightforward use, especially for evaluators already used to techniques of interview, triangulation, observation and data analysis.
  • its flexibility, making continuous adaptation to various situations possible.
  • its implementation stage is compatible with that of a country/region evaluation (a few months), and can be very short in some cases (for example, in exploratory case studies).
  • the opportunity to obtain and understand information at a sufficiently deep level. This tool allows evaluators to become familiar with the logic of action of the various actors and the different perspectives they have about what is at stake, the problems, what is taken for granted, difficulties, etc, all of which can be linked to policies or programmes that are to be evaluated.
  • its accessibility. It is one of the few tools within the reach of non-specialists, enabling them to understand complex situations.

The limitations of this tool

Although this tool has many advantages, it also has limitations beyond which it should not be used. Apart from the individual limits inherent in each type of case study (Illustrative, Exploratory, Programme implementation case study, etc…), a number of common limitations exist, such as:

  • the difficulty in country/region evaluation of identifying the appropriate targets, leading to a risk of bias in the collection or treatment of information,
  • the difficulty of identifying cases, setting boundaries (for example, a geographical zone, within a population, etc…) and linking them to problems as broad as those commonly addressed in a country/region evaluation (for example, the struggle against poverty, economic development, etc…),
  • the difficulty of determining the number of cases that provide a sound basis for generalisation,
  • the difficulty arising from generalisation to a global level (for example, a country) of themes that were studied at a local level,
  • the difficulty of ensuring that data will always be available when implementing studies which extend over a significant period of time,
  • the tool's cost. In multiple sites country/region evaluation, this tool should be limited to evaluations where its added value is identified beforehand (for example, case studies in several countries designed to evaluate a regional strategy),
  • the fact that this tool rarely allows statistical interpretation of data, even if a valid survey can be done in each study,
  • the fact that this tool is based on the judgement of one or more evaluators. This can lead to partiality, even with the most careful use of case study methods.

Can the case study be combined with other tools?

The case study has very flexible implementation requirements and evaluators can combine it with other tools and methods. Among them are tools which are fully incorporated within case studies, such as:

  • interviews,
  • basic data analysis tools.

Other tools can only be applied to specific studies, such as:

  • the reconstitution of historical facts, mostly needed in cumulative case studies,
  • the use of specific tools dedicated to the processing of data from multiple sites studies (for example, matrix of categories, tabulating event frequencies, etc…).

Some tools can be added to the study, such as:

  • surveys, and particularly those targeting beneficiaries of the implemented programme,
  • focus groups targeting beneficiaries or people working in particular economic sectors.

The case study could overlap with existing investigations if the contexts were similar, but as a general rule, the case study has a specific purpose and would not be made redundant by other interviews or focus group activities carried out during the evaluation.

What are the pre-conditions for the use of the case study in evaluation ? 

The time span

The preparation stage of case studies can take a relatively long time, especially during the identification of sites, finalisation of the content and determining the logistics. Typically, an elapsed period of 2 to 3 months should be planned for a multiple sites study, and 15 to 30 working days should be assigned to its preparation.
The implementation of studies can vary greatly, but planning should at least include:

  • 3 to 4 working days allocated to the field mission, which could take 10 days or more if coupled with a survey.
  • 3 to 4 working days (up to 10) allocated for data analysis (excluding any survey analysis) and report preparation.

As one expert cannot implement a large number of case studies during an evaluation (due to a lack of time), several evaluators should work concurrently on-site. In this case, a period of training of 1 to 2 days should be planned.

For multiple sites case studies, the benefits of a pilot case study, implemented before the other case studies should be strongly considered. Its time span would normally be a little longer than average.

The analysis and conclusions from multiple case studies can take 10 to 30 days, depending on the complexity of the cases and on their number.

Human resources

Case studies must be carried out by qualified people, who:

  • fully understand the problems associated with the evaluation,
  • have sufficient experience of interview techniques,
  • speak the language of the people interviewed.

Financial resources

The full cost of case studies is very variable and depends, among other things, on:

  • the complexity of the subject to be studied,
  • the number of people to be met during the case study,
  • the distance between sites to be visited,
  • the location.

A minimum budget of €15,000 should be fixed and allocated to the case studies preparation stage.
A budget of at least €5,000 to €7,000 should be planned for each case study, not including long-distance transportation, assuming that no survey is to be undertaken.

When should a case study be used in project evaluations ?

Three types of case studies can be useful in project evaluations:

  • Exploratory case studies: these are used to help specify the questions at the start of the evaluation, or to design a tool (for example, a questionnaire). The findings of a simplified case study cannot be generalised automatically (the validation of any hypotheses would require further, specific case studies).
  • Illustrative case studies: these are undertaken for an in-depth study of all the categories of stakeholders, including final beneficiaries. Such a study aims at bringing out the factual elements of the project for its quality assessment without generalising. These case studies can be organised during the evaluation field.
  • Case studies about the effects and impact of the project on various categories of stakeholders.

.

IN COUNTRY / REGION EVALUATION

.

Why are case studies used in country/region evaluation?

The role of case studies in country/region evaluation

Case studies have a role to play in country/region evaluation at 4 levels:

  • Exploratory case studies allow the formulation of more precise questions at the beginning of the evaluation.
  • Illustrative case studies including an in-depth study reach the final beneficiaries and bring out the main features, without the obligation to establish a basis for generalisation.
  • Programme implementation case studies and / or programme effects case studies can deal with a series of programmes within a country evaluation.
  • Programme implementation case studies and / or programme effects case studies can also deal with a series of countries within a programme evaluation or a regional strategy evaluation.

The specific use conditions in country/region evaluation

To make the utilisation of the results of case studies as credible and useful as possible, and to derive generalised recommendations that work for all case studies' applications, the evaluators should:

  • find an effective local partner, without any conflicts of interest, and competent in the subjects to be studied and evaluation methodology,
  • establish procedures for the supervision of the work carried out by the local and international teams, so that the evaluators can manage issues relating to the physical distances involved,
  • monitor the selection of people to interview and avoid time pressure on interviewees if there is a long list of appointments,
  • ensure a balance between institutional interviews and beneficiaries' interviews.

In addition, in country/region evaluation, the main problem consists of determining what kind of detailed information to seek and who should be interviewed, in a relatively short time and on loosely defined questions. Case studies have similar problems.
Thus, one of the main tasks of country/region evaluation is to determine very precisely the types of actors that have benefited from the implemented policies. It is also necessary to analyse the strategies and logic of implementation developed by the various actors vis-a-vis the policies under evaluation.
The process of selecting and targeting the interlocutors is therefore very important, as is defining the questionnaire grid.

What kind of use for each type of case study in coutnry/region evaluation?

6 Types of Case Study and Their Possible Use in Country/Region Evaluation

Types of case study Adaptation to country assistance evaluation Place in the evaluation Benefits expected Limits
Illustrative case study May be used in country evaluation with the objective of giving realism to the study During the main stage of the field evaluation Gives a sense of reality to statements developed in the evaluation The limited number of sites makes generalisation difficult or even impossible
Exploratory case study May be used in country evaluation to better define evaluation context, problems and questions Before the evaluation or during the first stage of field evaluation Helps define the shape of the evaluation, the questions to be explored in depth and the organisation of the evaluation The findings from this study (less detailed than in a conventional case study) cannot be generalised
Critical instance case study Not usually used in country evaluation because it is too focused on a very specific instance
Implementation case study
---------
Programme effects case study
Recommended in country evaluation in their multiple sites version During the main stage of field evaluation and also during the initial stage with a pilot study in order to help strengthen the methodology Give a strong basis and legitimacy to the evaluation assertions on implementation conditions and effects Need to be implemented at several sites to allow the generalisation of findings
Cumulative case study Very difficult to use in country evaluation because of a lack of previous cases

.

HOW IS A CASE STUDY CARRIED OUT?

.

What are the stages to follow when carrying out a case study?

STAGE 1: THE CASE STUDY PREPARATION

How is the methodology designed?

The design of the methodology includes the basis for analysis and conclusions, the importance of the study in the overall organisation, the general design and the sources of information.

During the design of the methodology, 5 successive steps should be taken before deciding how many case studies and which type to implement.

First step
Second step

Determine the case study role within the evaluation process (designing the methodology or giving data on the implementation process, etc.).

Third step

Determine the guidelines for the study's general approach. Stipulate the primary hypotheses (where they exist), the selected analysis units (for example, regions), the kind of information to be collected, etc. These guidelines must determine a precise process, the more so when multiple sites studies are planned. The final stage of this general approach and the case study process will be achieved in the pre-operational stage, once adequate data has been collected.

Fourth step

Propose the sources of information that have been identified before the implementation of the case study. These will be used to collect each type of information.

The documentation:

Documentation can be very varied and may include, for example:

  • Data concerning the country, the region, or the programmes evaluated and their implementation within the scope of the case study,
  • Previous study reports,
  • Activity reports from operators and programme monitoring,
  • Monitoring data from projects and programmes,
  • Maps,
  • Statistics,
  • Newspaper articles,
  • Contracts, etc.

This documentation can be collected on-site but may also be obtainable before the departure of the evaluators from the administration or the operators working in Europe and/or in the country to be evaluated. Some documentation can also be acquired from the Internet (for example, other tender evaluations).

The interviews:

This is the most important source of information in case studies and should be systematically planned. In country/region evaluation, however, one of the most difficult tasks is to decide on whom to focus. If some targets are obvious, others are less so. Thus, the list of the target group will need to be reviewed on-site and after preliminary interviews. This should also be done to avoid being restricted to interviewing officials nominated by the local authorities. Focus groups can also be conducted, especially with the beneficiaries.

Direct observation:

Direct observation is an absolute necessity in case studies because the evaluator has to have first-hand knowledge of the events he is studying. Most of the time, the observation includes:

  • A simple review of the situation on-site (for instance, firms that are being helped, work in progress, classrooms, community clinics, etc.) and of the people (for example, the standard of living, the health standard, schooling, etc, and also the relations between stakeholders, with the authorities, etc.).
  • If needed, physical measurements.
  • A pre-determined minimum number of interviews, and particularly with beneficiaries.
The other sources of information:

Many other sources of information are at the disposal of the evaluator in case studies. Among them, the most common is the carrying out of concurrent surveys (in particular, beneficiary surveys) in order to add quantitative data to the qualitative data. Thus, a larger number of surveys within case studies gives the evaluators a better quantitative understanding of the observed events.

Instance selection

This selection is a crucial stage because a wrong basis for selecting an instance can lead to a flawed evaluation outcome and can jeopardise its generalisation.
If multiple sites case studies are scheduled, methods that have been chosen for the instance selection should be detailed. The United States General Accounting Office suggests 3 possible keys for instance selection:

  • Convenience
  • Purposive samples
  • Probability

The instance selection varies with the combination of the 3 keys and also with the number of case studies.

Convenience

Using this key, the selection is justified because the data collection is straightforward (for example, a site near the capital where most of the fieldwork is implemented; a site of a limited size that will only need a short mission, etc…).
Although this key is not likely to be the best fit for a fully representative study, it can give an acceptable introduction to some illustrative case studies with a minimum of resources.

Purposive samples

This second key is the most used, and is sub-divided into several options, depending on the evaluation objectives:

  • Bracketing: the evaluation seeks to explain why important differences occur within several situations that could have been identical. Case studies will focus on extreme outcomes, selected for their particular meaning to the evaluator who will attempt to demonstrate a point (gap between implementation costs, between effective outcomes, etc…).
  • Best cases/worst cases: the case studies focus on projects or programmes of interest to the evaluation because of their successes/failures.
  • Cluster: while most projects and programmes are very different, some belong to homogeneous sub-categories (for example, different projects focusing on one country in a regional programme; programmes implemented by two different operators in the same country, etc…). In this case, the instance selection includes these cases that can be clustered on points of interest for the study, forming sub-categories (for example, are programmes implemented by one operator better conducted than those implemented by another?).
  • Representative or typical: the instance selection focuses on one or more points of interest to be dealt with by the study (for example, important programmes, health programmes, etc…).
  • Special interest: for very specific cases in which the information is extraneous to the data organisation carried out by the evaluator. For example, the evaluation terms of reference specify that the programme should be considered as a case study because it presents serious dysfunctions.
Probability

Under this third key, instances are selected from a list with the help of traditional criteria for purposive sampling, with an equal chance of being included. This selection method is not widely used, mostly because the number of cases is insufficient to lead to a representative sample. It can also be argued that case studies focus by definition on very specific problems.

Combination of the 3 keys

Some combinations of different instance selections are possible. For example, representative elements can be coupled with a bracketing purpose, or specific cases.

Number of case studies

There is no minimum number of case studies; however, the more included, the more detailed the future analysis will be, along with the increased likelihood of making useful generalisations. For most of the time, case studies will not normally permit statistical analyses, therefore unduly multiplying the use of case studies should be avoided. Five to 10 studies for an evaluation should be enough to make generalisations. This figure can rise to 15 or so if a series of distinctive problems and situations need to be studied in various case studies. This is however seldom encountered.
It is essential to confirm with the authorities in charge of programme management that the selected cases have not already been the subject of other case studies implemented during past evaluations, and this should be done before fixing the list of case studies to be carried out. Otherwise, the target group will probably show signs of irritation that could affect the quality of the study.

STAGE 2: THE PRE-OPERATIONAL STAGE

Once the decision to carry out a case study has been taken and guidelines for its contents defined, the pre-operational stage can begin. This stage is designed to facilitate the operational stage. Several activities have to be conducted before starting the case studies:

Finalise the process

A modus operandi defining how to carry out one or more case studies is always useful. It is part of the design of the evaluation methodology but goes beyond it by additionally including operational aspects. While the modus operandi is brief for a case study conducted by only one evaluator, it has to be very detailed where multi-site case studies are concurrently carried out by different evaluators. The modus operandi generally includes:

  • Data giving the evaluators a good insight into the context: the evaluation and the case studies' objectives, the logical frameworks for the programmes or the projects to be evaluated, the main implementation data, the methodology and the reasons why case studies have been selected, and the evaluation and the case study timetable (detailed phase by phase).
  • Data collection methods: definition of the scope of the study, the period, the people to meet, the data to be collected (analysed by type: documentation, interviews, focus groups, observation, surveys, etc…) before or during the case study fieldwork, and the time to spend on each phase. The methodology should always be based on the chain of evidence techniques that are being used, i.e. based on explicit links between evaluation questions given to the evaluator, those formulated by him, the data he collects, and the evidence and analysis produced to arrive at his conclusions.
  • Actual elements of methodology on which the evaluator can rely: guides for interview by type of actor, questionnaires, and field measurements if they are useful, etc.
  • Questions which the case study has to answer and possible interpretations, methodologies used to provide the answer, evaluation criteria, and indicators that are sought. In order to ensure that the multi-site case studies contribute to the general synthesis, it is essential that each study responds to an identical series of questions.
  • The framework for the case study report, including the presentation of the findings (the template sheet, for example) and the content of the annexes. At this stage, the development of a pilot case study before the other case studies is a great advantage: the pilot will be an example of the size and the content that other evaluators will have to manage in their studies. Also, the synthesis of all the studies' findings will be easier to make.
  • The names and the contact details of the main source, who can be asked to supply explanations to the evaluator at any time.
Choosing the teams to implement the case studies

Concurrent with the finalisation of the organisation stage (if it has not already been done in the consultant's answer to the call for tenders), a list of the evaluators in charge of the case studies and their supporting staff (for example, the investigators) should be agreed with the client.
At this stage, two essential requirements should be fulfilled before proceeding with the implementation of these studies:

  • The selected people must have the required competence for the various tasks that have been defined. Case studies should always be directed by one of the evaluators belonging to the team (national or international).
  • The selected people should speak the language of the country, region or area. Interpretation is always possible but should be limited to situations in which all the other solutions have failed.

Once this list has been established, the consultants should ask their client for a formal notification of mission, describing their intervention objectives and asking the people they are going to meet to assist them as much as they can.
In certain cases, junior assistants may carry out part or the whole of a case study, under the evaluator's supervision and responsibility. It is however more sensible to limit the task to straightforward collection of information (for example, statistics, bibliography, etc.). Interviews with beneficiaries and the authorities in charge of implementation should be conducted by the evaluator, whenever possible.

Scheduling the case studies

Scheduling a case study is a very detailed operation that needs to be carried out well in advance in order to get the required appointments on-site. In country/region assistance, the use of a local representative is almost obligatory in order to:

  • find the co-ordinates of the people to meet,
  • give them a meeting request form, a copy of the interview guide, the client's notification of mission and, if necessary, basic project or programme documentation,
  • organise the interviews and the meetings (reservation of vehicles, local meeting rooms and offices, accommodation, etc…),

give assistance with translations or with the conduct of some parts of the study (for example, surveys) where required

If possible conduct a pilot study

There are many reasons to justify the conducting of a pilot case study before starting a multiple sites case study. Indeed, this good practice allows, inter alia, the:

  • validation of the methodology within which the studies are to be conducted, by testing it on-site,
  • informed reduction of the list of the people to meet and of the scope of the core bibliography,
  • finalisation of the interview guides and the questionnaires,
  • review of the questions given to the evaluators, the criteria, and the indicators in respect of what is available on-site,
  • writing of a standard report that will be followed by other evaluators. It is often more understandable than a study production guide.

The pilot site may be chosen because of its practicability (accessibility, sufficiency of data or interlocutors' availability). However, as the evaluator who will supervise all the case studies often carries out the pilot case study, it will usually be selected on the basis of its potential contribution to subsequent case studies. These criteria could be:

  • a study that identifies a large range of problems and contexts that other studies will have to deal with (those elements will therefore be used as a reference point for other case studies),
  • the most complex or difficult study to conduct, regarding logistics. This will ensure that the other case studies are viable.
Train the evaluation team when multiple sites case studies are scheduled

Arranging for one evaluator to implement all the case studies can be very difficult or impracticable, in particular because of delays in completing the studies. As a consequence, case studies are usually conducted by several evaluators, which inevitably risks an "evaluator effect" in the conduct of the case study and in the interpretation of the findings. This means that the evaluator's subjectivity should be controlled as much as possible.
To do so, in addition to the standardisation of methodology and practice described in the previous stage, the evaluator in charge of the case studies should train all field evaluators in order to get their understanding and agreement on the objectives, the methods and what the reports are expected to achieve.
The content of this training consists of a group review of elements of methodology and the reporting of standards, with a view to getting a consensus on the content of the work.

This training should be linked with permanent monitoring, with each evaluator being able to contact the person in charge of the case studies quickly to get information and advice concerning their conduct or reporting. If the studies have to be written in several languages (for example, where regional evaluations include case studies from many countries), this information should also cover the meaning of particular concepts which could be ambiguous or unclear in translation.

STAGE 3: HOW ARE THE DATA COLLECTED?

This stage, which can start before the fieldwork with the bibliographical data collection or preliminary meetings, is crucial, especially when the evaluator goes on-site. To ensure that the case study findings are reliable, a number of fundamental elements should be carefully taken into account:

  • The data collection should include adequate longitudinal data, i.e. data covering a sufficiently long period of time in order to avoid taking an anomalous situation as a reference point.
  • The data collection should be based on a principle systematically adopted in the evaluation: information verification through triangulation. This means that the data are collected from different sources and combined (for example, interviews, data taken from the implementation and surveys, or many interview sources, such as both programme administrators and programme beneficiaries).
  • The evaluator must exercise care in collecting and presenting the needed to support his argumentation. This evidence (similar to the judiciary field) can be:
    • facts that he himself has observed,
    • facts observed by people or institutions above suspicion (for example, public statistics),
    • written documentation whose value in terms of evidence can be relatively variable (for example, meeting reports),
    • the acknowledgement of mistakes made by certain institutions or individuals (for example, during the programme implementation),
    • evidence against or for someone. However, in the context of an evaluation, the veracity of such testimonies should be checked out through triangulation,
    • experts' opinions,
    • other material evidence, such as video tapes, audio cassettes, etc…
  • The evaluator must make sure that all the relevant sources are properly used: statistical data relating to the context, implementation data, existing reports, interviews with the appropriate persons taking account of mixed categories of actors (administrators, operators, research and NGO and control staff, beneficiaries…), and surveys.
  • The evaluator must ensure that information collected is thoroughly used, and that nothing important has been overlooked. It is essential to obtain as much information as possible, especially when opinions differ among the interviewed people. If this is the case, the evaluator should request their argumentation, examples and evidence. This type of inquiry gives validity to data collected which will be used at the analysis stage, and supports the chain of evidence.
  • By definition, the case study is open to any possible discoveries throughout the course of its implementation. Thus, the evaluator must know how to identify key features during the case study implementation and focus on them, even if they were not expected or scheduled in advance. More particularly, the data collection stage should be designed to progress the analysis process in order to test alternative interpretations put forward to explain a situation.
  • The reporting of the study findings is a fundamental element that increases the credibility of what has been found. Thus, the evaluator must carefully make notes on all the information he has collected, and especially everything dealing with the context, the relationship between stakeholders, and the reasons for the study's success or failure. To do so, it is important during the interviews to clearly dissociate the information collected (facts that can be checked) from points of view, analyses and opinions, in order to present them properly in the report.
  • The field stage leads to first-hand observations of what is occurring. The evaluator must also note them down carefully.

The language in which the case study is to be implemented is also an important point. Thus, the evaluator can be locally recruited or assisted by an interpreter for the whole study or only a part of it (for example, during exchanges with the beneficiaries).

STAGE 4: HOW ARE THE FINDINGS ANALYSED AND INTERPRETED?

This is the most challenging stage of the case study. Its goal is to analyse the data that have been collected during the fieldwork and to link as far as possible the effects of the observed facts to their causes. This analysis is difficult to conduct because it is less structured than at the conception and the collection stages.

The two important analysis patterns

Two main analysis patterns are linked to the commencement of the case study itself:

  • If the case study is implemented to check out a hypothesis: the analysis is focused on elements that confirm or invalidate this hypothesis. This type of case study is more often used in the research field than in evaluation.
  • If the case study starts with no preconceived ideas, then its implementation will progressively build up the evidence to explain the findings, predictable or otherwise. This constitutes the chain of evidence.
Process and link to other stages

The analysis overlaps the data collection stage, and this is particularly true for case studies in which:

  • The data collection stage includes a pre-established hypotheses test that may partly modify the study content during its implementation.
  • The study is large enough to allow the evaluator to review and refine his criteria for the next data collection as a result of the initial findings.

If the findings are to be investigated by other tools after the fieldwork stage, with a view to assembling and comparing data (for example, survey analysis, benchmarking, etc.), the analysis must always be based on the construction of a chain of evidence.
The analysis stage must include a number of steps, some being concurrent with the data collection stage:

  • The preparation of a database for the case study,
  • The process must always be iterative, i.e. based on a search for information sufficient for the construction of a chain of evidence,
  • The analysis must test all possible explanations for a situation in order to retain a limited number of probable explanations, and sufficient justification to eliminate others.
  • The evaluator must make sure that the analysis findings are reproducible, i.e. make sure another evaluator would come to the same conclusions when confronted with the same information.
  • For multiple sites evaluation, a generalisation is one of the main objective to be achieved. This can only be reached if the previous stages are properly implemented, and give the findings a consistency and plausibility to enable the evaluator to draw firm conclusions.
The particular case of the multiple sites case studies

The analysis methods for multiple sites case studies use different types of tools which make the management of various qualitative data easier. Among these techniques are:

  • Matrix of categories that places similar elements from different sites (for example, types of programme administration or number of educational programmes implemented in each site, etc.) into a table. In this technique, a coding scheme is established prior to data collection.
  • Data tables that emphasise the frequencies of occurrence of an event. These tables are particularly suitable for case studies and derived from surveys conducted during the implementation. They make comparisons between the findings of the surveys from the different areas considered in the study. However, as a case study rarely includes the construction of a statistically valid comparison, other means of data comparison should be found without resorting to classical statistical representations, such as the average, the standard deviation or the variance.
  • Graphic data displays with or without computer programmes. When they are being formalised, these techniques require a meeting between the evaluators conducting the case studies to validate or exclude hypotheses on each topic being studied, as illustrated by the graphics. This will enable the evaluators to progress towards their conclusions. The use of video-conferences and emailing between a limited number of evaluators could be an alternative and cost-effective solution to such meetings.
  • Time-series analyses that compares, site by site, the chronological events in order to draw conclusions. These series may be a useful technique to explain why timing leads to the success of some programmes, and failure in others.

This list of tools is not exhaustive. Each evaluator must be free to show the findings from several of his case studies by using any tools providing a chain of evidence, the ultimate goal. With the benefit of studies carried out at various sites, the chains of evidence are reinforced when similar effects happen and are identified in different sites at the same time.

STAGE 5: THE REPORTING

This stage is crucial because the reader of the report must get the same insight from the evaluation as the evaluator. Thus, the importance of this section of the case study should not be underestimated. For the presentation, the report must include:

  • the objectives of the evaluation and the case studies,
  • the selected sites, and the reasons for selection, (for a multi-site case study),
  • the collecting and analysis methods which have been used and their limitations,
  • basic data for each case study (statistics, implementation data, survey findings, etc…) and the sources of information,
  • factual descriptions and, particularly, those relating to implementation that will ease the reader's insight,
  • the answers to questions dealing with:
    • comparisons between different sites and the case studies' content, summarised in a readable way,
    • the chains of evidence leading to well-argued findings.
  • the building of conclusions designed to lead as far as possible to useful generalisations (mostly for the programme implementation case studies and programme effects case studies).

Ideally, the in-mission report should be distinguished from the final report in order to avoid any bias from empathy. Thus, the author of the final report should not normally be responsible for the production of the in-mission report. However, this can be difficult to achieve in country/region evaluation, where time and budget constraints are significant.

The editing style of case study reports largely depends on the role that the evaluators want them to play. This role should be defined in advance, for example, in a context of:

  • a single illustrative case study. This should give a detailed description of the observed event, in order to bring factual elements to the evaluation.
  • a multiple sites case study with numerous sites. The summary analysis report will not be as detailed as in the single illustrative case study because it has to be readable. A sites' comparison should be made, and each study report should be included in the annex.

The target group (to whom the case study findings are directed) must also be identified in advance. In country/region evaluation, the target group will mostly be the evaluation users. Report production will therefore be essentially focused on the answers to the evaluation questions, and on the conclusions and recommendations directly linked to programme management.

If the case study is addressed to a general audience, reporting should reflect this by taking a less technical approach to the content. Giving a non-technical summary at the beginning of the report can also be helpful to non-specialists.

What are the pitfalls to avoid?

Point 1: inapropriate design of the study

Within the general design of the study, the evaluator must always be able to answer to the question "How?" and, wholly or partially, "Why?". Its validity must be tested once the design stage begins and must include:

  • the questions at stake,
  • the spatial and temporal analysis units that have been selected,
  • the collection and data analysis process, linked to the questions at stake,
  • findings' interpretation criteria and their reporting type.

If one of these criteria is poorly conceived, doubts will be raised about the validity of the whole case study. Thus, checks must be made as to whether:

  • The questions are directly linked to the evaluation questions, and will lead to all or part of the answers sought.
  • The units selected cover the problems and the context (for example, is the selected period long enough? Does the area studied sufficiently cover the problems and contexts at stake? etc…).
  • The methodology deals with all the study sections, from the data collection to the report. Does it lead to the establishment of a chain of evidence? Is the process iterative and is it implemented with a view to validating/excluding alternative interpretations?
  • The findings' interpretation criteria are explicit and sufficiently argued?

It is important to bear in mind that in such a difficult field as in country/region evaluation:

  • It is a sensible precaution to plan several studies during the case study design because a single study may lead to bias.
  • Multiple sites case studies should be designed with the same methodology to make comparisons between sites possible and provide a sound basis for generalisation, unless good reasons otherwise are produced.
  • Despite the required rigour in the implementation of case studies, the study design must always include the flexibility to "explore" a new field if the evaluator thinks it justified. When this occurs, the reasoning should be validated with the manager of all the case studies.

Point 2: poorly selected sites

The sheet "preparation for case studies" gives details of the various choices for site selection assisting evaluators to implement case studies.
Therefore, selections must be carefully made and be the result of:

  • detailed argumentation in an intermediary report illustrating the choices and proposed selections, directly linked to the evaluation questions (for example, when to choose a bracketing (extreme outcomes) case study or a worst case study. Why choose 5 sites rather than 3 or 7?),
  • a process of discussion/validation with the client, including local consultation if possible (with a delegation, host country counterparts, etc…), in order to validate that the proposed sites fit the purpose of the study and do not present any specific problems (for example, sites already subjected to evaluation, dangerous sites, etc…),
  • the drafting of the final document, integrating the different stages and stipulating the final selections.

Point 3: insufficient information collection and weak argumentation

Case studies are mostly used because they offer the opportunity for a detailed examination of a situation. That is their main justification. To be decisive, the information collection must be focused. As a consequence, there must be a systematic search for exhaustiveness and quality when planning and carrying out a case study. The collection and the analysis stages detail the points that have to be met in order to reach this objective.

If the evaluation design has been well conducted, there is less chance of insufficient information collection and weak argumentation; however, some precautions may improve the case studies content, such as:

  • producing criteria to secure a good data validity (on the selection of information sources, complementarities between them, etc…),
  • requiring the lead evaluator devise and enforce quality assurance procedures in order to check the accuracy and plausibility of the multiple sites case studies' findings produced by other evaluators.

Starting the process of production of the report, by drafting appropriate sections during the collection stage, is an excellent way to identify any lack of information. This is because the evaluator's judgement is of the utmost importance.

In any case, an assertion without any evidence is a pitfall to be avoided at all costs in a case study evaluation. The analysis and production stages must always reflect impartiality, which is the only guarantee of the findings' credibility. This is because the only element eventually taken into account is the evaluator's judgement of the evidence. This judgement can be tested, particularly by submitting the case study report to the people from whom data were collected to obtain their views. This leads to:

  • checking the accuracy of the data used in the report (for example, eliminating factual errors).
  • correcting any subjective assertions.
  • limiting any analytical bias.

Point 4: Excessive generalisation

Generalisation from findings is one of the most valuable achievements of an evaluation. But, it can also be the case studies' weakest point if they are badly conducted, especially when case studies are contrasted with tools based on statistical analysis. Thus, being able to generalise from case studies implies bringing enough evidence to be convincing. Excessive generalisation must be avoided. This is also often linked to the following points:

  • a bad design (that can make the findings unusable),
  • an incorrect site selection (one that does not demonstrate what the evaluator is looking for),
  • poor argumentation (insufficient evidence to support the conclusions, let alone to generalise).

Usually, generalisation is possible when case studies have an external validity, for example, as demonstrate by a survey (if the interviewees are representative). This external validity is easier to deduce from several case studies rather than from one. Questions on the external validity may also concern only specific points of the study, and not the whole case study (for example, conclusions about implementation problems can be generalised but not those about the programme impacts).

The pitfall to avoid in terms of generalisation is to "imitate" surveys when implementing case studies. In a case study, generalisation is derived from analysis, not statistics, as with experimentation. Generalisation can only be used if the case studies' selection, and the way they are carried out, supports it. In particular, multiple sites case studies can produce useful, comparative analyses, from which common conclusions are derived. In order to know if a generalisation is valid, it is possible to use the replication technique (as in the scientific field), which tests whether replication is possible on a second site, or a second series of sites, and if a general theory is emerging which can be universally applied.

EXAMPLES

 BIBLIOGRAPHY

  • Case study Evaluation, November 1990. GAO/PEMD-91-10.1.9.
  • Case study research. Design and methods. Third Edition. Robert K Yin. Vol 5. 2003. www.sagepublications.com
  • Embedded case study Methods. Integrating quantitative and qualitative knowledge. Roland W. Scholz - Olaf Tiedje. 2001. www.sagepublications.com
  • Cahiers MEANS Volume 3 "Principales techniques et outils d'évaluation". Commission Européenne. 1999. Office des publications officielles des Communautés Européennes.
Evaluation methodological approach > EN: Methodological bases and approach > Evaluation tools > Case study > Check-list

Check-list

CHECK-LIST FOR EVALUATORS

.

Questions
Answer
Preparation and design
 
Is the use of the case study tool in the evaluation backed up by adequate argumentation?  
Is the choice of the case study application well-argued?  
In the context of multiple sites case study, is the number of case studies justified?  
Has the design methodology been properly elaborated?  
In the context of multiple sites case studies, does the methodology assure consistent reports?  
Has a pilot case study been scheduled?  
Is the use of triangulation clarified in the methodology and included in the mission reports?  
Have the sources of information (documentation, interview, monitoring data, direct observation) been included in the mission reports?  
Do the methodology and reports distinguish facts from opinions?  
Is the plan for the development of a chain of evidence well-argued in the mission report?  
Implementation
 
Does the iterative process, initiated at the collection stage, carry on to the analysis stage, and support the chain of evidence?  
Were alternative explanations studied and rejected after a full review of the evidence?  
Are the facts supporting the argumentation strong enough to guarantee systematic replication elsewhere?  
Does the analysis include research into causality?  
Are the techniques used for the analysis of multiple sites data set out and argued?  
Is the case study report sufficiently understandable and explicit?  
In the case of multiple case study has the team leader checked the relevance /consistency of the studies ?  
Are the limitations of the impact of the study findings sufficiently well explained?  

.

CHECK-LIST FOR MANAGERS

.

Questions
Answer
Is the use of the case study tool in the evaluation backed up by adequate argumentation?  
Do the methodology and reports distinguish facts from opinions?  
Are the techniques used for the analysis of multiple sites data exposed and argued?  
Is the case study report sufficiently understandable and explicit?  
Are limitations of the impact of the study findings sufficiently well explained?  
Evaluation methodological approach > EN: Methodological bases and approach > Evaluation tools > Context indicators

Context indicators

SUMMARY

.

Definition

A context indicator is a datum which provides simple and reliable information describing a variable relative to the context. It gives information about a situation and its evolution in a country, or an area relevant to the assistance policy.

Other types of indicators:

  • Programme indicators, which detail the resources, implementation, results, and, if possible, the impacts of an ongoing activity
  • Evaluation indicators, which, in addition to the evaluation criteria, enable the evaluator to judge the programme's relevance, coherence, efficiency and effectiveness, and support answers to evaluation questions

Indicators are designed by national statistical services and in the context of specific programmes. National statistical services, many donors and international organisations have co-operated to establish standard indicators, to ease comparisons over time and between geographical areas.

The European Commission's use of context indicators in country evaluations:
Economic indicators: GDP, growth, balance of payments
Social indicators: population, unemployment, educational level, health
Indicators of services provided to the population: education, health, drinking water, electrification
Others: indicators for the analysis of poverty in ACP countries

These indicators are often designed to highlight the specificities of a local context without, however, enabling the evaluator to make a comparison between countries (for example, information by comparison of the level of the indicator in another country) or get a global and normative view of the country's situation.

What use can be made of context indicators?

To present the country

Context indicators are usually displayed in the introductory chapter of the evaluation. They deal with:

  • Economic and financial fields (GDP, trade flows, debt)
  • Social fields (demography, occupation, gender)
  • Specific important sectors (education, health, environment)

In certain cases and countries, the emphasis may be put on particular sectors or issues (for example, poverty analysis, conflict analysis). 

To portray the country's level of assistance

Context indicators also describe the nature of the assistance provided to the country. They should indicate the type of assistance, the amounts disbursed, the sector-based allocation of assistance and the European Commission's activity, compared with that of other donors. 

To answer evaluation questions

Context indicators can also be used to answer evaluation questions which need a preliminary presentation. They facilitate the understanding of the country's situation for the readers.

The following table shows a selection of indicators which are internationally comparable. Tanzania's situation is compared with the situation of a group of 7 African countries considered to be similar: Burkina Faso, Gambia, Ghana, Madagascar, Mali, Nigeria and Zambia.

Figure 1 : Governance Research Indicator Country Snapshot (GRICS) in 2004
 

Source: World Bank

* Progress is assessed from the most positive (++) to the most negative (--) evolution with respect to the indicator's evolution over the last 3 to 10 years

How are context indicators selected, collected and used during an evaluation?

A good indicator should:

  • Be relevant: it should correspond to an interest, an objective or a need.
  • Be sensitive: the quantity of measurement must reflect, to a sufficient degree, a change that has occurred in the variable to be measured.
  • Be practical: reliable data should be collected quickly and at a reasonable cost through a rigorous process of construction.
  • Be easy to interpret and use: the concrete, visual and intellectual use of the indicator should be straightforward for the user.

 

European Commission indicators

DG Dev indicators for the appraisal of country assistance performance 

The indicators used for the appraisal of country assistance performance by the European Commission's DG Dev have been constructed with the assistance of various donors, including the European Union Member States, the World Bank, the UNDP and the OECD/DAC).

Their construction is mostly based on the following typology, and particularly on impact indicators.

Figure 2 : Typology of indicators

 

The construction and implementation of such indicators target two goals and require the monitoring of two distinct series of indicators, in order to:

  • Measure the results of the country's policies in terms of poverty alleviation and improvement of living standards
  • Detail the outcomes of sector-based policies which are targeted by the Commission's assistance
EUROSTAT data

EUROSTAT provides Member States with numerous data on several topics which are partly expressed by indicators and ranked into short-term (balance of payments, consumer prices, etc.), long-term (economy and ecology, the business structures, etc.) and structural factors (employment, general economic background, environment, etc.). Moreover, EUROSTAT holds data on trade flows between European Union Members and the rest of the world. 

United Nations indicators

Indicators related to the Millennium Development Goals

For the Millennium Goals, the levels to be targeted and 48 Indicators Development Goals have been identified. They are available on the websites of the United Nations and the World Bank. They deal with more than 200 countries, and their methodologies and their precise definitions are presented to ease their understanding and use. These indicators focus more on outcomes than inputs. 

For example, the progress accomplished in the implementation of a programme aiming at promoting gender equality and women's empowerment is measured with 4 indicators:
 

  • Girls' primary, secondary and higher education enrolment ratios in relation to boys
  • Literacy rate of women aged 15-24 in relation to men
  • Share of women in non-agricultural wage employment
  • Share of women in single or lower houses of parliament

These indicators assess the progress of a country in the achievement of an objective year after year. They can also facilitate comparisons between countries.

Common Country Assessment indicators

In 1997, the United Nations decided to create a system of Common Country Assessment (CCA). In this system, the CCA is used as a tool for analysing the country's development situation and identifying the main development challenges.

Development index

The United Nations database is one of the most developed in the world. Approximately 200 indicators have been developed.

This database is less developing country-oriented than other databases; yet it presents indicators and development indices, which are designed by the United Nations. 

Human Development Report

Each year, the UNDP publishes its Human Development Report, which includes a large proportion of the United Nations indicators. 

World Bank indicators

International Development Association indicators

The International Development Association has recently presented in "IDA Results Measurements System: Progress and Proposal, April 2003" a series of indicators aiming at improving the monitoring of the countries' development outcomes, and particularly for countries benefiting from a Poverty Reduction Strategic Programme.

IDA indicators cover the following fields: income and poverty, malnutrition, maternal and child health, HIV, gender, primary education, drinking water, infrastructure, private sector development, public sector management, and economic growth.

International Comparaison Program

Founded in 1968, the International Comparison Program is a statistical system used to produce data by country. These data facilitate international comparisons based on prices, expenses value and purchasing power parities.

Because of the information about the purchasing power, this statistical system provides the evaluators with comparable data that are valuable for economic and social topics.

Purchasing Power Parities (PPP) are monetary conversion rates which express the purchasing powers of various currencies in a common unit. In other words, PPPs can determine how much national currency is needed to buy the same amount of goods and services in any countries. In that sense, PPPs are monetary conversion rates which erase price differences between countries.

OECD indicators

In its statistic portal, OECD offers a range of precise and updated information about its member countries, from which indicators can be constructed. This descriptive information covers sector-based, social and economic fields.

Precise data (geographical, such as country-, regional-, or sector-based) dealing with non-member economies and their development are available on the OECD website. They describe the context in which assistance to a country is carried out.
The OECD series of thematic and environmental indicators may be usefully consulted.

Other sources

Transparency International indicators

Transparency International seeks to provide reliable quantitative diagnostic tools regarding levels of transparency and corruption, both at global and local levels.

The best known of Transparency International tools is the annual Corruption Perceptions Index. It ranks more than 150 countries in terms of perceived levels of corruption, as determined by expert assessments and opinion surveys. The Global Corruption Barometer (GCB) and the Bribe Payers' Index (BPI) complete the CPI.

NGOs such as Human Rights Watch and Amnesty International publish reports on a large number of countries dealing with human rights and other important issues.

What are the preconditions for its use?

Figure 3 : the preconditions for its use
The time span
  • The time span is the time dedicated to finding available indicators. The evaluator can quickly complete this task using the Internet.
Human resources
  • The human resources required for the collection and selection of indicators differ largely with the themes under study. 
  • The selection of an appropriate indicator, such as maternal health progress, may require the advice of a health expert.
Financial resources
  • The financial resources required for data collection are very limited because indicators are available on Internet, where access is free most of the time.

What are the advantages and limitations of the tool?

Figure 4 : The advantages and limitations of the tool
Advantages
  • A way to quantify information, preferably in a standardised form, in order to make comparisons in time and space. 
  • A way to simplify situations in order to understand them better. 
  • Elements that can be used as evidence for presentation.
Limitations
  • A simplistic explanation of the situation, which, in turn, becomes exaggerated. 
  • Errors of measurement. 
  • Difficulty in developing indicators which are sensitive to slight changes in the context at the macro-economic level. 
  • Problems with the availability of reliable and standardised data over a long period of time. 
  • Differences in the understanding of the meaning of an indicator between various users, and particularly between the donor and the beneficiary country.