Evaluation methodological approach

EVALUATION SCOPE

.

What is this?

The scope of the evaluation is everything that is judged. It is defined in terms of various dimensions such as the territory concerned, the period under consideration and the regulatory framework, for example:

• All the funds allocated by the European Union to Latin America since 1998.
• Local development actions planned in Albania over the past four years.

The central scope of the evaluation matches the very contour of the intervention that is under evaluation. A second broader perimeter specifies the related actions and the elements from the context to be taken into account, especially for the analysis of external coherence/complementarity. This is the extended scope, as opposed to the central scope. 

The central scope is specified in the terms of reference and the extended scope in the inception report.

Why specify the evaluation scope?

• To identify the commissioning body's expectations better and define clearer priorities for the evaluation
• For the evaluation team to focus on priorities and not to waste its resources on areas of secondary interest.

How to delimit the central scope

- in terms of a territorial perimeter

• the world (global evaluation)
• a region
• a country
• one or more areas within a country.

- in terms of the period under consideration

• the entire period during which the evaluated intervention was implemented (case of an ex post evaluation)
• or only that which was implemented during the years [n-4 to n-2] so that the evaluation team can observe real effects in the field
• or by including the preceding and/or next intervention.

- in terms of the regulatory framework

• a project
• a support programme to sector policy
• a country strategy
• a regional co-operation agreement
• an instrument.

- in terms of the sector

Only where relevant:

• an entire sector or field of intervention (education, agriculture, etc.)
• or only a sub-sector
• or a set of sectors.

How to delimit the extended scope

- in terms of related policies

• The interventions of national authorities or other donors covering the same territories
• … or targeting the same groups
• … or addressing the same problems.

- in terms of co-funding

• interventions co-funded with national authorities
• interventions co-funded with other donors.

Recommendations

• Limit the scope to allow more in-depth analysis of effects and a better understanding of the phenomena.
• In the case of co-funded, multisectoral interventions, select a portion of the institutional and sectoral scope for a high-quality evaluation.

A warning

Be careful not to confuse a sample (of individuals, firms, etc.) selected for survey purposes, with the evaluation scope. The sample is used to collect data and to infer results extrapolated for the entire survey group. In this case, the evaluation scope is the surveyed group. 

Don't confuse scope and question, Examples:

• scope = the entire country strategy; question = the effect on gender equality (question)
• scope = multi-donor programme to support the national education policy; question = sustainability of the impact on the management of the education system.

A sector (e.g. education) can constitute a delineation of the scope, but the same does not apply to the cross-cutting issues (e.g. equal opportunities) that resemble questions. For example, a programme to support the national education policy (scope) will be evaluated from the point of view of gender equality (question).

.

EVALUATION OF PROJECTS, PROGRAMMES AND STRATEGIES

.

What does this mean?

The evaluation may concern a simple intervention such as a project that produces effects directly in the field, or a more complex intervention that produces its effects through other interventions at a lower level. The questions will obviously differ, depending on what is evaluated.

Why make the distinction?

• To adjust the questions and the process to the nature of the evaluated intervention.
• To clearly understand that a complex intervention is not evaluated with the same methods as a simple intervention.

What distinguishes the three types of evaluation?

- Evaluation of a project

A project is an indivisible operation, delimited in terms of schedule and budget, and usually placed under the responsibility of a single operator. The simplest projects to evaluate are homogeneous, that is, characterised by:

• One main activity
• Implemented in a single context
• Intended for a single group
• A single direct result expected for that group
• A single expected impact at a more global level.

The evaluation of a simple project is facilitated by the fact of focusing on a single cause (the main activity) and a single effect (expected result for the targeted group). Owing to this simplicity, techniques such as questionnaires or comparison groups can be used. It is also possible to ask many different questions and to answer them satisfactorily. 

A project evaluation often focuses on questions of interest to managers, operators and the targeted group.

- Evaluation of a programme

A programme is a set of simple, homogeneous interventions grouped together to attain global objectives. It is delimited in terms of schedule and budget and usually placed under the responsibility of a monitoring committee. 

A homogeneous programme is made up of parallel and similar projects. Evaluation of a homogeneous programme can consist of syntheses of project evaluations or extrapolation from a sample of projects. 

So-called "integrated" programmes encompass heterogeneous but complementary interventions with a common objective. Such programmes are complex objects in so far as they are designed with:

• Several activities (subsidies, direct investments, technical assistance, etc.)
• Implemented in several contexts (different geographical areas, different sectors, etc.)
• Targeted at several groups
• Expected to produce multiple results for the different groups
• But expected to generate one common impact at a more global level.

The evaluation of a complex, heterogeneous programme is complicated by the fact that multiple causes (activities and instruments) and effects need to be studied, not to mention the synergy effects between the different components of the programme, which in fact justifies the programme's very existence. 
Due to this complexity, some traditional evaluation techniques such as questionnaires or comparison groups cannot be used, although approaches based on case studies remain applicable. 
To avoid adding to the complexity and to preserve the quality of answers, it is necessary to limit the number of questions processed. 
Evaluation of a complex programme is neither the sum nor the synthesis of evaluations of components of the programme. It focuses on questions that are relevant to the programme managers but are seldom if ever addressed at lower levels (effects of synergy, relevance of the sharing of resources between components, contribution to meeting the goal).

- Evaluation of a strategy

A strategy covers a wide set of simple or complex, weakly inter-related interventions. It is defined by priorities rather than as a set of clearly delimited interventions. 

Unlike a programme, a strategy generally has several objectives with differing degrees of priority. 

Evaluating a strategy involves the same difficulties as those encountered in the evaluation of a complex programme. The same approaches are used to deal with these difficulties:

• Focus the evaluation on a few key questions
• Ask questions that specifically concern the strategy and not only components of the strategy.
• Answer these questions with more than simply a combination or synthesis of evaluations of the components.

.

SECTORS, THEMES AND CROSS-CUTTING ISSUES

.

What does this mean?

Public interventions are usually classified by sector or theme. A particular intervention or set of interventions can be evaluated from the point of view of a cross-cutting issue. This is also referred to as a thematic evaluation. These terms need to be clarified.

Examples

How to make the distinction?

- Sectors

Interventions are classified in a particular sector according to the nature of the activities and outputs. For instance:

• Support to training primary school teachers will be classified in the Education sector
• Advice provided to groups of farmers will be classified in the Agriculture sector

- Cross-cutting issues

A cross-cutting (or horizontal) issue is defined in relation to impacts and not outputs, as shown by the two examples in the table below:

- Themes

The term "theme" is sometimes used in the sense of a sector and sometimes in the sense of a cross-cutting issue. It is therefore advisable to redefine it every time it is used.

Recommendations

• Take cross-cutting issues into consideration in the preparation of evaluation questions. Questions on cross-cutting issue s are particularly relevant in the evaluation of a complex intervention such as a strategy or programme.

Warning:

The same term can denote a sector or a cross-cutting issue , as shown in the table below:

TIMING OF THE EVALUATION

.

What does this mean?

An evaluation can be performed before, during or after the evaluated intervention. Depending on the timing, its purpose and use will differ.

What is the point?

• To optimise the resources allocated to the evaluation by launching it at the time it is likely to have the most added value.
• To meet the needs of the main users of the evaluation at the most appropriate time.
• To ensure that a critical mass of results and impacts are already materialised in the field and they are ready for data collection.
• To avoid conflict with the concomitant exercises of review or audit.

Different times of an evaluation

Ex ante

An ex ante evaluation is performed before adopting or implementing the intervention. It gives support to the intervention design and contributes to ensuring the design quality. It is concerned with the following points:

• Need to satisfy in the short or in the long run.
• Objectives to be met.
• Expected results and necessary indicators for their evaluation.
• Added value of the community intervention.
• Risks linked to the proposals.
• Open alternative options.
• Lessons from similar experiences already undergone.

It aims at having a direct influence on the decisions upstream from the implementation, to the extent that it transposes lessons from past experiences into the framework of the new intervention.

Mid-term or final

An evaluation during or at the end of the implementation is intended to draw lessons from the first years of the intervention implementation and to adjust the contents of the ongoing intervention in relation to realities in the field and/or contextual developments. It often includes a report on outputs and an analysis of the first results and impacts achieved. It aims at improving the intervention under way and its conclusions may be supported by observations in the field.

Ex post

The ex post evaluation is performed right after or a long time after completion of implementation. It is mainly concerned with checking achieved impacts, identifying and judging unexpected impacts and assessing the sustainability of the intervention's benefits. 

It enables to detect the real changes in the field and, if the changes occur soon enough, they can be analysed to estimate those that are attributable to the intervention. 

The ex post evaluation often aims to report to the institutions that have allocated the resources. Likewise, it helps to transfer acquired experiences to other countries or sectors. 

Recommendations

• Mid-term evaluation offers a good compromise between utility and reliability, especially if it takes place in the second half of the cycle.
• If the evaluation is to draw on the experience of previous programmes in order to improve future ones, it is recommended to establish a multiannual evaluation plan covering several programming cycles.

.

EVALUATION AND THE INTERVENTION CYCLE

.

What does this mean?

The evaluation can be scheduled:

• For the beginning of the cycle, in the design phase;
• During the cycle (mid-term or at the end of the implementation phase);
• After the end of the cycle.

Is there a best time to evaluate or should three evaluations be performed for each cycle?

What is the purpose?

Many interventions are characterised by successive cycles and show a relative continuity between cycles. 

A new cycle can start before the intervention of the previous cycle has yielded all its effects. 

In light of this, several cycles need to be considered when choosing the timing of the evaluation, in order to:

• Draw conclusions from an observation of the effects of the intervention in cycle "n-1".
• Feed into reflection on the intervention in cycle "n-1".
• Observe the implementation of cycle "n" and rapidly provide feedback.

How to take the cycles into account?

For defining the purposes of the evaluation:

• If the evaluation takes place at the beginning of the cycle, it verifies the relevance and coherence of the objectives of intervention "n" by analysing needs and also the effects of previous interventions "n-1" and "n-2".
• If the evaluation takes place in the middle of the cycle, it aims to improve intervention "n" by taking into account its first observable effects, and also by analysing the effects of intervention "n-1".
• If the evaluation takes place at the end of the cycle, it draws conclusions from the analysis of the first observable effects and also from the effects of interventions "n-1", with a view to preparing the strategy and objectives of the next intervention "n+1".

For choosing evaluation questions

• Retrospective questions relating to the impacts and sustainability of preceding interventions.
• Concomitant questions relating to the direct results and relevance of the intervention under way.
• Prospective questions relating to the strategy of the interventions in the next cycle.

Recommendations

Where possible, ensure that the evaluation ends about a year before the end of the cycle. At that stage it will still be possible and useful to adjust the implementation because it always lasts for a year or two after the end of the cycle. It will also be possible to assist in planning the intervention strategy of the following cycle. Finally, it is not too late to perform a sound analysis of the impacts of the previous cycle's intervention.

USERS OF AN EVALUATION

.

Who do we mean?

Evaluation is intended for a variety of users:

• policy-makers and intervention's designers
• managers, partners and operators involved in the implementation
• institutions that granted funds and to which the managers of the intervention are accountable
• public authorities that conduct related or similar interventions
• actors in civil society
• experts.

What is the purpose?

To optimise the usefulness of the evaluation for the various partners, and especially:

• To ensure that the evaluation meets the expectations of the targeted users, in a way and at a time that fulfills their needs
• To ensure that the evaluation has the required credibility vis-à-vis the targeted users.

Policy-makers and designers

Policy-makers and designers use the evaluation to prepare the launching of new interventions, the reform of existing interventions, the choice of strategic orientations, and decisions on allocation of budgetary, human and other resources, etc. 

They need information that:

• is directly exploitable in the decision-making process,
• arrives on time
• answers their questions, clearly, concisely and reliably.

They are interested in strategic issues, external coherence and global impacts, which constitute the ultimate goal of the intervention.

Managers, partners and operators

The managers are responsible for the implementation and monitoring of the intervention, from headquarters to the field. The actors closest to the public are the operators. Field level operators may either belong to the EC or to partner organisations sharing the responsibility of implementation. 

They use evaluation findings about the results of their action as a feedback. 

They need information that arrives as early as possible to remedy problems or validate changes. They are able to interpret complex and technical messages. 

They are interested in the direct results of the intervention, in the needs and behaviour of the targeted group, and in interactions between partners.

Other actors

The institutions that funded an intervention expect accountability. This applies to Parliament or the Council of Ministers, but also to all the co-funders. The taxpayers and citizens are also addressees of an evaluation. 

The public authorities that conduct related or similar interventions are potential users of the evaluation, especially in the form of transfer of lessons learned. The same applies to the expert networks concerned by the intervention. 

Finally, an evaluation is likely to be used by the actors in civil society, especially those representing the interests of the beneficiary groups.

Recommendations

• From the launch phase, draw up an inventory of the potential users and classify them in the above categories.
• Question key informants to understand the users' expectations.
• Choose the expectations to focus on, especially in relation to the challenges and responsibilities of the institution that initiated the evaluation.
• Draw up a communication and dissemination plan suited to the main users targeted.
• Take into account the different levels of information in relation to the users: strategic for the decision-makers, technical and operational for the managers, general for outside actors.

.

TYPES OF USE

.

What does this mean?

An evaluation can be used:

• As an aid in decision-making.
• As an aid in making judgements and informing public opinion.
• To promote knowledge and understanding.

These three types of use are not mutually exclusive. It is necessary to understand in order to judge, and to judge in order to decide.

What is the purpose?

To optimise the usefulness of an evaluation, that is:

• To choose and draft the questions in relation to the expected use.
• To adjust the schedule and dissemination in relation to the expected use.

Assisting decision making

The evaluation may be undertaken for the benefit of those who have to decide or to negotiate an adjustment or reform of the evaluated intervention. In that case it is used to adjust the implementation, to design the next intervention cycle or to redefine political orientations. 

To facilitate this type of use, also called feedback, the evaluation questions must be asked in relation to the decision-makers' expectations and to their planned decision-making agenda at the time the report is submitted. 

The evaluation can aid decision-making in two different ways:

• By formulating conclusions independently, and then proposing recommendations. This type of evaluation is referred to as "summative".
• By favouring the involvement of the decision-makers concerned, or at least their close collaborators, with a view to creating a phenomenon of take-up or direct feedback. This type of evaluation is referred to as "formative".

Evaluations may assist decision-making in different ways, depending on the context of the decision:

• Recommendations may be made to the managers and/or designers of the intervention. Evaluations that favour this type of use are referred to as "managerial".
• Recommendations may be made to all the partners and co-funders of the intervention. Evaluations put to this type of use are referred to as "partnership".
• Finally, an evaluation may be conceived as an aid to negotiation and problem-solving between a wider range of stakeholders, including interest groups and actors in civil society. Evaluations used for this purpose are referred to as "pluralistic".

Assisting the formulation of judgements

The evaluation may help users to shape their opinion on the merits of the intervention. 

The formulation of an overall assessment is particularly useful for accountability purposes. In this case, the evaluation examines the merits of the intervention in relation to the different points of view (summative evaluation). It answers questions that are important for the funding institutions. The report is accessible to the general public. The independence of the evaluation and the transparency of the judgement are highlighted. 

In this instance, particular attention is paid to the definition of judgement criteria (also called "reasoned assessment criteria"). Yet the judgement itself is definite only when the final report is submitted and its conclusions are discussed. Using the evaluation for accountability purposes therefore means having to wait for the end of the process.

Knowing and understanding

Apart from assisting in making decisions and formulating judgements, which are the two main forms of use, the evaluation may also enable users to learn from the intervention, to better understand what works and what does not, and to accumulate knowledge.. Indirectly, it contributes to transferring knowledge thereby acquired, to the benefit of professional networks that may not have a direct link with the evaluated intervention. 

Unlike feedback, which directly concerns those responsible for the evaluated intervention, the transfer of lessons is an indirect process that takes place through networks of experts both within and outside the European Commission. 

Capitalising on knowledge often starts during the evaluation process, through the experts who belong to the evaluation team or reference group. However, the transfer of lessons learnt may only occur after the final report has been delivered. A key step in this perspective is presentation of the evaluation in specialised networks, in the form of seminars or technical articles.

Recommendations

From the outset, the evaluation manager should prioritise one or more types of use, and then optimise the evaluation process in order to make it as "user friendly" as possible, for instance in adjusting:

• Membership of the reference group.
• Evaluation questions.
• The dissemination strategy.

.

EVALUATION AND DECISION MAKING

.

What is this about?

Evaluation provides feedback and thus facilitates decision-making, for instance in adjusting the implementation of the intervention, designing the next cycle, or helping to redefine political priorities. In this context the formulation and follow-up of recommendations are key steps in the process.

What is the purpose?

Facilitating future decision-making is generally the main type of use of evaluation. Decision-makers' needs have, therefore, to be taken into account throughout the evaluation process to increase the chances of the evaluation being useful to them. 

What is the link between evaluation and decision-making?

Some evaluations are designed primarily to provide information for management decisions or for reform of the evaluated intervention. They are intended for operational actors in the field, management services, and the authorities responsible for the intervention or their partners. In this perspective, mid-term evaluation is to be preferred and careful attention needs to be paid to the formulation and follow-up of recommendations. These evaluations are referred to as formative. 

Other evaluations are designed primarily to learn lessons from the experience and to serve decision-making in other contexts. In this perspective, ex post evaluation is the most appropriate and careful attention must be paid to the formulation and transferability of the lessons learned. These evaluations are referred to as summative. 

Yet one has to be realistic: decision-makers will not necessarily follow the recommendations and lessons. The decision-making process almost always involves many actors and multiple factors, evaluation being only one of them.

Advice for performing a decision-making oriented evaluation

• Hold targeted interviews, from the outset, in order to establish exactly what decision-makers expect and to anticipate the decision-making agenda.
• Target the questions and determine the evaluation schedule in relation to the decision-making process.
• Involve individuals in the reference group whose point of view is close to decision-makers'.
• Organise quality assessment, the drafting of documents and their dissemination with a view to decision-makers' needs.
• Emphasise the quality of recommendations and verify their operational feasibility from the decision-makers' point of view.
• Stick to the schedule to allow timely feedback in the decision-making process.
• Follow-up recommendations by asking the decision-makers concerned to respond rapidly to them and after one year to report on decisions made accordingly.

.

EVALUATION AND KNOWLEDGE TRANSFER

.

What is this about?

Evaluation is a learning-intensive exercise in so far as lessons learned from experience can be capitalised on, and knowledge acquired can be transferred and reused by actors who have no direct link with the evaluated intervention. In this context the identification of good practices and transferable lessons is a key step.

What is the purpose of knowledge transfer?

• Without an effort to capitalise on and manage knowledge, organisations tend to forget the lessons learned and to repeat costly efforts.
• Evaluation reveals and validates certain knowledge of the actors involved in the interventions, which may be of use to people working in other countries, institutions or sectors.

Draft the report to promote the accumulation of knowledge

The only pages of a voluminous evaluation report that are oriented towards knowledge transfer are those concerning good practices and lessons learned. It is therefore important to identify them and to draft them with that in mind. 

Identifying a good practice means judging that it produced the expected effects in a particularly effective, efficient or sustainable way. 

Identifying a transferable lesson means judging that a given practice generally succeeds (or fails) to produce the expected effects in a given context. 

The summary of the evaluation report highlights the main lessons learned with a view to facilitating their accumulation and transfer. For each lesson, references to the body of the report serve to specify:

• The exact content of the lesson learned.
• The soundness of the conclusions on which it is based.
• The context in which knowledge was learned and the factors of context that make it transferable or not.

Mobilise available knowledge

Certain public institutions or professional networks accumulate knowledge in Intranet or Internet databases. Knowledge may also be accumulated informally through expert networks that build on lessons learned in a particular sector or on a particular topic. In the latter case, the members of the network try to validate the knowledge before capitalising on it, through meta-evaluations or expert panels. 

Not only does evaluation contribute towards the accumulation of knowledge, it also facilitates the circulation of the acquired knowledge. The evaluation team mobilises available expertise and documentation to provide an initial partial answer to the questions asked. If the transferable lessons have been learned through other evaluations and properly capitalised, the evaluation team identifies and reuses them. 

.

DISSEMINATION OF THE EVALUATION

.

What does this mean?

Dissemination concerns the final evaluation report, as well as all other means of publicising the conclusions, the lessons learned and the recommendations. Dissemination activities target the services of the Commission, European Institutions, external partners, networks of experts, the media and the wider public.

What is the purpose?

The dissemination process promotes the use of the evaluation if it is done well. It serves to:

• Transmit the key messages of the evaluation to decision-makers, designers, managers, operators and partners concerned by the evaluated intervention (feedback).
• Report to the authorities and institutions concerned (accountability).
• Further knowledge within professional networks.
• Influence opinions throughout society at large.

Which measures need to be taken?

• Plan dissemination when drafting the terms of reference, especially by specifying how the report will be published and what the evaluation team's role will be in that phase.
• Throughout the process, keep in mind the quality of the evaluation and its products (report, summary, annexes), and formally assess the quality at the end of the process.
• In the last evaluation reference group meeting, identify the main messages delivered by the evaluation and the targeted audiences.
• After approval of the report, finalise the communication plan by choosing the messages to be highlighted and the most suitable information channels for each audience.
• Ensure the necessary cooperation for the implementation of the dissemination plan and divide the work and responsibilities between the evaluation manager, the team (check that this is part of its mission and is specifically mentioned in the terms of reference) and the members of the reference group.

Which channels for dissemination?

The evaluation report is disseminated on the Internet and is thus accessible to all audiences. More active dissemination is also undertaken for specific audiences:

• The report and/or its summary are sent to the services concerned and to the partners.
• A one-page summary is written specifically for the hierarchy of the service that managed the evaluation. It highlights the main conclusions and recommendations.
• A summary is also published on the relevant Intranet sites, with a link to the report.
• A two to three pages abstract is sent to the OECD for publication on its website. This summary is intended for the international development aid community. It highlights the lessons learned if they are transferable.
• One or more articles may be written for the general public or specialised networks.
• Finally, the report may be presented in meetings, workshops or seminars.

Recommendations

• Draw up the rules for quoting names in the report in relation to the intended dissemination.
• Draw up a budget for the communication plan at the launching stage of the evaluation (mention whether the external evaluation team has to include it in its proposal).

.

DOCUMENTS PRESENTING THE EVALUATION

.

What does this mean?

Since full-length evaluation reports are available on the Internet, it is advisable to disseminate one or more shorter documents that are suited to the different audiences and present the evaluation in an attractive and accessible way.

What is the purpose?

These documents are intended to:

• Inform decision-makers of the main conclusions and recommendations.
• Inform the wider public, experts and the media of the existence and content of the evaluation report.
• Facilitate knowledge transfer, particularly best practice and lessons learned.
• Promote accountability and democratic use of evaluation.
• Encourage people to consult the report on the Internet.

Formats and uses of the different documents

The executive summary is one of the starting points for drafting presentations. It is usually three pages long and no particular audience is targeted in the content. Beyond the executive summary, various documents are or may be produced:

• A one-page note focused on conclusions and recommendations at a strategic level and intended for policy-makers.
• A two-page abstract published by the OECD (EvInfo) to inform the development aid community. As far as possible, this abstract mentions transferable lessons learned.
• A paragraph presenting the evaluation in synthesis documents.
• A short article intended for a particular audience.

Advice for drafting a presentation document

- Choose the orientation of the document

• Define the target: is the document for the public at large or for specialists and, if the latter, which ones? 
• Select a few messages to highlight in the title and sub-titles, in relation to the targeted audience. 
• Adjust the length of the document and the style to suit the targeted audience.

- Describe the evaluated intervention

• Who initiated and financed the intervention? When?
• Budget and typical outputs.
• Expected effects and rationale.

- Describe the evaluation

• Who initiated the evaluation? Why?
• Evaluation scope.
• Main evaluation questions asked.

- Main messages

• Findings and new knowledge acquired.
• Conclusions.
• Transferable lessons and replicable good practices.
• Recommendations.

- Method

Describe the methodological design in a non-technical way and make a statement about the soundness of the main messages. 

If one aspect of the method is of particular interest to the targeted audience, add a more technical boxed-in section.

Recommendations

• Avoid the technical jargon of evaluation, the specialised vocabulary of the sector of intervention, and administrative acronyms, especially if the document is intended for the general public.
• Focus on a few key messages and propose hyperlinks towards the executive summary and the report available on the Commission's website.

.

PRESENTING AN EVALUATION IN A SEMINAR

.

What is the purpose?

In an environment with an over-abundance of written information, an effective way of reaching potential users consists in presenting the evaluation results in meetings or seminars. 

An oral presentation of the evaluation helps to:

• Facilitate knowledge transfer, especially best practice and lessons learned in professional networks
• Inform people who will relay messages to decision-makers or to a broader audience
• Draw the attention of the institutions to which one is accountable and reinforce the democratic use of evaluations
• Discuss the conclusions.

Practical advice

A short presentation (10 to 20 minutes) is enough for the main points but more time needs to be left for questions (20 to 40 minutes). 

The presentation covers the following points:

- The evaluated intervention

• Who conducted the intervention, when, with which resources? What were the outputs, objectives and rationale?
• Main lines of the intervention logic

- The evaluation

• Who decided on the evaluation and why?
• Who carried out the evaluation and who is responsible for the conclusions?
• Which part of the intervention was evaluated and which questions were asked?

- Messages resulting from the evaluation

A few particularly important messages from the point of view of the people participating in the meeting or seminar (key data, findings, conclusions, transferable lessons and/or recommendations).

- Strengths and weaknesses of the messages

Explanation of the methodology employed and reasons for which a particular message is sound (valid) or fragile. Recommendations for using messages if they are fragile.

Recommendations

• Adapt the level of language to the audience, e.g. without jargon for a mixed audience, more technical for an audience of sector experts.
• Indicate how to access the final report and the summary, especially via Internet.

.

THE DISTRIBUTION OF ROLES

.

What does this mean?

The three key players are the evaluation manager, the reference group and the external evaluation team. The distribution of roles is specific to each phase and each stage of the evaluation.

All the actors in the evaluation process have to know what their role and responsibilities are, for example: Who is responsible for the recommendations? Who has to assess the quality of the report?

What is the purpose?

• To mobilise the different actors, depending on their specific competencies and resources.
• To avoid deadlocks due to possible confusion regarding roles.
• To preserve the impartiality of the evaluation and avoid biased judgements.

Who does what?

The three key players in an evaluation are the evaluation manager, the reference group and the external evaluation team. Their roles differ, depending on the following phases and steps:

- Decision to evaluate

The decision is taken by the authorities responsible for the intervention and is reported in a document such as:

• the annual evaluation programme
• the document scheduling the intervention.

The decision refers to an evaluation framework that defines the rules of the game. If necessary, it completes this framework.

- Preparation (phase 0)

Constitution of the reference group:

• by the evaluation manager.

Drafting of the Terms of Reference:

• by the evaluation manager.
• after consulting the reference group.

Selection or approval of the external evaluation team:

• by the evaluation manager,
• after consulting the reference group.

- Desk (1)

Clarification of the intervention logic:

• by the external evaluation team,
• after interaction with the evaluation manager and the reference group.

Choice and drafting of evaluation questions:

• by the evaluation team,
• validation by the reference group,
• arbitration by the evaluation manager if necessary.

Definition of the judgement criteria (also called reasoned assessment):

• by the external evaluation team,
• approval by the reference group,
• arbitration by the evaluation manager if necessary.

Choice of indicators:

• by the external evaluation team,
• after consulting the reference group.

Design of the method (data sources, tools and work programme):

• by the external evaluation team,
• after consulting the reference group.

Collecting available data at the head office of the Commission and/or the Delegation (s):

• by the external evaluation team,
• if necessary, with the aid of members of the reference group for access to informants and in interaction with the evaluation manager.

- Field (2)

Data collection:

• by the external evaluation team,
• if necessary, with the aid of members of the reference group for access to informants, and in interaction with the evaluation manager in cases of serious problems.

- Synthesis (3)

Report, including the conclusions and recommendations:

• by the evaluation team,
• after interaction with the reference group.

Verification of the quality of the report:

• by the evaluation manager, possibly with an external support,
• after consulting members of the reference group.

- Dissemination and follow up of recommendations (4)

Internal and external dissemination:

• by the evaluation manager

Use of the conclusions and recommendations:

• by the Commission's services concerned
• with follow up by the evaluation manager.

.

THE EVALUATION MANAGER

.

Who is this?

The evaluation manager is a member of the service in charge of the evaluation. He/she manages the entire evaluation process, from A to Z on behalf of the commissioning body.

Why appoint an evaluation manager?

• To ensure consistency throughout the evaluation process, from the terms of reference to the dissemination of the report and the follow-up of recommendations.
• To be the contact person for administrative issues and to coordinate the activity of the different actors (reference group and evaluation team).
• To organise, supervise and assess the quality of the different phases of the work.
• To ensure the smooth running of the evaluation.

What does he / she do?

• He / she forms the reference group after consulting the heads of the services concerned.
• He / she drafts the terms of reference after consulting the reference group members.
• He / she engages the external evaluation team after consulting the reference group.
• He / she helps the evaluation team to get access to information and informants.
• He / she organises the discussion and approval of conclusions and recommendations in interaction with the reference group and the evaluation team.
• He / she performs a methodological quality assessment of the reports, double checked by a second person.
• He / she disseminates the evaluation to the different actors concerned.
• He / she monitors whether the recommendations are taken into account in interaction with the authorities concerned.

Recommendations

• Appoint a deputy to help the evaluation manager on a daily basis and to stand in when he/she is absent.
• Appoint an evaluation manager who knows both his/her institution and the evaluation methods.
• Ensure that the evaluation manager is not too close to the managers of the evaluated intervention.

.

THE REFERENCE GROUP

.

What is this?

A small group gathering the services concerned by the intervention and possibly other stakeholders. One or more experts may also be invited to participate.

The reference group is an interface between the evaluation manager and the external evaluation team. Its members help the external evaluation team to identify data sources and to access them. They validate the evaluation questions, and discuss the conclusions and recommendations.

The reference group allows the variety of points of view on the evaluated intervention to be expressed.

What is its role?

Reference group members:

• Discuss and comment on the terms of reference drawn up by the evaluation manager.
• Aggregate and summarise the views of the Commission services and act as an interface between the evaluation team and the services, thereby supplementing bilateral contacts.
• Ensure that the evaluation team has access to and consults all information sources and documentation on activities undertaken.
• Validate the evaluation questions.
• Discuss and comment on notes and reports produced by the evaluation team. Comments by individual members of the reference group are compiled by the evaluation manager and subsequently transmitted to the evaluation team.
• Assist in feedback of the findings, conclusions and recommendations from the evaluation.

Whom to invite and how?

Different types of actor can be invited to participate in the reference group. If the evaluation is managed at headquarters, membership includes:

• Services of the Commission concerned by the intervention, including the Delegation concerned in case of a country/region level evaluation.
• Specialists on a sector or a cross-cutting issue within the Commission.
• Embassy of the partner country in the case of a country level evaluation.

If the evaluation is managed in the partner country, membership may extend to:

• Government services.
• Selected development partners.
• Selected civil society organisations.
• Experts

Experience has shown that it is preferable for the group to be limited in size (10 to 12 members) if it is to function effectively.

After identifying the services, institutions or organisations to invite to the reference group, the head of the service managing the evaluation sends them an invitation to:

• Officially announce the launching of the evaluation
• Request that they appoint one person as a member of the reference group.

When should it be asked to meet and how should it be involved?

The reference group is generally involved in the following way:

Opening the evaluation process

In addition to inputs made by the reference group, the evaluation may benefit from a wider variety of views through other channels like:

• Consultation of key stakeholders by the evaluation team in the first stage of the process
• Gathering of participatory workshops, focus groups or expert panel(s) during the evaluation process
• Discussion of the draft evaluation report in a widely attended seminar in the partner country (when relevant).

Experts invited to the reference group

What does this mean?

One or more members of the reference group may be invited as a resource person, for their expertise.

Why invite them?

• To orientate the evaluation work towards the acquisition of new knowledge and to learn from others' experience.
• To describe the point of view of certain stakeholders, where relevant.
• To contribute towards quality assessment of work in their field of expertise.
• To facilitate knowledge transfer.

Who should be invited?

• The experts invited may or may not belong to the services of the Commission.
• It is preferable for them to be familiar with the entire evaluation field.
• As part of the global network they can mobilise knowledge about the entire development aid community.

Charging a reference group meeting

What to do?

• Prepare a precise agenda and ensure that you adhere to it: (see: inception meeting, first phase meeting, debriefing meeting, final discussion meeting)
• Ensure that the discussions are balanced, impartial and constructive.
• Draft clear and synthetic minutes that highlight the points debated and report on points of agreement (this task is preferably accomplished by the deputy manager).

What to do in case of disagreement?

In case of lasting disagreement on the evaluation method, the service in charge of the evaluation arbitrates. If necessary, the decision is taken by the head of the Unit.

.

THE EXTERNAL EVALUATION TEAM

.

What does this mean?

The external evaluation team is responsible for data collection and analyses and for the evaluation report, including the formulation of judgements (also called reasoned assessment) and the drafting of conclusions and recommendations. It interacts with the reference group and the evaluation manager. It provides evaluation services to the commissioning body under contract.

The members of the external evaluation team are independent of the organisations that participated in the design and implementation of the evaluated intervention. They belong to national or international research organisations or consultancy firms, or are experts working independently. Where possible, some members of the team are from the country or countries targeted by the evaluated intervention.

Why contract an external evaluation team?

• To ensure the independence, impartiality and credibility of the evaluation process.
• To provide expertise on evaluation methods and techniques.
• To provide expertise in the sector(s) concerned.

How to select the team?

The external evaluation team is selected on the basis of proposals via different procedures, e.g. pluri-annual evaluation service contract or specific call for tenders.

The evaluation manager engages the team after consulting members of the reference group.

The following criteria are used as guidelines:

• Relevant experience in evaluation.
• Knowledge of the evaluated development aid modalities.
• Knowledge of the evaluation scope (e.g. sector, country).
• Presence of local experts and consultants.
• Team leader's managerial skills (organisation, coordination, dialogue with the client, etc.).
• Team's independence as regards the evaluated intervention and absence of conflicting interests.
• Evaluation skills and particularly the ability to formulate judgements, to draw up conclusions and recommendations and to draft syntheses.
• Mastery of data collection and analysis tools.
• Diversity of team members' profiles and expertise.
• Price.

The evaluation manager sets the relative weight of criteria and specifies it in the terms of reference.

Recommendations

• Involve local consultants, as far as possible, to promote the development of local capacity (development of national evaluation expertise) and benefit from their close knowledge of the field.

Evaluation capacity

What does this mean?

Having evaluation capacity implies sufficient knowledge and experience to:

• Guarantee impartiality in the choice of questions and the formulation of conclusions
• Clarify the intervention logic
• Design and apply a data collection and analysis method suited to the questions and context
• Guarantee the reliability of data and the validity of analyses
• Treat value judgements in a structured, transparent and impartial way
• Distinguish the findings, conclusions, lessons and recommendations
• Present evaluation work synthetically
• Optimise the usefulness of the evaluation for different audiences.

Evaluation capacity is required both:

• in the service managing the evaluation
• and in the external evaluation team.

How to assess professional experience?

An evaluation professional meets most of the following criteria:

• Varied evaluation experience (regions, themes, sectors, tools)
• Training and experience in evaluation methods and tools
• Active participation in one or more professional evaluation networks
• Practical experience.

.

THE STAKEHOLDERS

.

What does this mean?

The stakeholders are individuals, groups or organisations that have responsibilities and/or direct or indirect interests in the evaluated intervention. They may be affected by the intervention or not.

Why identify them?

• To take their points of view into consideration
• To benefit from their expertise and knowledge
• To encourage them to use the evaluation.

How to take them into account?

• Identify the different stakeholders from the outset, on the basis of available documentation and interviews.
• Identify their expectations and take them into account when drafting the evaluation questions.
• Analyse each one's possible contributions (information, expertise, contribution to the interpretation of data).
• Choose an option to benefit from their contribution, for example: individual interviews, participatory workshops, focus groups, expert panel(s), integration in the reference group, etc.
• Choose the modalities of dissemination of the evaluation results to ensure that they reach the stakeholders.

Recommendations

• Talk to the stakeholders to know what their needs and priorities are.
• Involve the stakeholders in the interpretation of information to avoid mistakes and reach sounder conclusions.

What does this mean?

.

Monitoring, auditing and evaluation are all exercises in sound public management. They are similar and complementary but cover different issues.

All three contain value judgements but the judgement references are not the same. 

.

What is the purpose?

.

• To avoid misunderstandings and deceptions. If the external evaluation team deals with questions that fall under auditing or monitoring, it may not answer them correctly because it lacks the appropriate resources.
• To reformulate a question formulated in a spirit of auditing or monitoring so that it becomes an evaluation question.
• So that informants adopt an attitude of cooperation with the evaluation, knowing that the relationship with an evaluator is not the same as one with an auditor.
• To promote synergy between the three exercises.

.

What distinguishes the three exercises?

.

Subject of study

• An audit (in the traditional sense) verifies the legality and regularity of the implementation of inputs.
• Monitoring verifies the sound management of interventions and informs on the progress and use of inputs, and on the progress of outputs and results. Monitoring produces systematic information with a short periodicity, unlike auditing and evaluation.
• Evaluation judges the design and implementation of the intervention from the standpoint of results and impacts mainly, whether they were expected or not.

Judgement criteria

• Auditing judges on the basis of criteria that are known and clarified in advance (budgets, regulations, management standards).
• Monitoring judges progress in the use of inputs and the delivery of outputs, in relation to operational objectives. The quality of outputs is judged along professional standards.
• Evaluation judges the consequences of the intervention in relation to social needs, economic challenges, environmental problems, etc. It refers to objectives but often has to clarify them. The judgement criteria that it uses are seldom established in advance.

Time scale

• Auditing is carried out periodically, according to pre-established rules, or ad hoc if a risk analysis justifies it.
• Monitoring is a continuous process of collection, with frequent reports (one to four per year).
• Evaluation is an in-depth exercise that mobilises complex tools for collecting and analysing data, and that cannot be carried out annually.

Use

• Audit is intended to detect and prevent irregularities.
• Monitoring gives warnings and allows rapid feedback in case of problem. It is also used for reporting on performance throughout the management chain, from the field operators to the top managers.
• Evaluation furthers understanding on the effects of interventions. It makes recommendations for reform, suggests strategic and political options, and puts forward proposals to adjust the implementation.

The required professional qualities

Each exercise requires an appropriate professional capacity and distinct core competencies:

• The auditor has legal and accounting competencies. His/her action is authoritative.
• The monitoring manager has sound professional knowledge in the field of the intervention and management skills. His/her action is of a supervisory nature.
• The evaluation team has investigative skills and sector-specific expertise. It requires the cooperation of the actors concerned by the intervention to collect relevant data, interpret it correctly and formulate a legitimate judgement.

.

How to ensure complementarity between the three exercises?

.

A good monitoring system considerably facilitates the evaluation. When an evaluation deals with intended effects, the first thing to do is to check whether these effects have been monitored, which indicators have been used, and whether baseline data have been recorded. In turn, evaluation reports often include suggestions for improving the monitoring system.

Auditing and monitoring may detect apparent success or failure and recommend that the evaluation deepens the causes and effects analysis, clarifies the judgment criteria, or validates a good practice.

An evaluation can show that the right application of a legal or professional standard is a factor of either effectiveness or ineffectiveness, and therefore provides feedback into auditing or monitoring practices.

.

Recommendations and clarification

.

• If illegal practices or bad management are assumed, then it is better to opt for an audit, to stabilise the situation and to postpone the evaluation.
• If monitoring information is absent or unreliable, it complicates evaluation but does not preclude it. The insufficiency of monitoring should never be used as a reason to refuse to deal with an evaluation question.
• The auditing practice described here is the traditional practice of regulatory audit. Sometimes professional auditors may be requested to judge results and impacts and to answer questions of an evaluative nature. This practice, often called performance auditing, can become equivalent to evaluation. In this case the auditors adapt their attitudes, judgement criteria and professional competencies. The absence of a clear boundary between performance auditing and evaluation is often a cause of confusion.
• The monitoring practice described here is the traditional one. Traditional monitoring systems are increasingly integrated into performance assessment frameworks. Performance monitoring involves the setting of performance targets in terms of outputs and results, the systematic measurement of progress towards targets, periodic performance-oriented discussions, and incentives for good performance.
• The monitoring practice described here is also different from the EC's Result Oriented Monitoring (ROM). The ROM system rates all external aid projects and programmes with a view to the achievement of intended results and impacts. However, the assessment is not continuous and does not rely on systematically quantified indicators.

RATIONALE OF THE INTERVENTION

.

What is this?

The rationale of an intervention is to address the needs, problems or issues that are considered to be priorities in a given context, and that cannot be addressed more effectively in another way.

It is in the programming stage that the rationale of an intervention must be justified. At the evaluation stage it is enough to note the main points or to redefine them if the programming documents lack precision.

The evaluation identifies the reasons for which the priorities have been chosen, for example: priorities of the policy in which the intervention takes place, urgency of the needs to be satisfied, and comparative advantages compared to alternative options.

What is the purpose?

• To help in defining the criteria of the relevance family.
• To present the intervention in the evaluation report.

How to clarify it?

.

- Reviewing the problems and responses

The evaluation report succinctly sets out the following:

• Context of the intervention when it was initiated
• Main problems diagnosed (needs, issues)
• Why the institution responsible for the intervention and its partners (where relevant) are in the best position to solve the problems diagnosed.
• Why certain strategic options have been chosen rather than others.

The evaluation team searches for this information in the official documents that instituted the intervention and in the preparatory design work. It completes its understanding with interviews with the key informants. 

Certain evaluation questions may concern problems addressed by the intervention, for example:

• Are there alternative options for solving the problems identified?
• Has the nature of the problems that justified the intervention changed?
• What is the precise extent of the needs justifying the intervention?

- Reviewing the rationale

Where relevant, the evaluation report highlights the following:

• Justification of the fact that the needs, problems and issues cannot be solved ad hoc by private initiatives.
• Justification of the fact that they cannot be solved more effectively by other public initiatives.

The evaluation team looks for this information in the official documents that instituted the intervention. It completes its understanding of the rationale by interviews with key informants. 

Certain evaluation questions may concern the rationale of the intervention, for example:

• Could the institution that initiated the intervention have chosen other partners to address these problems?
• Would the intervention have been more effective had it been run by other public institutions?

Recommendations

• The official documents often focus on the strategic options that were finally selected. If we want to know what the alternatives were, it may be useful to hold interviews with key informants, whilst remaining wary of possible biases.

.

INTERVENTION LOGIC

.

What is this?

All the activities and expected effects (outputs, results and impacts) of an intervention, as well as the assumptions that explain how the activities will lead to the effects in the context of the intervention.

The intervention logic may be "faithful" to the programming documents and to the documents establishing the policy to which the intervention is related. In this case, the expected effects are inferred from the stated objectives in the official documents.

When the intervention logic is reconstructed during the evaluation, implicitly expected effects that were not mentioned in the initial documents may be taken into account. The fact that this is no longer a "faithful logic" must then be mentioned. The "faithful" approach is relevant when the objectives are expressed precisely and in a verifiable way. The other option is preferable if objectives are too vague or ambiguous.

The intervention logic often evolves over time. In such cases, it is justified to reconstruct it several times, for successive periods.

The intervention logic is a useful simplification of reality, but one has to bear in mind the complexity of the real world. In addition to the reconstruction of the intervention logic, it is useful to identify the main external factors that condition or limit the implementation and effects. One also has to remember that real causal explanations are often more complex than initial assumptions.

What is the purpose?

• To help to clarify the objectives and to translate them into a hierarchy of expected effects so that they can be evaluated.
• To suggest evaluation questions about these effects.
• To help to assess the internal coherence of the intervention.

How can it be reconstructed?

- Collect and analyse reference documents:

• Collect the official documents establishing the intervention and allocating resources.
• Identify the main activities.
• Identify the objectives.
• Translate the objectives into expected results and impacts.
• Connect the activities to the expected impacts by reconstructing the logical cause-and-effect relations (if… then…).
• Check that cause-and-effect relation are logical, i.e. considered as plausible in the light of available knowledge.
• Discuss the reconstructed logic with a few key informants (designers and managers) and with the experts of the policy domain/country concerned.
• Present and discuss the intervention logic in a reference group meeting.

The most common presentations

.

- Logical framework

This technique consists in producing a presentation in the form of a matrix that specifies the objectives (first column), how they should be verified (second column: indicators, and third column: sources), and the main assumptions on the external factors (fourth column).
The lines of the logical framework have a simplified presentation of the intervention logic, on only four rows: activities, results, purpose (i.e. direct benefit for the targeted group) and goal.
This representation is adequate for a simple intervention, like a project or well-targeted programme. However, it cannot fully grasp the complexity of a heterogeneous intervention such as an integrated rural development programme, a country strategy or support for a sector policy.
Due to its simplified nature, the logical framework allows for specifying the indicators that should be used at each row, as well as most external factors.

- Objectives diagram

This technique consists in the identification of officially stated objectives and a graphical presentation of the logical relations between objectives, from the most operational to the most global. The intervention logic is represented in the form of boxes and arrows.
A particular form of representation is the objectives tree. It is applied in the case where each objective on a lower rank contributes towards a single objective on a higher rank.
Unlike the logical framework, the diagram can have as long chains of objectives as necessary. It can be used to highlight synergies and complex relations between objectives (except in the case of the objectives tree).
This representation is appropriate for complex interventions such as integrated programmes, country strategies or sector policies.

- Diagram of expected effects

This technique is similar to the diagram of objectives since it also builds upon officially stated objectives. However, the objectives are translated into expected effects before being presented as a diagram. By translating objectives into expected effects, more concrete and easily verifiable concepts can be worked on.

Recommendations

• Always re-examine the intervention logic. In the ideal world the intervention logic should be part of the intervention design and should be available at the time of the evaluation in the form of a framework or diagram. In the real world these points should be verified.
• In the case of complex interventions (and especially country level evaluations), it is useful to produce several detailed diagrams and a synthesis diagram. A highly complex diagram cannot be used.
• Beware of the risk of ex post rationalisation. To reduce this risk, clearly explain the reconstruction process and the options chosen.

.

OBJECTIVES

.

What are these?

Objectives are the initial and formal statements on what is to be achieved by the intervention. They may relate to outputs, direct results or impacts. 

Objectives may be expressed in a verifiable way, which specifies clearly the expected achievements. They may be expressed in more political or rhetoric way, making difficult to connect them to a precise achievement. 

The evaluation process may uncover implicit objectives, i.e. cause-and-effect assumptions that necessarily explain how an objective may be achieved.

What is the point?

Identifying and clarifying objectives is a necessary step towards:

• Reconstructing the intervention logic.
• Asking questions and defining evaluation criteria from the families of relevance and effectiveness.
• Judging internal coherence.

How to clarify them?

- Examining the clarity of objectives

The evaluation identifies the objectives and, for each one, sets out the following:

• statement of the objective
• possible ambiguities in some of the terms and clarifications proposed
• means proposed for checking whether the objective has been achieved
• date at which the objective has to be achieved.

The evaluation team searches for the objectives in the official documents that instituted the intervention and in the documents framing it at a higher political or strategic level. 

When the statement of an objective contains ambiguities, the evaluation team proposes clarifications based on interviews with key informants.

- Examining the hierarchy of objectives

The evaluation ranks the objectives of the intervention:

• global objectives correspond to the rationale of the intervention,
• operational objectives correspond to the products and services delivered by the intervention (outputs),
• one or more levels of intermediate objectives can be identified.

The reconstruction of the intervention logic helps to differentiate the levels of objectives as well as the effects, direct finding s and impacts corresponding to them. 

Some evaluation questions may concern the articulation of objectives, e.g.:

• Is there a high level of internal coherence between the operational objectives and the more global objectives?

Recommendations

If the intervention is framed by numerous documents on a political or strategic level, or if these documents also contain many objectives, the evaluation team is faced with an over-abundance of objectives. Rather than presenting them all, the evaluation team underlines this fact and concentrates on the most important ones in the context of the evaluated intervention. 

There is a continuum between more or less explicit objectives:

• an objective clearly stated in the official documents
• an ambiguous objective clarified with the stakeholders' help
• an intermediate effect that has to be obtained if an objective is to be met (implicit objective)
• an effect expected by the stakeholders but not initially anticipated, even implicitly

All these elements warrant explanation in the report, but the status of each one must be specified.

.

RELATED POLICIES

.

What does this mean?

The idea is to identify the main related interventions and to situate the evaluated intervention in relation to them.

What is the purpose?

• To help ask questions belonging to the coherence/complementarity and relevance families
• If necessary, to examine the quality of the intervention design and especially the fact that the objectives are complementary to those of the other related policies.

How to proceed?

The evaluation report briefly sets out the following:

• Main national and international institutions active in the same region and sector and/or targeted at the same group.
• Main policies implemented by these institutions in the same region, in the same sector and/or targeted at the same group.

This analysis offers a brief insight of the main related policies, if possible with their starting and finishing dates, the tools used, the groups and objectives aimed for, and the resources allocated. 

Certain evaluation questions may concern related policies for a more in-depth examination, for example:

• Is the intervention consistent with and complementary to the policies and priorities of the partner country and/or other Community policies?
• Has the implementation been coordinated with the actions of the other sponsors and has the complementarity been improved?
• To what extent is there value added when the intervention is conducted at EC rather than member-State level?

JUDGEMENT CRITERION

.

What does this mean?

A judgement criterion specifies an aspect of the evaluated intervention that will allow its merits or success to be assessed. Whilst "judgement criterion" is the appropriate word, an acceptable alternative is "reasoned assessment criterion". The criterion is used to answer an evaluation question. One or more judgement criteria are derived from each question.

What is the purpose?

• To avoid subjectivity and to formulate judgements on accepted terms.
• To improve the transparency of the evaluation by making the judgment explicit.
• To structure the answers to the questions asked, since the judgement criteria will determine the indicators and, more generally, the nature of the data collected and the type of analysis.

How can a judgement criterion be clarified on the basis of a question?

All the evaluation questions relate to one or more judgement criteria, unless they are designed only to further knowledge or understanding about the intervention or its effects. The following is an example of a question:

Like most evaluative questions, it has two parts:

• What is being judged: "EC aid".
• The way of judging: Has it "… improved the capacity of the primary educational system to enrol pupils from underprivileged groups without discrimination".
• " This question belongs to the effectiveness family

The judgement criteria develop and specify the second part of the question, for example:

The judgement criteria derive from the question, for instance in the case of the first criterion:

• It concerns the way of judging and not what is judged. This is why the beginning of the question concerning "EC aid" has been removed.
• It specifies the type of success to be evaluated, that is, an improvement in the "capacity of the primary school system to enrol pupils from underprivileged groups without discrimination", and specifically "pupils from ethnic minority X".
• It emphasises the judgement and not the causality analysis. That is why the terms "To what extent … has it improved" have been removed.

To be used in practice, each judgement criterion has to be accompanied by a target level and one or more indicator(s).

Recommendations

• Always define the judgement criterion before selecting an existing indicator or designing a new indicator. This is essential in order to clarify the concepts. By focusing too soon on indicators, one is likely to get trapped into existing information, even if it is inadequate for answering the question asked.
• Have the definition of the judgement criteria discussed by the reference group so that the diversity of points of view relevant to the intervention can be taken into account.
• There may be disagreement on the judgement criteria, for instance the same effect may have a dimension that is judged positively by certain members of the reference group and another dimension judged negatively by others. In this case there are two options: (1) choose only one judgement criterion but be careful to avoid biasing the evaluation; or (2) choose several criteria, although this will increase and complicate the data collection and analysis work.
• To optimise the collection and analysis of data, it is best to define a limited number of judgement criteria for each question. This recommendation also takes into account users' capacity to absorb information.
• Where relevant, explain any gaps between the criteria used to formulate the judgement at the end of the evaluation process and those identified in the first phase (desk) of the evaluation.

Be careful not to confuse concepts

On this site, the word criterion is used for three different concepts:

• Judgement criteria presented on this page
• The evaluation criteria: relevance, effectiveness, efficiency, sustainability, impact, Community value added, and coherence/complementarity.
• The quality assessment criteria of evaluation reports.

According to the EC, the value added of an evaluation is the formulation of value judgements on the basis of evidence and explicit judgement criteria. When dealing with organisations which are not familiar with evaluation, it may be wise not to use the word "judgement", which may induce resistance. An acceptable alternative is "assessment", or preferably "reasoned assessment".

.

TARGET

.

What does this mean?

The concept of a 'target' is widely used in the context of public management for setting a verifiable objective or a level of performance to be achieved. In an evaluation context it is used in a much wider sense since the evaluated intervention may have to be judged against targets that were not set in advance but that are specifically identified, such as a benchmark, a success threshold or a comparable good practice.

What is the purpose?

• To avoid subjectivity and formulate a judgement on accepted and recognised terms.

How can they be determined?

- By reference to an objective defined in a verifiable way

The target may appear in one of the intervention objectives, that is, as long as they have been established in a verifiable way. In this particular case, the same indicator helps to define the objective, to make the judgment criterion operational and to determine the target.

• Example: the number of qualified and experienced teachers per 1,000 children of primary-school age is at least 20.

- In relation to comparable good practices outside the intervention

In this case, the target is established at the outset of the evaluation. It is not related to an objective or a performance framework existing prior to the evaluation.

• Example: the access to primary education with qualified and experienced teachers is at least as satisfactory as in the case of X (recognised good practice at regional level).

The procedure is as follows:

• Identify a comparable practice recognised for its quality (similar EC intervention in another country; intervention by another donor, intervention in another sector though using the same instruments).
• Obtain information on the practice for comparison (this is easier if it has already been evaluated).
• Ensure that the contextual conditions are close enough that they allow for comparison.
• Proceed to carry out the comparison (essentially qualitative).
• Discuss and validate the comparison with the reference group.

- Compared to best practices identified within the intervention

The target can be found within the evaluated intervention itself during the synthesis phase, provided that specific practices can be considered as good as regards the judgement criteria under consideration.

In this case, the good practices will serve as benchmarks to judge the others. Of course, it is advisable to check that the contextual conditions are close enough so as to allow for comparison.

• Example: In areas where ethnic minority X concentrates, the number of qualified and experienced teachers per 1,000 children of primary-school age is close to the best performing areas in the country.

When should they be determined?

- Earlier or later in the evaluation process

If the target is derived from a verifiable objective or a performance framework, then it can be determined at the very first stage of the evaluation process.

If the target is derived from an outside benchmark, then it should be identified during the early stages of the evaluation. However, the process may involve the gathering of secondary data with a view to specifying the benchmark, as well as a careful examination of comparability. This means that the target will not be completely defined in the first phase of the evaluation.

If the target is to be derived from the best practices discovered within the intervention by the evaluation team, it will be determined in the synthesis phase.

- After choosing the judgement criterion

Determining the target takes place in a three-step process:

• Choice and finalisation of the evaluation question.
• Choice of the judgment criterion (or criteria).
• The targets, the indicators and the sources of information are determined together in the third step.

Evaluation targets and others

When the evaluation question pertains to an intended result or impact, the target level is usually derived from a verifiable objective or borrowed from a performance assessment framework.

Performance monitoring may however be of little or no help in the instance of evaluation questions relating to cross-cutting issues, sustainability factors, unintended effects, evolving needs and problems, coherence, etc.

.

INDICATOR

.

What does this mean?

The evaluation team may use any kind of reliable data to assess whether an intervention has been successful or not in relation to a judgement criterion and a target.

Data may be collected in a structured way by using indicators. Indicators specify precisely which data are to be collected. An indicator may be quantitative or qualitative. In the latter case the scoring technique may be used.

Unstructured data are also collected during the evaluation, either incidentally, or because tools such as case studies are used. This kind of evidence may be sound enough to be a basis for conclusions, but it is not an indicator.

What is the purpose?

• To collect and process data in a form that can be used directly when answering questions.
• To avoid collecting an excessive amount of irrelevant data and to focus the process on the questions asked.

Evaluation indicators

The main evaluation indicators are those related to judgement criteria, that specify the data needed to make a judgement based on those criteria.

An indicator can be constructed specifically for an evaluation (ad hoc indicator) and measured during a survey, for example. It may also be drawn from monitoring databases, a performance assessment framework, or statistical sources.

A qualitative indicator (or descriptor) takes the form of a statement that has to be verified during the data collection (e.g. parents' opinion is that their children have the possibility of attending a primary school class with a qualified and experienced teacher).

A quantitative indicator is based on a counting process (e.g. number of qualified and experienced teachers). The basic indicator directly results from the counting process. It may be used for computing more elaborate indicators (ratios, rates) such as cost per pupil or number of qualified and experienced teachers per 1,000 children of primary-school age.

Indicators may belong to different categories: inputs, outputs, results or impacts. 

Evaluation indicators and others

When an evaluation question pertains to an intended result or impact, it is worth checking whether this result or impact has been subject to performance monitoring. In such cases, the evaluation team uses the corresponding indicators and data, which is a considerable help, especially if baseline data have been recorded.

Performance monitoring may, however, be of little or no help in the instance of evaluation questions relating to cross-cutting issues, sustainability factors, unintended effects, evolving needs or problems, coherence, etc.

Quality of an indicator

An indicator measures or qualifies with precision the judgement criterion or variable under observation (construct validity). If necessary, several less precise indicators (proxies) may be used to enhance validity.

It provides straightforward information that is easy to communicate and is understood in the same way by the information supplier and the user.

It is precise, that is, associated with a definition containing no ambiguity.

It is sensitive, that is, it generates data which vary significantly when a change appears in what is being observed.

Performance indicators and targets are often expected to be SMART, i.e. Specific, Measurable, Attainable, Realistic and Timely. The quality of an evaluation indicator is assessed differently. 

Indicators and effects: a warning!

The indicator used to evaluate an effect is not in itself a measurement or evidence of that effect. The indicator only informs on changes, which may either result from the intervention (effect) or from other causes.

The evaluation team always has to analyse or interpret the indicator in order to assess the effect.

Categories of indicators

- Indicators and the intervention cycle

Indicators are used throughout the intervention cycle. They are first used to analyse the context; then, for the choice and validation of the intervention strategy, afterwards for monitoring outputs and results and, finally, for the evaluation.

Indicators and intervention design

Context indicators may be used to support the identification of the needs, problems and challenges which justify the intervention.

As far as possible, objectives and targets are defined in a measurable way by using indicators.

Indicators, monitoring and performance assessment

Monitoring systems and performance assessment frameworks also use indicators which derive from the diagram of expected effects (also called results chain).

Monitoring indicators primarily relate to inputs and outputs. Performance indicators primarily focus on intended results and impacts. The EC's Result Oriented Monitoring (ROM) does not rely that much on indicators. It delivers systematic assessments of external aid projects in the form of ratings with a view to intended results and impacts.

Indicators and evaluation

Evaluation indicators are used to help answering specific evaluation questions. Depending on the question, they may relate to the needs, problems and challenges which have justified the intervention, or to the achievement of intended outputs, results and impacts, or to anything else.

- Global and specific indicators

Global or contextual indicators apply to an entire territory, population or group, without any distinction between those who have been reached by the intervention and those who have not. They are mainly taken from statistical data. This site offers help to look for contextual indicators.

Specific indicators concern only a group or territory that has actually been reached. With specific indicators, changes among those affected by the intervention can be monitored. Most of these indicators are produced through surveys and management databases.

- Indicators and intervention logic

Input indicators

Input indicators provide information on financial, human, material, organisational or regulatory resources mobilised during the implementation of the intervention. Most input indicators are quantified on a regular basis by the management and monitoring systems (providing that they are operational).

Output indicators

Output indicators provide information on the operators' activity, especially on the products and services that they deliver and for which they are responsible. To put it simply, one could say that outputs correspond to what is bought with public money.

Result indicators

Result indicators provide information on the immediate effects of the intervention for its direct addressees. An effect is immediate if the operator notices it easily while he/she is in contact with an addressee. Because they are easily recognised by the operators, direct result indicators can be quantified exhaustively by the monitoring system.

Impact indicators

Impact indicators provide information on the long-term direct and indirect consequences of the intervention.

A first category concerns the consequences that appear or last in the medium or long term for the direct beneficiaries.

A second category of impacts concerns people or actors that are not direct beneficiaries.

Impact indicators cannot be produced in general from management information. They require statistical data or surveys specially conducted during the evaluation process.

Indicators derived from scoring

What does this mean?

Scoring (or rating) produces figures that synthesise a set of qualitative data and or opinions. Scoring is guided by a scoring grid (or scorecard) with varying degrees of detail.

From an evaluation point of view, both words scoring and rating can be used.

What is the point?

Scoring allows the production of structured and comparable data on judgement criteria that do not lend themselves to a measurement using quantitative indicators.

How to construct a scoring grid

• Examine several possible dimensions for the criterion that has to be assessed (sub-criteria).
• For each dimension or sub-criterion, write a short sentence defining the success of the intervention (descriptor of full success).
• For each dimension or sub-criterion, write another sentence (descriptor) defining the failure of the intervention.
• Write one or more sentences (descriptors) that represent intermediate levels of success.
• Associate a score with each descriptor (e.g. from 0 to 10, from 0 to 3, from -3 to +3).
• Weight the sub-criteria if necessary.
• Test the grid on a few pilot examples.
• Discuss the test with the reference group if relevant

How to use the scoring grid

Scoring grids usually apply to projects or components of the intervention and allow for comparing these.

The evaluation team puts together all the data it has on the project or intervention to be assessed. It then chooses the level (or descriptor) in the scoring grid that corresponds best (or the least badly) to this information. The score results from this choice. 

Recommendations

The more detailed the scoring grid the less subjective the score will be and the more comparable the scores allocated by two different evaluators will be.

.

FROM QUESTIONS TO INDICATORS

.

From questions to judgement criteria

The judgement criterion (also called reasoned assessment criterion) specifies an aspect of the evaluated intervention that will allow its merits or worth to be assessed in order to answer the evaluation question, . For instance:

The judgement criterion gives a clear indication of what is positive or negative, for example: "enhancing the expected effects" is preferable to "taking potential effects into account".

A more precise judgement criterion than the question

The question is drafted in a non-technical way with wording that is easily understood by all, even if it lacks precision.

The judgement criterion focuses the question on the most essential points for the judgement.

Yet the judgement criterion does not need to be totally precise. In the first example the term "satisfactory quality" can be specified elsewhere (at the indicator stage).

Not too many criteria

It is often possible to define many judgement criteria for the same question, but this would complicate the data collection and make the answer less clear.

In the example below, the question is treated with three judgement criteria (multicriteria approach):

• "capacity of the primary school system to enrol pupils from ethnic minority X with satisfactory quality"
• "capacity of the primary school system to enrol pupils from the poorest urban areas with satisfactory quality"
• "capacity of the primary school system to enrol girls ".

A judgement criterion corresponding to the question

The judgement criterion should not betray the question. In the following example, two judgement criteria are considered for answering the same question:

• "capacity of the primary school system to enrol pupils from ethnic minority X with satisfactory quality"
• "primary school leavers from ethnic minority X pass their final year exam "

The first judgement criterion is faithful to the question, while the second is less so in so far as it concerns the success in primary education, whereas the question concerns only the access to it. The question may have been badly worded, in which case it may be amended if there is still time.

Also specify the scope of the question

Most questions have a scope (what is judged) and a judgement criterion (the way of judging). In addition to the judgement criterion, it is therefore often desirable to specify the scope of the question, for example: "European aid granted over the past X years", "design of programme X", "the principle of decentralisation adopted to implement action X".

Also specify the type of cause-and-effect analysis

Some questions imply a cause-and-effect analysis prior to the judgement. It may therefore also be useful to specify the type of analysis required by means of terms such as "has European aid led to", "has it contributed to", "is it likely to".

From judgement criteria to indicators

An indicator describes in detail the information required to answer the question according to the judgement criterion chosen, for example:

Not too many indicators

It is possible to define many indicators for the same judgement criterion. Relying upon several indicators allows for cross-checking and strengthens the evidence base on which the question is answered. However, an excessive number of indicators involves a heavy data collection workload without necessarily improving the soundness of the answer to the question.

In the examples below three indicators are applied to a judgement criterion ("capacity of the primary school system to enrol pupils from ethnic minority X with satisfactory quality"):

• "Number of qualified and experienced teachers per 1000 children of primary-school age in areas where ethnic minority X concentrates"
• "Number of pupils per teacher in areas where ethnic minority X concentrates"
• "Level of quality of the premises (scale 1 to 3) assigned to primary education in areas where ethnic minority X concentrates ".

Indicator corresponding to the judgement criterion

The indicator should not betray the judgement criterion. Two indicators are considered below:

• "Number of qualified and experienced teachers per 1000 children of primary-school age in areas where ethnic minority X concentrates."
• "Primary education enrolment rate in areas where ethnic minority X concentrates"

The first indicator corresponds faithfully in so far as it describes an essential aspect of the judgement criterion. The second indicator is less faithful because it fails to reflect the concept of "satisfactory quality". Its construct validity is not good.

Unambiguous indicators

An indicator must be defined without any ambiguity and understood in the same way by all the members of the evaluation team. For instance, in the above examples it is necessary to specify what a "qualified and experienced teacher" is. This can be done with reference to an existing definition, or else a definition can be formulated as precisely as possible until there is no more ambiguity whatsoever.

Indicators independent from the observation field

The same indicator should be able to serve to collect data in several contexts, for example:

• "Number of qualified and experienced teachers per 1000 children of primary-school age" - in areas where ethnic minority X concentrates.
• "Number of qualified and experienced teachers per 1000 children of primary-school age" - in areas where ethnic minority X is absent.

In this case the same indicator is applied in both types of area and serves as a comparison, on the basis of which a judgement is formulated.

Quantitative and qualitative indicators

The following two examples present an alternative between a quantitative indicator and a qualitative indicator for treating the same judgement criterion:

• "Number of qualified and experienced teachers per 1000 children of primary-school age" (quantitative)
• "Surveyed parents confirm that their children have the possibility of attending a primary-school class and benefit from a qualified and experienced teacher" (qualitative);

An indicator is preferably associated with a target

The target indicates which comparison should be made in order to answer the question, for example: "In the areas where ethnic minority X concentrates, the indicator is at least as good as in the entire country in average".

The target and the indicator are often specified interactively in successive steps. It is important not to digress from the judgement criterion during this process.

When the evaluation question pertains to an intended result or impact, the target is usually derived from a verifiable objective or borrowed from a performance assessment framework.

Feasible indicators

The indicator makes it possible to focus and structure data collection but serves no purpose as long as data does not exist. To ensure the feasibility of an indicator, it is necessary to indicate the source of the information to use, for example:

• management data from the national education system,
• periodical national surveys on the education system,
• questionnaire survey in several specially selected areas and in the framework of the evaluation.

If no source is available or feasible, the indicator should be changed. If no feasible indicator can be found, excluding the question should be envisaged.

Example of a country evaluation

Question

• The question relates to a family of evaluation criteria: coherence/complementarity

Judgement criterion

The judgement criterion is derived from the question in the following way:

• It shows the type of success that the question is supposed to evaluate, that is, "coherence between the aid and other EC policies".
• It specifies several concepts, primarily that of coherence but also the term "other policies".
• It concerns the way of judging and not what is judged. That is why the beginning of the question concerning "To what extent does the EC ensure" has been removed.

Indicator

The indicator is derived from the judgement criterion in the following way:

• It corresponds faithfully to the judgement criterion.
• It describes in detail the data to be collected in order to apply the judgement criterion chosen. However, terms such as "positive/negative synergy" still have to be defined further. The definition must be drafted as precisely as possible, until there is no more ambiguity whatsoever.
• It is qualitative, although a quantitative indicator could have been defined, for example: "proportion of positive synergies among identified synergies".
• It makes it possible to define a target, for example by comparing the respective importance of positive and negative synergies.
• Its feasibility still has to be verified by ensuring that one or more sources of information and evaluation tools will be available, for example: opinion of a panel of independent experts, questionnaire administered at the end of a focus group composed of administrators.

Example of a sector evaluation

Question

The question refers to a family of evaluation criteria: effectiveness.

Judgement criterion

The judgement criterion is derived from the question in the following way:

• It shows the type of success that the question is supposed to evaluate, that is, the best "capacity of the education system to enrol pupils from disadvantaged groups without discrimination".
• It clarifies several concepts such as "educational system", "disadvantaged groups" and "discrimination". The term "satisfactory quality" has yet to be specified.
• It concerns the way of judging and not what has been judged. That is why the beginning of the question concerning "EC support" has been removed.
• It focuses on the judgement and not on the causal analysis. That is why the term "has … enhanced" has been removed.

Indicator

The indicator derives from the judgement criterion in the following way:

• It describes in detail the information required to judge according to the judgement criterion chosen. But this is not enough. In particular, it is necessary to specify what a "qualified and experienced teacher" is. This can be done by referring to an existing definition or else by formulating a definition as precisely as possible until there is no more ambiguity whatsoever.
• The indicator corresponds faithfully to the judgement criterion ("capacity of the primary school system to enrol pupils with satisfactory quality"). It does not encompass all the dimensions of the judgement criterion but highlights what is considered as essential.
• It is quantitative, but a qualitative indicator could also have been defined, for example: "surveyed parents confirm that their children have the possibility of attending a primary school class and benefit from a qualified and experienced teacher".
• It will make it possible to define a target, for example by comparing "areas where ethnic minority X concentrates" with "the entire country".
• Its feasibility still has to be verified by ensuring that one or more sources of information are available, for example: management data of the national education system, periodical national surveys on the educational system, questionnaire survey run as part of the evaluation.

WHAT DOES IT MEAN? 

.

What is the purpose?

To set up the method that will allow the external evaluation team to answer the questions and to come to an overall assessment. In addition to selected questions, judgement criteria, indicators and targets, the evaluation method includes:

• a strategy for collecting and analysing data
• selected investigation areas
• a series of specifically designed tools
• a work plan.

When does it take place?

The evaluation team starts designing a provisional method as early as from the drafting of its proposal in order to draw up cost estimates. A key assumption at this stage is the extent to which the evaluation will rely on secondary data or will involve specific data collection work in the field.

The main frame of the method is then established during the inception stage, in line with the evaluation questions, judgement criteria, indicators, data collection tools and analysis strategy.

The method is refined and finalised before the field phase and fully described in the first phase report (desk).
The final report includes a short and sharp presentation of the evaluation method, together with its limitations, if there are any. The method is fully described in annex, including initial design, problems encountered, solutions found, method actually implemented, and limitations.
The evaluation method is designed through an iterative process at three levels:

• A question-by-question approach, allowing the evaluation team to prepare the design tables with an aim to adequately answer each question.
• An overall approach which cuts across the questions, and which allows the evaluation team to optimise the evaluation method as a whole, whilst matching time and resource constraints.
• A series of specifically developed tools.

.

DESIGN TABLE PER QUESTION

.

What does this mean?

The design table explains how an evaluation question will be answered, including the chain of reasoning which connects data, findings and conclusions.
An example of a design table is provided on this site.

When is it constructed?

A design table is developed for each question and progressively refined in successive versions:

• Preliminary version appended to the inception report.
• Draft version(s) prepared during the first phase of the evaluation as the methodological design is progressively optimised.
• Final version attached to the first phase report (desk).

Opening the table

The first lines of the table summarise the steps which have already been taken (see from question to indicator), i.e.

• Text of the question
• Comment
• Scope of the question
• Judgement criterion (or criteria)
• Indicators
• Targets

Sub-questions

What does this mean?

Together with the judgement criteria, indicators and targets, the sub-questions are meant to explain how the evaluation team will answer the question, for instance:

• How have indicators changed over the evaluated period?
• To what extent can the change be qualified as a success?
• How far have EC activities contributed to explaining the observed change? Through which mechanisms? Is there evidence that such mechanisms have been working as assumed?
• How far does evidence support alternative explanations? Is there evidence that the observed change is owing to other development partners, other EC policies, the Government or other external factors?

Who does what, and when?

A set of evaluation questions is drawn up at the inception stage, together with sub-questions.
The responsibilities are as follows:

• The reference group validates the questions.
• The external evaluation team breaks down each question into sub-questions in a design table.

What is the purpose?

The sub-questions describe the chain of reasoning through which the evaluation team plans to answer the question. The intended reasoning is indicative but it is worth clarifying it in advance because:

• The reference group members get an opportunity to provide specific advice and inputs
• All evaluation team members understand why they are collecting and analysing data, and therefore work more effectively
• The team members who are not familiar with evaluation receive useful guidance on which data are to be collected and analysed.

It may furthermore be worth suggesting provisional sub-questions together with the draft set of key questions. Members of the reference group will then realise that many of their expectations will be satisfied through answering the sub-questions. It will help them to accept the principle of a limited list of well focused evaluation questions.

Sub-questions pertaining to indicators

These sub-questions may pertain to:
(1) the current level / status of the indicators, possibly with a break-down per country, area, social group, etc., for instance:

• What is the current value of quantitative indicator X at country level, and for targeted areas/groups?

(2) changes in the indicators, for instance:

• Do stakeholders perceive a positive change in qualitative indicator Y over the evaluated time period?

As seen in the examples above, the sub-questions may be quantitative or qualitative.

Question:

• To what extent has EC support improved the capacity of the educational system to enrol pupils from disadvantaged groups without discrimination?

Sub-question:

• What is the change in the number of experienced and qualified teachers per 1000 primary school age children respectively at country level, in poor urban areas and in areas where ethnic minority X concentrates?

In this example, the sub-question is simply meant to show how the indicator (number of experienced and qualified teachers) will be applied.

Sub-questions pertaining to analysis

These sub-questions are written with a view to:
(3) Confirming assumptions about the success of the intervention and substantiating a positive answer to the question, for instance:

• Has the design of EC support included a commitment to monitor performance indicators related to effect X?
• Was such monitoring actually undertaken?
• Was the monitoring data subject to periodic discussion among development partners?
• Have partners taken action as a follow-up to such discussions?
• Were such actions designed with a view to achieving effect X?
• Etc.

(4) Challenging assumptions about the success of the intervention and substantiating a negative answer to the question, for instance:

• Have other development partners pushed for monitoring performance indicators related to effect X?
• Have non-state actors contributed to putting the issue of achieving effect X onto the political agenda?
• How far did other partners contribute towards shaping the actions taken in favour of disadvantaged groups?

Question:

• To what extent has EC support improved the capacity of local authorities to design adequate rural development strategies?

Sub-question:

• To what extent has the EC been able to put decentralisation at the top of the agenda in its policy dialogue with the Government?

In this example the sub-question relates to a specific short-term result (promotion of decentralisation through policy dialogue) which is a driver to the wider impact highlighted in the question (capacity building).

Sub-questions pertaining to judgement

These sub-questions are meant to assist in the formulation of conclusions involving explicit value judgements. They are written with a view to:
(5) applying and possibly refining the judgement criteria in the specific context of the intervention, for instance:

• Do stakeholders spontaneously focus on the same judgement criteria as those selected for the evaluation? If not, why not?

(6) applying or developing the targets in the specific context of the intervention, for instance:

• Which are the areas / groups in the country with the best performance as regards the selected judgement criterion? Among them, which ones can legitimately be compared with targeted areas / groups?

Question:

• To what extent has EC support been efficient in strengthening the road network?

Sub-question:

• To what extent has the EC strengthened the road network? In this example, the sub-question relates to a prerequisite (the road network is strengthened) before applying the judgement criterion (efficiency in strengthening the road network).

Analysis strategy

Four strategies can be considered:

• Change analysis, which compares measured / qualified indicators over time, and against targets
• Meta-analysis, which extrapolates upon findings of other evaluations and studies, after having carefully checked their validity and transferability
• Attribution analysis, which compares the observed changes and a "without intervention" scenario, also called counterfactual
• Contribution analysis, which confirms or disconfirms cause-and-effect assumptions on the basis of a chain of reasoning.

The three last strategies are appropriate for the questions which require a cause-and-effect analysis. The first one is appropriate in other instances.

Investigation areas

The evaluation team may consider collecting and analysing data at the level of the intervention as a whole, or investigating some areas more specifically, for instance:

• All sub-questions will be addressed through an investigation into a selection of countries and will include country notes and a visit to each country.
• In addition to using national statistics, the evaluation team investigates a selection of districts respectively typical of (a) the targeted group /area, and (b) best performing groups / areas.
• The evaluation team will carefully select ten projects which will be subject to an in-depth investigation in order to address some of the sub-questions.

Example

click this image to expand the preview

.

OPTIMISING THE OVERALL DESIGN

.

What is the purpose?

To finalise the overall evaluation method in a way which cuts across the evaluation questions and which makes a good enough mix of evaluation tools, considering the available time and resources.

When does it take place?

The evaluation method is designed through an iterative process at three levels:

• A question-by-question approach, allowing the evaluation team to prepare the design tables with the aim to adequately answer each question.
• An overall approach which cuts across the questions, and which allows the evaluation team to optimise the evaluation method as a whole, whilst matching time and resource constraints.
• A series of specifically developed tools.

Several iterations may be needed in order to allow the evaluation team to optimise the overall design whilst ensuring a high quality to each question.

Selecting tools

In parallel with the design tables, which are established on a question-by-question basis, the evaluation team designs its overall evaluation method which covers all questions and includes a number of tools such as:

• A documentary analysis applying to a given set of reports
• An analysis of context indicators
• A series of interviews with stakeholders belonging to a given category
• A questionnaire
• A series of case studies
• A series of focus groups or participatory meetings
• An expert panel
• The building and analysis of an ad hoc data base

These examples are all but limitative. The evaluation team may even have to develop a completely new tool, where relevant.

Combining tools and questions

The evaluation team draws up the list of all evaluation tools suggested in the design tables. Each tool is then considered from the viewpoint of its capacity to help answering several questions and sub-questions, for instance:

• A series of interviews with Government officials: this tool may help answering questions related to "alignment of EC strategy on Government priorities", "policy dialogue in major policy domains", and "co-ordination among development partners".
• A focus group gathering participants interested in primary education: this tool may help answering questions related to "efficiency of selected funding modalities", "mainstreaming of gender equality", and "improved access to basic education for the poor".

An image of this process is given by the matrix below in which the first tool helps answering sub-questions Aa, Cf and Eb, etc.

This image suggests that the set of tools will provide the evaluation team with possibilities of cross-checking data on sub-questions Aa and Eb. Tools and questions are combined in order to optimise such possibilities of cross-checking.
The above diagram is for explanatory purpose only. In the real world, the use of a matrix may be unpractical if the number of tools and/or sub-questions exceeds a dozen, which is typically the case in a country strategy evaluation. A pragmatic approach consists of drawing up a list of all questions and sub-questions addressed when developing each tool. An example is provided on this site.

Verifying the adequacy of tools

The evaluation team confirms that each tool envisaged is adequate in the sense that:

• It fulfils the required function, e.g. a focus group is adequate for collecting beneficiaries' perceptions; a case study is adequate for understanding cause-and-effect mechanisms; a series of interviews is adequate for collecting the views of officials in partner institutions, and so on.
• It is in line with the selected analysis strategy, e.g. a series of case studies is adequate in the instance of a contribution analysis; a questionnaire to beneficiaries and non-beneficiaries is adequate in the instance of an attribution analysis; a panel of experts is adequate in the instance of a meta-analysis, and so on.
• It fits into the identified constraints, e.g. a questionnaire survey is appropriate if, and only if a satisfactory sample can be drawn; a database can be analysed if, and only if missing data are in limited numbers.

The evaluation team explains and justifies its main technical choices, with alternative options, pros and cons, and associated risks.

Articulating tools

The evaluation team assesses whether the selected tools may reinforce one another, for instance:

• A series of interviews may help in identifying relevant papers which will be submitted to a documentary analysis.
• A series of interviews may help in selecting the stakeholders to be invited to a focus group, and the issues to be discussed.
• A series of interviews may help in refining the questions to be put to a sample of beneficiaries.
• A series of case study monographs may be reviewed by a panel of experts with a view to strengthening the analysis and to deriving more accurate findings.

Preparing the overall assessment

In the process of progressively optimising its design, the evaluation team examines all the design tables in a cross-cutting manner with a view to preparing its final synthesis, i.e. an overall assessment that draws upon the answers to all evaluation questions.
More than being a mere compilation of findings and conclusions, the overall assessment may involve specific analyses which deserve to be designed in advance, for instance:

• Specific themes like policy dialogue with the partner country's Government, decentralisation of EC procedures, etc. This calls for adequate sub-questions to be inserted in several design tables.
• If the selected questions, sub-questions and tools leave a significant part of the support unexplored, then a few additional studies may be planned in order to provide a broader picture of the overall support (e.g. meta-analysis of monitoring reports). An overall analysis of management databases may also be undertaken with the same purpose.

Allocating resources

In the process of designing its method, the evaluation team tries to adequately share its limited resources between questions and sub-questions. Some questions deserve to be addressed with costly tools such as questionnaire surveys of end users, several case studies, focus groups, etc. Other questions should rather be answered on the basis of a documentary analysis only and a few interviews with EC and Government officials.
One reason why a question should be allocated more resources is the need for credibility. If a question is considered to be politically sensitive and if the answer is likely to trigger debate, then it deserves a stronger design, possibly with more cross-checking of sources, larger samples, several focus groups or case studies, etc.
It is also wise to invest substantial resources in a question that raises feasibility problems. The following are examples of specific difficulties calling for a particularly strong design:

• A question relates to an innovative aspect of the intervention in an area where expertise is scarce.
• A question relates to a component of the intervention which has just reached the end users in the field, and the implementing actors have no idea on how the targeted people are reacting.
• A question relates to a far-reaching impact that is logically distant from EC activities and the abundance of external factors makes it difficult to assess cause-and-effect relations.

A question is rarely unevaluable in the absolute. It is more likely to be an accumulation of difficulties and constraints that leads to feasibility problems. At the earliest stages of the evaluation, it is often possible to amend a difficult question so as to make it more evaluable. This can be done, for example, by limiting the scope of the question or choosing to apply it to a less distant effect or to a probable effect if the real effect is not yet observable. Once a difficult question has been validated, the evaluation team has to design an appropriate method, and to allocate adequate resources.

Cost and time constraints

Successive versions of the method are designed within the evaluation team until the following constraints are matched:

• The implementation of the evaluation tools fit into the overall time schedule of the evaluation process
• The cost of the evaluation tools (human resources, technical costs, travel and daily subsistence) fits into the overall budget of the evaluation
• The availability of qualified workforce in the field is sufficient for implementing evaluation tools professionally

.

DEVELOPING A TOOL

.

No evaluation method without tools

Evaluation tools are needed in order to collect primary and secondary data, to analyse data, and to formulate value judgements (or reasoned assessments). The external evaluation team may carry out some of these tasks without evaluation tools, but several tools are always needed for answering each evaluation question.
Tools range from simple and usual ones like database extracts, documentary analysis, interviews or field visits, to more technical ones like focus groups, modelling, or cost benefit analysis. This site describes a series of tools that are frequently used.
By using an appropriate mix of tools, the evaluation team strengthens the evidence basis of the evaluation, the reliability of data, the validity of its reasoning, and the credibility of its conclusions.

How are tools chosen?

Tools are chosen during the iterative process of designing the overall evaluation method, with an aim to:

• Contribute to answering all questions and sub-questions, and to formulating an overall assessment
• Gather / collect data, assist in analyses and formulation of value judgement (or reasoned assessment)
• Facilitate cross-checking and triangulation
• Reinforce each other through appropriate combinations
• Match contextual constraints like: availability of expertise and data, allocated resources, time schedule.

When is it developed?

All evaluation tools are developed progressively and finalised before the beginning of the field phase, although some tools need to be implemented, and therefore developed earlier in the process, e.g. interviews with key stakeholders at the inception stage, analysis of management databases at the desk phase, etc.

Developing a tool

Whilst the set of evaluation tools is to be selected as a part of the overall evaluation design, each tool is to be developed in a separate way. An example of developed tool is provided on this site.
Development a tool may be a matter of a section in the inception or desk report. However the task may needs to be further formalised, including in the form of fully fledged terms of reference, when several members of the evaluation team work separately, for instance if the works extend to different countries, or if the tool is being sub-contracted.
Tool development proceeds through seven steps as follows:

Questions and sub-questions

The evaluation team lists the questions and sub-questions that have to be addressed by the tool. It refers to the design tables.

Technical specifications

The evaluation team develops the technical specifications of the tool through a preparatory stage. Technical specifications depend on the type of tool. They cover issues like:

• Sampling, selection of interviews, of case studies, of documents …
• Questions, themes of a documentary analysis, content of a monograph, fields of a database …
• Mode of implementation
• Duration of interview, focus group, visit, …

Caution! - When developing a questionnaire or an interview guide, the evaluation team should not proceed by copying and pasting evaluation sub-questions. If evaluation questions and sub-questions are naïvely passed to informants, there are considerable risks of biases.
Technical specifications need to be further formalised when several members of the evaluation team work separately, especially if the works extend to different countries.

Risk management

The evaluation team assesses the main risks with data collection and analysis, as well as potential biases. As far as relevant, it prepares second best solutions in case the tool cannot be applied satisfactorily. The following lines provide examples of risks associated with evaluation tools, and examples of second best solutions:

• Considering database X, if more than 20% data are still missing after 4 weeks, then analyse the available data and ask expert Y to comment upon the validity of findings.
• Considering the series of interviews X, if more than 30% informants cannot be reached after 3 weeks, then collect alternative "second hand" information from expert Y.
• Considering the questionnaire X, if the number of respondents falls below 200, then gather a focus group meeting and cross-check results with that of the questionnaire survey.

This list is not limitative.

Mode of reporting

The outputs vary from one tool to another. They may take the form of tables, lists of quotations, lists of verbatims, monographs, etc. The evaluation team decides on how the outputs will be reported, for instance:

• Full length data delivered to the evaluation team leader
• Full length data included in a CDROM attached to the final report (in which case, some documents may need to be anonymised)
• Tables or boxes to be inserted in the final report

The evaluation team also decides on how to report about the implementation of the tool and the associated limitations if any, e.g.

• Note to be inserted in the methodological annex appended to the final report
• Short methodological comment to be inserted in the final report itself.

Responsibilities

The tasks and roles are shared among the evaluation team members, e.g.

• Who is to implement the tool?
• Who will ensure harmonisation in case several team members implement the tool in various countries / areas?
• Who is to decide upon second best alternatives in case of difficulties?
• Who is responsible for delivering data?
• Who will write the methodological comments?
• Who is to assure quality?

Quality

Quality criteria are precisely defined. Depending on the tool they may cover issues like:

• Coverage of themes, questions, issues
• Accuracy
• Identification of potential biases
• Respect of anonymity rules or other ethical rules
• Formal quality, language, spelling, layout

This site proposes a series of quality check-lists for frequently used tools.

Time schedule and resources

In some instances, it may be necessary to specify practicalities like:

• date of start / end
• human resources allocated
• travel arrangements
• technical expenditures

Example

.

FEASABILITY (EVALUABILITY) OF A QUESTION

.

What is this?

Certain questions are easy to answer while others may raise evaluability problems. It is therefore necessary to assess the feasibility of evaluation questions from the outset.

What is the purpose?

• To ensure that the evaluation provides reliable and credible answers to the questions asked.
• To exclude or amend questions that are too difficult to answer.
• To adjust the available time and other resources in case difficult questions have to be answered.

What has to be taken into account?

To establish whether a question is evaluable, we check:

• Whether the concepts are stable (Are the main terms of the question understood by everyone in the same way?)
• Whether explanatory assumptions can be identified (What are the external factors and the cause-and-effect assumptions?)
• Whether access to the field and data collection entail major difficulties.

What are the most frequent limitations?

A highly innovative activity

If the question concerns an innovative instrument, activity or procedure, the following difficulties may arise:

• It is difficult to define the terms of the question without ambiguity.
• There is a lack of expertise to understand the cause-and-effect mechanisms.

A very recent activity

If the question concerns a recently implemented activity:

• The activity has not yet produced observable effects
• The informants have not yet stabilised their opinions.

Managerial weaknesses

If the question concerns activities in which there are or were managerial weaknesses, the following difficulties may be encountered:

• The monitoring data and reports are inadequate or unreliable.
• The managerial difficulties have generated conflicts that limit access to certain informants or cause those informants to bias their answers.

In case of a strong suspicion of illicit or illegal practices, it is preferable to postpone the evaluation question for later and to start with an audit.

A scope that is too complex

If the question concerns a multi-dimensional or multi-sectoral scope, the following difficulties may be encountered:

• In view of the available time and budget, there are too many data to collect, informants to meet and analyses to perform, and they are too dispersed.

A far-reaching impact

If the question concerns a far-reaching impact which is connected to the evaluated activity through a long chain of causes and effects, then the following difficulties may be encountered:

• There are so many external factors and they are so influential that it becomes impossible to analyse the contribution of the intervention.

An intervention that is too marginal

If the question concerns a very small activity compared to other neighbouring policies or to the context, the following difficulties may be encountered:

• " The evaluated activity does not attain the critical mass that would allow an analysis of its contribution.

Recommendations

A question is rarely unevaluable in the absolute. It is more likely to be an accumulation of difficulties and constraints that makes the question difficult.
When a question is considered too difficult, it is often possible to amend it to make it more evaluable. This can be done, for example, by limiting the scope of the question or choosing to apply it to a less distant effect or to a probable effect if the real effect is not yet observable. This option may be preferable to exclusion of the question.
Policy-makers or any other actor may insist on asking a question that is clearly too difficult. In such an instance, it is useful to provide a full explanation of the difficulties and to show that evaluations have limitations and cannot answer all questions.

For the purpose of answering questions, the evaluation team collects data that are already available (secondary data) and applies data collection tools with a view to obtaining new information (primary data).

.

USING AVAILABLE DATA

.

What are they?

To avoid duplicating efforts, running up unnecessary costs and tiring the informants, it is recommended wherever possible to rely on existing information (secondary data), such as administrative data, research, previous evaluations, management and monitoring databases, statistics.

This information can be obtained at a lower cost. It can help to provide partial answers to some of the questions asked.

Why is it important?

• Existing (secondary) data are cheaper and quicker to gather than primary data.
• If the evaluation team does not acknowledge relevant existing data, this will undermine the credibility of the whole evaluation.

Main channels for identifying and gathering secondary data

• Managers, implementing agencies, operators and partners
• Experts in the domain under consideration
• The Internet
• Statistical offices and monitoring bodies
• Scientific and professional literature

Reliability problems

Before making use of secondary data, particular attention should be paid to the following points:

• Definitions: do the data measure/reflect what is to be addressed?
• Missing data
• Accuracy: are the sources and measurement method reliable?
• Age: when did measurement actually take place?
• Reliability over time: did definitions or measurement methods change in the period covered?
• Comparability: are definitions and measurement methods consistent from one place to another?
• Aggregating data: are pooled data consistent with one another?
• Disaggregating: is it possible to break down data into sub-territories or sub-sectors while keeping a large enough statistical basis?

Recommendations

• Where possible, multiple sources of evidence should be used so as to follow cross-checking.
• Be aware of vested interests when using secondary data. Those responsible for their compilation may have reasons for showing an optimistic or pessimistic picture. For instance, it has been reported that officials responsible for estimating food shortages exaggerated the figures before sending aid requests to potential donors.

.

MOST FREQUENT DIFFICULTIES WITH DATA COLLECTION

.

What are they? How to cope with them?

Even if the data collection programme has been wisely prepared, the evaluation team often encounters problems during its field work. The most frequent difficulties occur with:

- Access to informants

- Cultural gap

- Lack or weakness of data

Recommendations

• Any evaluation creates a feeling of uncertainty, which makes some stakeholders reluctant to co-operate, if not hostile. In such cases keep a positive attitude, emphasise the intended use of the evaluation, promise impartiality, and focus on facts rather than opinions.
• If an information source is not accessible or if a survey technique is not manageable, change the data collection work plan in order to collect similar information through other sources.
• Pay sustained attention to biases and risks of unreliability. Strive to understand them. Report on them.
• Avoid relying on one single information source in order to facilitate cross-checking at the analysis stage. This will also make it easier to manage if one of the sources cannot be used.

.

RELIABILITY OF COLLECTED DATA

.

What are the risks?

While gathering information, the evaluation team faces various risks of biases which may undermine the reliability of collected data.

Why should biases be considered carefully?

• For improving the reliability of data collection
• For assessing the quality of the evaluation
• For understanding the limitations of conclusions which draw on unreliable data

Most frequent biases

- Confirmation bias

This risk is a threat to all data collection approaches. It results from a tendency to seek out evidence that is consistent with the intervention logic, rather than evidence that could disprove it.

When subject to this bias, the evaluation team and informants tend to focus on intended effects and systematically to overlook external factors, unintended effects, negative effects, interactions with other policies, outside stakeholders, alternative implementation options, etc.

This bias is avoided by relying on independent and professional evaluators.

- Self-censorship

In some instances, informants may be reluctant to freely answer questions, simply because they feel at risk. They tend to rigidly express the views of their institution or their hierarchy.

This bias is combated by guaranteeing confidentiality and anonymity in the treatment of answers. The interviewer should also insist on factual questions and avoid collecting opinions.

- Informants' strategy

Those who have stakes in the intervention may distort the information they provide, with the aim of obtaining evaluation conclusions closer to their views.

This bias will be reduced if the whole range of stakeholders is included in the data collection work plan and if various sources of information are cross-checked.

- Unrepresentative sample

This bias may be a matter of concern if the evaluation team generates quantitative data through a questionnaire survey. It should also be considered when using secondary data obtained from a questionnaire survey.

In this instance, the evaluation team should verify that the sample of surveyed informants is large enough and representative of the population as a whole.

- Question induced answers

This bias and the following ones are frequent in interviews and questionnaires. The way in which questions are asked by interviewers or the interviewer's reaction to answers can generate a bias which is either positive or negative. Even the order of the questions in a questionnaire may change the substance of the answers. This bias will be limited by having questionnaires designed and tested by experienced professionals.

- Empathy bias

Interviewees may not have a pre-determined opinion about the questions put to them. They try to make up their mind in a few seconds when responding to the interviewer or to the questionnaire. While doing so, they may be strongly influenced by the context.

Especially in the case of interviews, the evaluation team has to create a friendly (empathetic) atmosphere, at least for the sake of achieving a high rate of answers and fast completion of the survey.

The combination of the two introduces a systematic positive bias in the answers, which tends to overestimate the benefits of the intervention and to underestimate the role of external factors.

This bias is prevented by relying on properly trained interviewers.

- Sample selection bias

People who agree to be interviewed may not be representative of the overall target audience.
This bias could be controlled by undertaking a special qualitative survey on a few "non-respondents", although this exercise brings additional costs.

Recommendations

• Rely on an independent and professional evaluation team in order to limit confirmation biases.
• Systematically mix positive and negative questions in order to reduce empathy bias and question bias.
• Be highly credible when promising confidentiality and anonymity in order to limit respondents' self-censorship - and keep such promises strictly.
• Never rely on a single category of stakeholder (e.g. programme managers, beneficiaries) in order to reduce strategic bias.

KEY ELEMENTS

.

An analysis is required to convert data into findings, which themselves call for a judgement in order to be converted into conclusions. The analysis is carried out on a question-by-question basis, in the framework of an overall design cutting across all questions.

Data, evidence and findings

Any piece of qualitative or quantitative information that has been collected by the evaluation team is called data, for instance:

• Document X indicates that the number of pupils has grown faster than the number of teachers in poor rural areas (data)

A piece of information is qualified as evidence as soon as the evaluation team assesses it as reliable enough, for instance:

• Document X, quoting Education Ministry data that are considered reliable, indicates that the number of pupils has grown faster than the number of teachers in poor rural areas (evidence)

Findings establish a fact derived from evidence through an analysis, for instance:

• The quality of primary education has decreased in poor rural areas (finding)

Some findings are specific in that they include cause-and-effect statements, for instance:

• The EC has not significantly contributed to preventing the quality of primary education from decreasing in poor rural areas (cause-and-effect finding)

Findings do not include value judgements, which are embedded in conclusions only, as shown below:

• The EC has successfully contributed to boosting the capacity of the educational system to enrol pupils from disadvantaged groups, although this has been at the expense of quality (conclusion).

Strategy of analysis

Four strategies can be considered:

• Change analysis, which compares measured / qualified indicators over time, and/or against targets
• Meta-analysis, which extrapolates upon findings of other evaluations and studies, after having carefully checked their validity and transferability
• Attribution analysis, which compares the observed changes with a "without intervention" scenario, also called counterfactual
• Contribution analysis, which confirms or disconfirms cause-and-effect assumptions on the basis of a chain of reasoning.

The first strategy is the lightest one and may fit virtually all types of questions, for instance:

• To what extent are the EC priorities still in line with the identified challenges?
• To what extent has the support taken into account potential interactions and conflicts with other EC policies?
• To what extent has the EC mainstreamed a given cross-cutting issue in the implementation of its interventions?

The three last strategies are better at answering cause-and-effect questions, for instance:

• To what extent has the EC contributed to achieving effect X?
• To what extent has the EC contributed to achieving effect X sustainably?
• To what extent has the EC contributed to achieving effect X at a reasonable cost?

The choice of the analysis strategy is part of the methodological design. It depends on the extent to which the question raises feasibility problems. It is made explicit in the design table.
Once the strategy has been selected and the data collected, the analysis proceeds through all or part of the following four stages: data processing, exploration, explanation, confirmation.

Data processing

The first stage of analysis consists in processing information with a view to measuring or qualifying an indicator, or to answering a sub-question. Data are processed through operations such as cross-checking, comparison, clustering, listing, etc.

• Cross-checking is the use of several sources or types of data for establishing a fact. Systematic cross-checking of at least two sources should be the rule, although triangulation (three sources) is preferable. The cross-checking should involve independent documents. A document that quotes another document is not an independent source. An interviewee who has the same profile as another interviewee is not an independent source.
• Comparison proceeds by building tables, graphs, maps and/or rankings. Data can be compared in one or several dimensions such as time, territories, administrative categories, socio-economic categories, beneficiaries and non-beneficiaries, etc. The evaluation team typically measures change by comparing quantitative indicators over time. Comparisons may also be qualitative, e.g. ranking a population's needs as they are perceived by interviewees.
• Clustering proceeds by pooling data in accordance with predefined typologies, e.g. EC support per sector, beneficiaries per level of income.
• Listing proceeds by identifying the various dimensions of something, for instance the various needs of the targeted group as expressed in a participatory meeting, the various effects of a project as perceived by field level stakeholders, the various strengths and weaknesses of the EC as perceived through interviews with other donors' staff.

Provisional findings emerge at this stage of the analysis. Further stages aim to deepen and to strengthen the findings.

Exploration

The exploratory analysis aims to improve the understanding of all or part of the evaluated area, especially when knowledge is insufficient and expertise is weak, or when surprising evidence does not fit available explanations.

The exploratory analysis delves deeper and more systematically into the collected data in order to discover new plausible explanations such as:

• New categories / typologies
• Unforeseen explanatory factors
• Factors favouring / constraining sustainability
• Unintended effects
• New cause-and-effect assumptions.

The exploratory stage may not be needed for all questions. When such an analysis is carried out, brainstorming techniques are appropriate. The idea is to develop new plausible explanations, not to assert them.

Explanation

This next stage ensures that a sufficient understanding has been reached in terms of:

• Precisely defined concepts, categories and typologies
• Plausible cause-and-effect explanations
• Identification of key external factors and alternative explanations.

Depending on the context and the question, the explanation builds upon one or several of the following bases:

• Diagram of expected effects
• Expertise of the evaluation team
• Exploratory analysis

A satisfactory explanation (also called explanatory model) is needed for finalising the analysis.

Confirmation

The last stage of the analysis is devoted to confirming the provisional findings through a valid and credible chain of arguments. This is the role of the confirmatory analysis.

To have a finding confirmed, the evaluation undertakes a systematic self-criticism by all possible means, e.g. statistical tests, search for biases in data and analyses, check for contradictions across sources and analyses.
External criticism from experts or stakeholders is also considered.

.

ANALYSIS STRATEGY

.

Cause-and-effect analysis

What does this mean?

Approach through which the evaluation team asserts the existence of a cause-and-effect link, and/or assesses the magnitude of an effect.

Attribution or contribution

- Attribution analysis

Attribution analysis aims to assess the proportion of observed change which can really be attributed to the evaluated intervention. It involves building a counterfactual scenario.

- Contribution analysis

Contribution analysis aims to demonstrate whether or not the evaluated intervention is one of the causes of observed change. It may also rank the evaluated intervention among the various causes explaining the observed change. Contribution analysis relies upon chains of logical arguments that are verified through a careful confirmatory analysis.
It comprises the following successive steps:

• Refining the cause-and-effect chains which connect design and implementation on the one hand, and the evaluated effect on the other. This step builds upon available explanations pertaining to the evaluated area. Explanations derive from the diagram of expected effects drawn in the first phase of the evaluation, from the evaluation team's expertise, and from exploratory analyses.
• Gathering evidence related to each link in the cause-and-effect chain, including findings of similar studies, causal statements by interviewees, and evidence from in-depth inquiries.
• Gathering evidence related to other explanations (other interventions, external factors).
• Developing a step-by-step chain of arguments asserting that the intervention has (or has not) made a contribution, and possibly ranking the intervention among other contributions.
• Submitting the reasoning to systematic criticism until it is strong enough.

Analytical approaches

- Counterfactual

The approach is summarised in the diagram below:

The "policy-on" line shows the observed change, measured with an impact indicator, between the beginning of the evaluated period (baseline) and the date of the evaluation. For instance: local employment has increased, as has literacy. The impact accounts for only the share of this change that is attributable to the intervention.
The "policy-off" line, also called the counterfactual, is an estimate of what would have happened without the intervention. It can be obtained with appropriate approaches like comparison groups or modelling techniques. Impact is assessed by subtracting the policy-off estimate from the observed policy-on indicator.
The assessed impact, derived from an estimate of the counterfactual, is itself an estimate. In other words, impacts cannot be directly measured. They can simply be derived from an analysis of impact indicators.
Only a counterfactual allows for a quantitative impact estimate. When successful, this approach therefore has a high potential for learning and feedback. It is nevertheless relatively demanding in terms of data and human resources, which makes it somewhat unusual in evaluation practice in developing countries.

- Case studies

Another analytical approach relies on case studies. It builds upon an in-depth inquiry into one or several real life cases selected in order to learn about the intervention as a whole. Each case study monograph describes observed changes in full detail. A good case study also describes the context in detail and all significant factors which may explain why the changes occurred or did not occur.
In a case study approach, the evaluation team analyses the whole set of collected facts and statements and checks whether they support assertions like "the change can be attributed to the intervention", "the change can be attributed to another cause", "the absence of change can be attributed to the intervention", etc.
Just one case study may fully demonstrate that the intervention does not work as intended and may provide a convincing explanation for that. However, it is worth confirming such a finding by one or two additional case studies.
On the other hand, it takes more case studies to demonstrate that the intervention works, because all alternative explanations should be carefully investigated and rejected.
If professionally implemented, case studies provide for a highly credible and conclusive contribution analysis. The approach is nevertheless fairly demanding in terms of time and skilled human resources.

- Causal statements

The approach builds upon documents, interviews, questionnaires and/or focus groups. It consists in collecting stakeholders' views about causes and effects. Statements by various categories of stakeholders are then cross-checked (triangulated) until a satisfactory interpretation is reached. A panel of experts may be called to help in this process.
A particular way of implementing this approach consists in collecting beneficiaries' statements about impacts or direct results. Typically, a sample of beneficiaries is asked questions like "How many jobs would you say have been created/lost in your firm as a result of the support received?" or "To what extent is your present situation/behaviour attributable to your participation in the intervention?" In this approach the interviewee is asked to apply the policy-off scenario on his/her own.
Evaluation teams tend to prefer this approach which is far more feasible, but nobody should forget that the difficulty is transferred to the respondents. Most often, interviewees do not have a clear view of the policy-off scenario. They try to make up their minds in a few seconds during the interview and in doing so are subject to all kinds of biases.
When interviewing beneficiaries, the evaluation team often faces difficulties due to deadweight, since interviewees tend to exaggerate the effect of the evaluated intervention on their own behaviour or situation. In other words, they tend to underestimate the changes that would have occurred in the absence of the intervention. This results in a bias which is called deadweight.
In order to avoid this bias, the evaluation team should never rely upon a single naive question like "How many new jobs have been created as a result of the support received?" or "How much has your income increased as a result of the project?" By contrast, multiple triangulated questions may enable the evaluation team to assess and reduce the bias. Beneficiaries' statements are called "gross effects" (including bias) whilst the evaluation team's estimate is called a "net effect" (corrected from bias).

- Meta-analysis

This approach builds upon available documents, for instance:

• Previous works pertaining to the evaluation as a whole (monitoring, audit, review, etc.)
• Recent reports related to a part of the intervention, e.g. a project, a sector, a cross-cutting issue (evaluation reports but also monitoring, audit, review, etc.)
• Lessons learnt from other interventions and which can be used in answering the question.

In performing meta-analyses, the evaluation team needs to (1) assess the quality of information provided by the reviewed documents, and (2) assess the transferability to the context of the evaluation underway.

- Generalisation

The first two approaches (counterfactual and case studies) have the best potential for obtaining findings that can be generalised (see external validity), although in a different way. Findings can be said to be of general value when all major external factors are known and their role is understood. Counterfactual approaches build upon explanatory assumptions about major external factors, and strive to control such factors through statistical comparisons involving large samples. Case studies strive to control external factors through an in-depth understanding of cause-and-effect mechanisms.

Recommendation

The evaluation team should be left with the choice of its analysis strategy and analytical approach.

Cause-and-effect questions

What does this mean?

Cause-and-effect questions pertain to the effects of the evaluated intervention. They are written as follows:

• To what extent has [the intervention] contributed to [the expected change]?
• How far has [the intervention] helped to achieve [the expected change]?

These questions call for an observation of change, and then an attribution of observed change to the intervention, or an analysis of the intervention's contribution to observed changes.
Questions pertaining to direct and short-term effects can easily be answered. Far-reaching effects (e.g. poverty reduction) raise feasibility problems.

Causality and evaluation criteria

Effectiveness and impact questions tend to be cause-and-effect questions in the sense that they link the evaluated intervention (the cause) to its effects.
Efficiency and sustainability questions are also cause-and-effect questions since actual effects have to be analysed first, before being qualified as cost-effective or sustainable.
Generally speaking, relevance and coherence questions are not cause-and-effect questions. Typical examples are:

• How far are EC objectives in line with needs as perceived by the targeted population?
• To what extent are the effects of the intervention and the effects of other EC policies likely to reinforce one another?

The latter example involves causes and effects, but only in a prospective and logical manner. The evaluation team is not expected to assert the existence of cause-and-effect links and/or to assess the magnitude of actual effects.
Exceptionally, some relevance questions may call for cause-and-effect statements, for instance:

• How far did the involvement of non-state actors in the design of the EC strategy contribute to better adjustment of priorities to needs as perceived by the targeted population?

Questions pertaining to the EC value added may be cause-and-effect questions if the evaluation team attempts to assert the existence or the magnitude of an additional impact, due to the fact that the intervention took place at European level.

Caution!

Questions which do not require a cause and effect analysis do nevertheless call for a fully-fledged analysis covering all or part of data processing, exploration, explanation and confirmation.

.

Counterfactual

.

What does this mean?

The counterfactual, or counterfactual scenario, is an estimate of what would have occurred in the absence of the evaluated intervention.
The main approaches to constructing counterfactuals are

• Comparison group
• Modelling

What is the purpose?

By subtracting the counterfactual from the observed change (factual), the evaluation team can assess the effect of the intervention, e.g. effect on literacy, effect on individual income, effect on economic growth, etc.

Comparison group

One of the main approaches to counterfactuals consists in identifying a comparison group which resembles beneficiaries in all respects, except for the fact that it is unaffected by the intervention. The quality of the counterfactual depends heavily on the comparability of beneficiaries and non-beneficiaries. Four approaches may be considered for that purpose.

Randomised control group

This approach, also called experimental design, consists in recruiting and surveying two statistically comparable groups. Several hundred potential participants are identified and asked to participate or not in the intervention, on a random basis. The approach is fairly demanding in terms of preconditions, time and human resources. When the approach is workable and properly implemented, most external factors (ideally all) are neutralised by statistical rules, and the only remaining difference is participation in the intervention.

Adjusted comparison group

In this approach a group of non-participants is recruited and surveyed, for instance people who have applied to participate but who have been rejected for one reason or another. This approach is also called quasi-experimental design. In order to allow for a proper comparison, the structure of the comparison group needs to be adjusted until it is similar enough to that of participants as regards key factors like age, income, or gender. Such factors are identified in advance in an explanatory model. The structure of the comparison group (e.g. per age, income and gender) is adjusted by over- or under-weighting appropriate members until both structures are similar.

Matching pairs

In this approach a sample of non-participants is associated with a sample of beneficiaries on an individual basis. For each beneficiary (e.g. a supported farmer), a matching non-participant is found with a similar profile in terms of key factors which need to be controlled (e.g. age, size of farm, type of farming). This approach often has the highest degree of feasibility and may be considered when other approaches are unpractical.

Generic comparison

The counterfactual may be constructed in abstracto by using statistical databases. The evaluation team starts with an observation of a group of participants. For each participant, the observed change is compared to what would have occurred for an "average" individual with the same profile, as derived from an analysis of statistical databases, most often at national level.

Comparative approaches

Different forms of comparison exist, each with pros and cons, and varying degrees of validity.

• An "afterwards only" comparison involves analysis of the differences between both groups (participants and non-participants) after the participants have received the subsidy or service. This approach is easy to implement but neglects the differences that may have existed between the two groups at the outset.
• A "before-after" comparison focuses on the evolution of both groups over time. It requires baseline data (e.g. through monitoring or in the form of statistics, or through an ex ante evaluation), something which does not always exist. Baseline data may have to be reconstructed retrospectively, which involves risks of unreliability.

Strengths and weaknesses in practice

A well-designed comparison group provides a convincing estimate of the counterfactual, and therefore a credible base for attributing a share of the observed changes to the intervention. A limitation with this approach stems from the need to identify key external factors to be controlled. The analysis may be totally flawed if an important external factor has been overlooked or ignored. Another shortcoming stems from the need to rely upon large enough samples in order to ensure statistical validity. It is not always easy to predict the sample size which will ensure validity, and it is not infrequent to arrive at no conclusion after several weeks of a costly survey.

Modelling

The principle is to run a model which correctly simulates what did actually occur in reality (the observed change), and then to run the model again with a set of assumptions representing a "without intervention" scenario. In order to be used in an evaluation, a model must include all relevant causes and effects which are to be analysed. These are at least the following:

• Several causes including the intervention itself and other explanatory factors.
• The effect to be evaluated.
• A mathematical relation between the causes and the effect, including adjustable parameters.

Complex models (e.g. macro-economic ones) may include hundreds of causes, hundreds of effects, hundreds of mathematical relations, hundreds of adjustable parameters, and complex cause-and-effect mechanisms such as causality loops. When using a model, the evaluation team proceeds in three steps:

• A first simulation is undertaken with real life data. The parameters are adjusted until the model reflects all observed change correctly.
• The evaluation team identifies the "primary impacts" of the intervention, e.g. increase in the Government's budgetary resources, reduction of public debt, reduction of interest rates, etc. A set of assumptions is elaborated in order to simulate the "without intervention" scenario, that is to say, a scenario without the "primary impacts".
• The model is run once again in order to simulate the "without intervention" scenario (i.e. the counterfactual). The impact estimate is derived from a comparison between both simulations.

Modelling techniques are fairly demanding in terms of data and expertise. The workload required for building a model is generally not proportionate to the resources available to an evaluation. The consequence is that the modelling approach is workable only when an appropriate model and the corresponding expertise already exist.

.

EXTERNAL FACTORS

.

What are they?

Factors which are embedded in the context of the intervention and which hinder or amplify the intended changes while being independent from the intervention itself.

External factors are also called contextual, exogenous or confounding factors.

Why are they important?

• To understand their influence in order to properly disentangle the effects of the intervention from those of other causes.
• To identify the contextual factors which may prevent the transferability of lessons learned.
• To adjust samples of participants and non-participants in order to make them truly comparable and to achieve internal validity.

Typical examples

Factors explaining participation in the intervention:

• Potential applicants are (are not) familiar with the implementing agency.
• They belong (do not belong) to a "club" of recurrent beneficiaries.
• They have (do not have) a systematic grant-seeking behaviour.
• They have (do not have) social or economic difficulties preventing them from applying for grants / loans.

Factors explaining the achievement of specific impacts:

• Trainees had a high (low) level of education before entering the programme. In addition to the intervention this factor partly explains the skills they acquired.
• The target public had a high (low) level of knowledge before being reached by an awareness-raising campaign. In addition to the intervention this factor partly explains their current level of awareness.
• Beneficiary enterprises were large (small), young (old), and growing (declining) when they received support. In addition to the intervention these factors partly explain their current level of competitiveness.

Factors explaining global impact

• Local agriculture has benefited from exceptionally favourable weather during the programming period (or the opposite). In addition to the intervention this factor partly explains the change in farmers' income.
• International economic trends were exceptionally favourable during the programming period (or the opposite). In addition to the intervention this factor partly explains the evolution of exports.

When dealing with such external factors, the evaluation may usefully consult the contextual indicators that are available on the web.

How can they be identified?

In a given evaluation, external factors are potentially numerous and it is crucial to highlight the most important ones. The following approaches may help:

• Making the intervention logic explicit and uncovering implicit cause-and-effect assumptions
• Analysing previous research and evaluation work
• Taking expert advice
• Carrying out qualitative surveys

Identifying external factors is one of the main purposes of the exploratory analysis.

Recommandations

Do not try to identify all possible external factors when clarifying the intervention logic in the structuring phase of the evaluation. They are simply too numerous. This task should be undertaken only when working on a given evaluation question, and only if the question involves a cause-and-effect analysis.

.

EXPLORATORY AND CONFIRMATORY ANALYSIS

.

Exploratory analysis

What does this mean?

If necessary, the evaluation team delves into the collected data in order to discover new plausible explanations such as:

• New categories / typologies
• Explanatory factors
• New cause-and-effect assumptions
• Factors favouring / constraining sustainability

What is the purpose?

• To improve the understanding of all or part of the evaluated area, especially when existing knowledge and expertise are inadequate
• To develop and refine explanatory assumptions.

How to carry out the exploratory analysis

The analysis explores the set of data (quantitative and qualitative) with a view to identifying structures, differences, contrasts, similarities and correlations. For example, the analysis involves:

• Cross-cutting analyses of several case studies
• Statistical comparisons cutting across management data bases, statistical data bases, and/or the results of a questionnaire survey
• Comparisons between interviews and documents.

The approach is systematic and open-minded. Brainstorming techniques are appropriate. Ideas emerge through the first documentary analyses, interviews, and meetings. The exploration may continue through the field phase.

Confirmatory analysis

What does this mean?

Provisional findings progressively emerge during the first phases of the evaluation team's work. They need to be confirmed by sound and credible controls. That is the role of the confirmatory analysis.
In the particular case of cause-and-effect questions, the analysis is meant to disentangle several causes (the intervention and the external factors) in order to demonstrate the existence and/or assess the magnitude of the effect.

What is the purpose?

• To ensure that the findings are sound and able to withstand any criticism when the report is published
• To ensure that the findings are credible from the intended users' viewpoint
• In the particular case of cause-and-effect questions, to distinguish actual effects from observed change

How is a confirmatory analysis performed?

For a finding to be confirmed, it is systematically criticised by all possible means, e.g.:

• If the finding derives from a statistical analysis, are the validity tests conclusive?
• If the finding was suggested by a case study, is it contradicted by another case study?
• If the finding derives from a survey, can it be explained by a bias in that survey?
• If the finding is based on an information source, is it contradicted by another source?
• Is the finding related to a change that can be explained by external factors that the evaluation team may have overlooked?
• Does the finding contradict expert opinions or lessons learned elsewhere and, if so, can this be explained?
• Do the members of the evaluation reference group have arguments to contradict the finding and, if so, are these arguments justified?

Recommendations

• Devote relatively long interactions to the discussion of the final report in order to allow for a careful confirmatory analysis. Ensure that the evaluation team has put aside sufficient resources for that purpose.
• Not all findings require the same level of confirmation. Concentrate efforts on findings that support the most controversial conclusions, the lessons that are the most likely to be transferred, or the recommendations that are the most difficult to accept.
• In order to enhance the evaluation's credibility, it is valuable to present the criticisms that the findings withstood during the confirmatory analysis, in an annex.

.

VALIDITY

.

What does this mean?

Validity is achieved when:

• Conclusions and lessons are derived from findings in a way which ensures transferability (external validity)
• Findings are derived from data without any bias (internal validity)
• Collected data reflect the changes or needs that are to be evaluated without bias (construct validity)

What is the purpose?

A lack of validity may expose the evaluation to severe criticism from those stakeholders who are dissatisfied with the conclusions and recommendations, and who will point out any weaknesses they may have found in the reasoning.

Validity is part of the quality criteria. It should be given an even higher level of attention when the intended users include external stakeholders with conflicting interests.

External validity

Quality of an evaluation method which makes it possible to obtain findings that can be generalised to other groups, areas, periods, etc. External validity is fully achieved when the evaluation team can make it clear that a similar intervention implemented in another context would have the same effects under given conditions.

Only strong external validity allows one to transfer lessons learned. External validity is also sought when the evaluation aims at identifying and validating good practice.
External validity is threatened when the analysis fails to identify key external factors which are influential in the context of the evaluated intervention but would have a different influence in another context. External factors should not only be identified; the magnitude of their consequences should also be assessed. In the instance of a survey, this calls for larger and more diverse samples. In the instance of case studies, it requires a multiplication of the number of cases.

Internal validity

This is the quality of an evaluation method which, as far as possible, limits biases imputable to data collection and analysis. Internal validity is fully achieved when the evaluation team provides indisputable arguments showing that the findings derive from collected facts and statements.

Internal validity is a major issue in the particular case of cause-and-effect questions. When striving to demonstrate the existence and/or to assess the magnitude of an effect, the evaluation team is exposed to risks such as:

• Overlooking cause-and-effect mechanisms which contradict initial assumptions.
• Deriving impact estimates from a comparison of samples (e.g. participants and non-participants) that are not similar enough.
• Having findings that do not withstand statistical tests because samples are not large enough.

Construct validity

This is the quality of an evaluation method which faithfully reflects the changes or needs that are to be evaluated. Construct validity is fully achieved when key concepts are clearly defined and when indicators reflect what they are meant to.

Construct validity is threatened if the evaluation team does not fully master the process of shifting from questions to indicators. Construct validity is also at risk when the evaluation team uses indirect evidence like proxies.

Recommandations

• Never start analysing data without a thorough understanding of the context. Pay enough attention to identifying key external factors.
• The valid analysis of an impact requires time and resources. For this reason, cause-and-effect questions may need more resources than others.

CONCLUSIONS AND LESSONS

.

What does this mean?

Conclusions provide clear answers to the evaluation questions. They incorporate value judgements. Lessons are transferable conclusions to subsequent cycles of the same intervention or of other interventions.

What is the purpose?

• To ensure that the report provides true answers to the questions asked.
• To clearly identify the points at which the evaluation involves value judgements, and to ensure that those judgements are not implicit or badly justified, as this would be a weakness.

How should they be formulated?

Conclusions and lessons stem from the preceding steps as follows:

Conclusions answer the questions

The questions asked at the beginning of the evaluation find their answers by means of the conclusions. A conclusion may answer several questions and several conclusions may answer a single question.

Provided that all the questions asked have been answered, the evaluation team can present additional conclusions to take into account unexpected and important information and results.

The conclusions follow from data and findings

Upon writing a conclusion, what is being judged is one aspect of the intervention, for example: a strategic guideline (Is it relevant?), a practice (Is it efficient?), an expected effect (Was it obtained?), or an unexpected one (Is it positive?).

Thus, conclusions stem from collected data and evidence, from analysis and interpretations performed, from findings and new knowledge generated.

Conclusions are based on judgement criteria

To formulate its conclusions, the evaluation team applies the judgement criteria (also called "reasoned assessment criteria") that were agreed upon in the first phase (desk) of the evaluation. Data collection and analysis are structured according to these criteria. As long as this is possible, the findings are compared against targets.

• Example: The support has contributed towards increasing the number of qualified and experienced teachers by 30% in the areas where ethnic minority X concentrates (finding), which has allowed that area to catch up with the country average (target).

At the stage of the draft final report, the evaluation team may have to refine its judgement criteria and targets. In such a case, the issue is discussed with the reference group.

A lesson is a transferable conclusion

A lesson is a conclusion that can be transferred to subsequent cycles of the same intervention or to other interventions.

• Example: By connecting its disbursements to specifically designed performance indicators, the EC can successfully contribute towards improving the capacity to enrol pupils from disadvantaged groups.

How should they be presented?

One chapter of the report introduces the conclusions relative to each question, as well as the conclusions that emerge from points not raised by the questions. The conclusions are organised in clusters in the chapter in order to provide an overview of the assessed subject.

The chapter does not follow the order of the questions or that of the evaluation criteria (effectiveness, efficiency, coherence, etc.)

It features references to the sections of the report or to annexes showing how the conclusions derive from data, interpretations, analysis and judgement criteria.

The report includes a self-assessment of the methodological limits that may restrain the range or use of certain conclusions.

A paragraph or sub-chapter picks up the 3 or 4 major conclusions organised by order of importance, while avoiding being repetitive. This practice allows to better communicate the evaluation messages that are addressed to policy makers within the Commission.

The conclusion chapter features not only the successes observed but also the issues requiring further thought on modifications or a different course of action.

Suggestions

• Drafting a good conclusion requires that attention be paid to clarifying and justifying value judgements. It is therefore preferable to focus on a few key conclusions rather than on a large number of poorly justified ones. The evaluation team and the people in charge of quality assurance are therefore advised to carefully reread the final report and to eliminate any unessential and/or unintended value judgements.
• It is difficult to calmly discuss the judgement criteria and the targets with the reference group at the end of the evaluation because everyone immediately sees the influence of the discussion on the conclusions. That is why it is preferable to go into as much detail as possible in the explanation of criteria at the inception stage.
• Whenever possible, the evaluation report states the findings (which follow only from facts and analysis) separately from the conclusions (which involve a value judgement). This approach demands explicit judgement criteria and enhances quality.
• The evaluation team presents its conclusions in a balanced way, without systematically favouring the negative or the positive conclusions.
• If possible, the evaluation report identifies one or more transferable lessons, which are highlighted in the executive summary and presented in appropriate seminars or meetings so that they can be capitalised on and transferred.

.

RECOMMENDATIONS

.

What is this?

The recommendations are derived from conclusions.They are intended to improve or reform the intervention in the framework of the cycle under way, or to prepare the design of a new intervention for the next cycle.

What is the purpose?

• To optimise use of the evaluation in the form of feedback.
• To create a positive approach and easier take-up when the evaluation reveals problems.

How to draft and present them

The recommendations must be related to the conclusions without replicating them. A recommendation derives directly from one or more conclusions.

The recommendations must be clustered and prioritised. The report mentions the addressees of the recommendations, e.g. EC Delegation, services in charge of designing the next intervention, etc.

The recommendations are useful, operational and feasible, and the conditions of implementation are specified. Wherever possible and relevant, the main recommendations are presented in the form of options with the conditions related to each option, as well as the predictable consequences of the implementation of each option.

The recommendations are presented in a specific chapter. This chapter highlights the recommendations derived from the three or four main conclusions. 

How to promote them

The recommendations are valuable as far as they are considered and, if possible, taken up by their addressees.

To promote their take-up, the manager drafts a fiche contradictoire in order to:

• List the recommendations in a shortened form
• Collect the addressees' responses
• Inform on actual follow-up to the recommendations, if any.

Advice

• " If a recommendation does not clearly derive from the conclusions, it probably reflects preconceived ideas or the tactical interests of one of the stakeholders. Its presence in the report could then discredit the evaluation. The evaluation manager must therefore be careful to refuse any recommendations that are not based on collected and analysed data, and on the conclusions.

.

ETHICAL PRINCIPLES

.

What is this?

The conclusions include value judgements on the merits and worth of the intervention. This dimension of the evaluation exercise is particularly sensitive and the evaluation team has therefore to respect specific ethical principles.

What is the purpose?

• To guarantee an impartial and credible judgement (also called "reasoned assessment").
• To ensure that the judgement does not harm anyone.

What are the main principles?

- Responsibility for the judgement

The conclusions are primarily a response to questions. Members of the group are partially responsible for the judgement in so far as they orientate it through the evaluation questions they validate.

The external evaluation team also intervenes in the preparation of the judgement by making proposals to define the questions, clarify the judgement criteria and set the targets.

In the synthesis phase, the evaluation team applies the judgement criteria agreed on, as faithfully as possible, and produces its own conclusions. The conclusions are discussed within the reference group but remain the entire responsibility of the evaluation team.

As part of the quality assurance process, the evaluation manager can require sounder justification of a judgement, or better application of an agreed judgement criterion. By contrast, he or she cannot require the removal or amendment of a conclusion if it is methodologically sound.

- Legitimacy of the judgement

The questions and criteria take into account the needs and point of view of the public institution that initiated the evaluation.

The members of the reference group contribute different points of view, which reinforces the legitimacy of the evaluation.

During the desk phase the evaluation team holds interviews, which may enable it to identify other points that were not expressed by the reference group members. It makes them known in reference group meetings and may take them into account in the judgement criteria.

More generally, the evaluation team has a responsibility to bring to light important findings and judgement criteria which have arisen during the evaluation process, even if they are not covered by the evaluation questions, provided that such points are legitimate.

A point of view is legitimate if:

• It is expressed by stakeholders or in their name.
• It expresses an aspect of the public interest and not the individual interest of one person or the private interest of an organisation.
• It is compatible with basic human rights.

- Impartiality of the judgement

The impartiality of the judgement concerns the entire evaluation, that is, the choice of questions and judgement criteria, the determination of targets and the formulation of conclusions.

The entire process is exposed to risks of partiality, for example:

• The evaluation team favours its own preconceptions.
• The evaluation team implicitly favours the point of view of one of the stakeholders.
• The evaluation team does not hear, understand or take into account the point of view of one of the stakeholders.
• The evaluation team systematically focuses on the negative or positive conclusions.

When there are differences in the way of judging, in the judgement criteria or in the target levels, impartiality consists in:

• Making sure that evaluation team members are familiar with and respectful of beliefs, manners and customs of concerned groups.
• Respecting all cultures and standpoints, whilst conforming to universal values as regards minorities and particular groups, such as women. In such matters, the United Nations Universal Declaration of Human Rights (1948) is the operative guide.
• Being aware of asymmetrical power relations, and correcting the biases arising from them.
• Making sure that all the opinions are heard, even if they are not conveyed by the loudest voices or the majority.
• Reporting on differences in the reports and at reference group meetings.
• Explaining the choices transparently (Who made the choice? Why? What were the alternatives?).

In case of divergence, a solution may consist in judging in relation to several criteria and/or formulating several conclusions that correspond to different points of view. This solution has the drawback of diluting the conclusions and thus of making the evaluation less conclusive.

It is often preferable to make choices and to explain them transparently.

- Protection of people

The conclusions concern the merits of the evaluated intervention, not the people who implement it or benefit from it.

Individuals' names are cited only when this enhances the credibility of the evaluation. The evaluation team must respect people's right to provide information in confidence and ensure that sensitive data cannot be traced to its source. Before citing a person or organisation, the evaluation team or any other evaluation actor anticipates and avoids the risks involved for that person or organisation.

Evaluations sometimes uncover evidence of wrongdoing. Bad professional practices are never reported in a traceable way. However, the evaluation team member who encounters illegal or criminal acts deals with them as any other citizen should do. In the later case, the issue should be discussed with the evaluation manager.

KEY ELEMENTS

.

What is this?

Establishing quality check-points at the main phases of the process, defining the responsibilities of quality assurance, and establishing the rules of the game in case of quality defects.

Quality assurance is organised on a step-by-step basis: successive aspects of quality are secured at each step, thus creating a sound base for the subsequent step.

What is the purpose?

• To enhance the conclusions of the evaluation in users' eyes by showing that this is impartial and rigorous work that meets professional standards.
• To gradually construct quality and thus avoid discovering a quality defect in the final report stage.
• To clarify relations between the evaluation manager, the reference group and the external evaluation team.

Rules of the game

The rules of quality assurance are specified in the terms of reference and pointed out when the external evaluation team is engaged.

These rules concern:

• Approval of documents
• Quality criteria
• Dissemination of the quality assessment of the final report

The quality assurance process benefits from the contribution of all actors whilst limiting the potential conflicts that might arise between them, e.g.

• The actors who hold responsibilities in the evaluated intervention (the evaluees) are in a good position for assessing the relevance of data collected and the fairness of interpretations. Their assessment may, however, be distorted by a confirmation bias. They should therefore be given only an advisory role, e.g. through their participation in the reference group.
• The evaluation team leader has a major role in assuring quality, especially in designing an adequate method, and in securing the accuracy and fairness of the report. He/she may, however, overweight the views of powerful stakeholders, especially those who are likely to commission other evaluations in the future. The evaluation manager's quality assessment helps to prevent such a risk.

Key steps

Inception report

In the inception report stage, quality assessment is used to ensure that the evaluation team has clearly understood the scope and aims of the evaluation, as well as the logic of the evaluated intervention and the questions to answer.

First phase report (desk)

In the first phase report stage (desk), quality assessment is used to check the quality of the documentary analysis as well as the appropriateness of the proposed method for the field phase, including the indicators, data sources and analytical approach.

Final report

In the final report stage, quality assessment checks whether the evaluation provides valid and impartial answers to the questions asked, and whether the form of the report is suited to the targeted users' needs.

At this stage the quality assessment concerns first the draft version of the report and then the final version.

Key players

It is expected that all key players involved in the quality assurance process have a sufficient evaluation capacity, except reference group members. If a key player is not qualified enough, he/she ensures that appropriate support is provided.

Evaluation team leader

The evaluation team leader is primarily concerned with preventing major risks that threaten quality, e.g. overlooking a major question, not consulting an important stakeholder, elaborating upon unreliable data, judging in a partial way.

He/she also ensures that each report is submitted to a detailed quality check before it is released to the evaluation manager.

Quality assessor(s)

The company or consortium in charge of the evaluation contract appoints one or more quality assessors who carefully check each report for quality before it is released to the evaluation manager. The quality assessor should be experienced in evaluation and should not belong to the evaluation team.

Evaluation manager

The evaluation manager has the ultimate responsibility for assessing whether or not the methodological quality of a report is sufficient for allowing the next step to be taken.

In this process he/she takes advice from the reference group members, and relies upon technical support as far as necessary.

Reference group

Reference group members receive all draft reports for comments. They may comment on the factual bases substantiating the evaluation, which is a contribution to quality assurance. When their comments apply to the content of conclusions, they are passed on to the evaluation team which freely decides on whether or not to take them into account.

.

APPROVAL OF DELIVERIES

.

What is this?

Formal recognition by the evaluation manager that a delivered document has an adequate content and meets the applicable quality criteria in methodological terms.

What are the approval steps?

Generally, a document is approved in the following way:

• The manager checks that the document has the required form and content, and has no major quality defect. If it does, he/she immediately requests a new version.
• The manager asks for the opinion of the individuals or services that have to be consulted, either at a meeting (reference group) or by email. He/she gives a deadline after which there is no guarantee that comments will be taken on board.
• The manager lists his/her own requirements regarding quality improvements, attaches other requests received, distinguishes requests applying to the methodological quality from requests applying to the substance of the text, and sends all the comments to the external evaluation team.
• The evaluation team takes into account all the requests for quality improvements, either by amending its document accordingly or by mentioning in an annex how the request will be treated at a later stage, or else by explaining in an annex why the request has not been treated.
• A new version of the document is soon presented to the manager who accepts or refuses it.
• When approval is required, it is also a prerequisite for moving on to the next stage.

Which documents are approved and by whom?

The following documents require approval:

• Inception report
• First phase report (desk)
• Pilot mission report (when the evaluation includes missions in several countries)
• Draft final report
• Final report

Approval is generally performed by the evaluation manager and may be confirmed by his/her superior.

The reference group members validate the set of evaluation questions, i.e. group members formally accept, or at least do not reject, the set of evaluation questions. If necessary, divergences are resolved by the evaluation manager. In that case dissenting views are noted in subsequent reports. 

Rules of approval and independence

The approval process does not contradict the external evaluation team's independence. In this respect the requests for amendments made during the approval exercise need to be differentiated:

• Comments concerning methodological quality have to be taken into account. If the evaluation team considers that it cannot take a demand into consideration, this should be fully justified. An assessment of the methodological quality is attached to the final report.
• Comments concerning the substance of the document, statements and conclusions are freely taken into account or rejected by the evaluation team. The team must nevertheless, in a note or in the annex, mention the requests not taken into account and comment upon its decisions in this respect.

.

QUALITY ASSURANCE SUPPORT

.

What is this?

Assessing the quality of an evaluation is a multifaceted task. The commissioning body and the stakeholders are in the best position to assess whether their needs have been treated impartially. The experts and those who implement the intervention are in a good position for assessing the relevance of data collected and the fairness of interpretations. It is the evaluation manager's role to assure that the evaluation has sound methodological and factual bases. The manager receives a quality assurance support if he/she lacks training or experience.

Who can play this role?

Within the EC headquarter Evaluation Unit, managers have the required training and skills to assess the methodological quality of reports. All evaluations are monitored by two members of the Unit, both of whom participate in quality assessment.

In the case of devolved evaluations, the manager uses the quality criteria proposed by this site as well as the indications given to fill out the quality assessment grid, if necessary with the help of the Evaluation Unit.

The Evaluation Unit can perform cross-cutting quality assessments autonomously.

What type of tools?

.

The evaluation's four tasks

Evaluations are usually divided into four tasks which are not strictly sequential: the evaluation's organisation (based on the evaluation questions wording leading to the overall assessment), the collection of quantitative and qualitative information, the analysis of the information collected, and the evaluation questions assessment leading to the formulation of conclusions and recommendations.
Objectives of these four tasks:

• The evaluation's organisation contributes to the selection (or definition) of evaluation questions, judgement criteria, and indicators for these criteria, so as to determine the evaluation's methodology.
• The information collection gathers all the primary and secondary data (quantitative and qualitative) available for answers to the evaluation questions.
• The analysis of the information is used to assess the assistance policy contribution to the observed evolutions of indicators.
• The assessment of each evaluation question leads to the formulation of an answer and conclusions for the evaluation.

The elaboration of an evaluation methodology based on the evaluation questions (which are designed for the development of an overall assessment) is crucial in the selection of tools used for each of the evaluation's four stages.

Importance of the documentary stage

Although no tool presented in this methodology is specifically dedicated to this purpose, the collection of information from the European Commission services and on-site (particularly the information collection using CRIS database) is a component of the evaluation process.
The following indicative list sets out the main documentary sources (secondary data) to be consulted for a country evaluation:

• Overall agreements, bilateral/multilateral and sector-based/thematic agreements (trade, investment, science and technology, etc.), agreements of co-operation, partnership, association, and conclusions of bilateral and multilateral conferences
• Country/Regional Strategy Papers (CSPs and RSPs) and National/Regional Indicative Papers (NIPs and RIPs)
• The available annual reports and mid-term reviews
• The European Court of Auditors' reports
• Governmental official papers (such as the Poverty Reduction Strategy Papers (PRSP) and available sector-based strategies) and papers produced by multilateral and bilateral co-operation agencies (strategy papers, project papers, etc.)
• The available thematic or sector-based project and programme evaluations
• Reports of monitoring (ROM) on projects and programmes in progress
• The European Commission's general regulations and all regulation documentation and political agreements covering the evaluation's period

Presentation of the tools

Thirteen tools have been developed. They are usually familiar to the evaluators. Their specificities are described here.

Objectives diagram and impact diagram

The objectives diagram displays the classification of the objectives to be achieved for the strategy implementation, from the European Union's global objective to activities carried out for operational programmes. The impact diagram displays the classification of activities, outcomes and expected impacts. The expected impacts are the objectives in terms of results.
The tool reconstitutes the intervention rationale and the expected impacts; as such, it plays a crucial role in the organisation stage of the evaluation's complex interventions through the wording of evaluation questions.

Problem diagram

Projects and programmes in development assistance aim at satisfying priority needs through the resolution of a range of issues. It is theoretically possible to construct a diagram taking the shape of a tree, with the trunk (the core problem), roots (the causes) and branches (the consequences and impacts).
The problem diagram, associated with the impact diagram, validates the relevance of a project, a programme or a strategy by relating expected impacts to the problems they should be contributing to solve.

Decision diagram

The decision diagram displays the process during which the strategic objectives and the overall co-operation policies with developing countries, which are defined by the European Union's assistance agreements, are converted into short-term and medium-term bilateral co-operation decisions.
Complementing the objectives diagram, the decision diagram facilitates the analysis of the strategy in terms of internal coherence (logical succession of the choices) and external relevance (contextual elements and position of stakeholders).

Interview

The interview collects information from stakeholders and beneficiaries throughout the evaluation stages: facts and verification of facts, opinions and points of view, stakeholders analyses and suggestions.

Focus group

The focus group is a means of discussing information, opinions and judgements already collected. The tool explains why opinions have been expressed (and the analyses supporting them) and checks their consistency. Focus groups are frequently used to collect the beneficiaries opinions concerning their participation in a programme and what they had drawn from it (positive and negative aspects). They are an alternative to interviews. Whatever their usage, the focus group's specificity is a collection of opinions which have been moderated by an in-depth discussion rather than a collection of spontaneous opinions.

Survey

The survey collects comparable answers from a sample of the population. When the sample is representative, the survey displays statistical measures which can be useful for quantified indicators.

Case study

Case studies are the preferred evaluation tool when "how" and "why" questions are being posed, because they allow a detailed examination of the actual elements in line with the evaluation goals. In contexts allowing or requiring it, the case(s) can be selected to yield general conclusions for the overall evaluation.

Expert panel

The expert panel is a group of independent specialists, recognised in at least one of the fields addressed by the programme under evaluation. The panel yields a collective assessment which is nuanced, argued and supported by the knowledge and experience of the experts.

SWOT

SWOT analysis combines the study of the strengths and weaknesses of an organisation, a geographical area, or a sector, with the study of the opportunities and threats to their environment. Frequently used in ex ante evaluations, it can also be used in ^{ex post} evaluations to assess the orientations taken.

Context indicators

The tool ranks a country through the comparison of its context indicators with that of other countries. A context indicator is a datum which produces simple and reliable information describing a variable relative to the context. The tool evaluates development dynamics through the comparison of the level and evolution of a country's main indicators with that of other countries with similar contexts.

Multicriteria analysis

In ex ante situation, multicriteria analysis is a decision-making assistance tool. In ex post evaluations, it usefully contributes to the formulation of a judgement based on a range of heterogeneous criteria.

Cost-effectiveness analysis

The tool identifies the economically most efficient way to fulfil an objective. It compares the efficiency of projects or programmes with comparable impacts. It usefully contributes to the formulation or validation of a judgement on the selection of the most efficient projects and programmes.

Cultural and social analysis

In country evaluations, the cultural and social analysis identifies the constitutive components of social, ethnic, religious and interest groups. It also highlights all the values shared in a society as well as its internal divisions.

Absence of statistical tools

In the context of assistance to developing countries, statistical data are often difficult to collect and their relevance is limited by the delays in their publication and a weak reliability. The available data are general and descriptive, and only allow for straightforward analysis. Thus, it is common practice to develop tools based on easily available data.

Rules for the methodology's application

Generally speaking, the feasibility of field work or the limitations of such a task should be checked. Indeed, specific contexts within the country where the study is to be carried out, such as logistical and implementation costs constraints, can constrain the choice of tools.
Prior to the presentation of the selection criteria for the most appropriate tools for the methodology to be applied, rules for the methodology's application should be recalled:

• No tool can answer a question or fulfil one of the four stages of the evaluation by itself, which means that evaluations need a combination of tools.
• Each of the tools is adapted to a specific stage, sometimes several.
• Several tools are used concurrently in the same stage with different approaches in order to facilitate the triangulation of information.
• The selection of evaluation tools depends on the tasks to be achieved and the context: the degree of difficulty of the intervention, quality of the available expertise, nature of the information to be collected, multiplicity of the interlocutors, etc.
• The tool selection must be guided, except for specific reasons, by the homogeneity of the detail and the degree of precision of the information required for the analysis.

In short, the evaluation team should use several tools and choose the most efficient combination of tools for the specific context of the evaluation.

.

When to use which tools?

.

The tools and the evaluation's four functions

A first selection can be made when the tools are classified within the four tasks of the evaluation.
The table below sets out the tasks for which each tool is normally used and other tasks where it could be usefully incorporated. Most tools have a main function and one or more secondary functions. This list is indicative only.

Organisation tools

The organisation of the evaluation can usefully be supported by a series of tools called organisation tools. In homogeneous project or programme evaluations, the normal organisation tool is the logical framework which describes the objectives (overall and specific) of the intervention, the issues to which the evaluation answer and the expected outcomes. A problem diagram can usefully complement the logical framework. Evaluations where the scope includes a range of heterogeneous objectives and activities (such as geographic evaluations), theoretically require three tools, the objectives diagram and the impact diagram being the core organisation tools. In complex programmes or strategies, the problem diagram brings precision on the relevance of objectives, identifies the goals and issues of the interventions, as well as the problems neglected by the evaluation's objectives. The decision diagram complements the objectives diagram with information about the reasons for the programme's orientations, and especially the basis for the rejection and negotiation of options. These tools are also useful for the analysis and judgement stages (polyvalent nature of the tools).

Collection, analysis and judgement tools

During these three evaluation stages, a large range of tools is available, complementary to each other and/or polyvalent. As a consequence, numerous factors are taken into account to make an efficient choice. Although no rule securing an optimal choice exists, a logical process can facilitate the development of a homogeneous methodology which will provide well-grounded answers to the evaluation questions.

Polyvalent nature of the tools

The resources allocated for an evaluation are not sufficient for the implementation of all the tools mentioned previously. Choices must be made with respect to the evaluation's priorities and the maximisation of the use of resources. The wider the scope and questioning, the greater the risk of dispersion, which means that the evaluation team must ensure that their observations and analyses provide answers to the most essential issues of the evaluation. The evaluation team should remember that several tools are usually relevant for the same evaluation stage and can be used to confirm the information, and that the tools are often polyvalent and can answer several questions at the same time.

Selection criteria for the tools

Besides the tools specific functions and their ability to be implemented in one of the four stages of the evaluation, other selection criteria should be examined.
The selection criteria set out below should guide the evaluator through a series of choices needed for the development of the methodology.

• Knowledge of the techniques. Although the majority of the tools presented are easily implemented, some may require prior experience. This is particularly the case for tools necessitating specific group moderating skills, and complex tools such as multicriteria analysis, cost-effectiveness analysis or surveys.
• Need for specific data. The implementation of some tools requires the collection of specific data without which their conclusions would be ill-founded. For example, the implementation of cost-effectiveness analysis is impossible without indicators of effectiveness measuring comparable projects. Their availability and reliability must therefore be checked before the cost-effectiveness analysis is used.
• The prerequisites for the tool's usage. This issue is particularly important for tools whose implementation takes place during the field stage in the partner country. As the implementation of such tools often generates high costs, their relevance within the overall methodology, cost-effectiveness and efficiency (particularly for the data collection) must be secured.
• Implementation time. Some tools, such as the surveys and some types of focus group investigations and expert panels, need a preparation stage before their implementation on-site
• The availability of qualified local experts, capable of conducting specific tools in the local language. This issue is particularly crucial for tools requiring group moderating skills (focus groups, etc.) for which available and skillful experts are sometimes hard to find.

The table below grades each tool using the five criteria and indicates those demanding particular attention. Each criteria is awarded a grade from 1 to 3. Grade 1 means that the criteria does not constitute a particular problem for the tool; grade 3 means that it is recommended that the feasibility of the tool's implementation should be checked with regards to this criteria.

The tools' specific requirements

A grade 3 does not mean that the tool should not be used. Indeed, the priority should be given to the tools providing the best answers to the evaluation questions, and then, the evaluator can check the possibility of using them in the context and with the available resources of the evaluation.

Development of a homogeneous methodology

• The methodology's development must target the organisation of a homogeneous and efficient technique. It is useless to schedule the implementation of sophisticated tools for one stage of the evaluation if the other stages are not addressed with the similar precision or logical process, or if the other tools cannot provide the required information or use the information collected.
• The collection stage must be carefully managed because if it leads to incomplete findings, the organisation of a new information collection is challenging during the field stage. The same attention should be given to analysis and judgement tools to be implemented in the country under evaluation.
• To do so, it is useful to assess the risk of an unsuccessful implementation for each tool, in order to plan alternative solutions if needed be and limit the impact of such a failure on the evaluation as a whole.

In essence, the methodology should be constructed with a range of available tools and take into account their advantages and limitations, the conditions for their implementation in the context of development assistance evaluations, the prerequisites for their implementation and the limits of their findings due to the context.

Analysis of the tools

Objectives diagram and impact diagram

Problem diagram

Decison diagram

Interview

Focus group

Survey

Case study

Expert panel

SWOT

Context indicators

Cultural and social analysis

Multicriteria analysis

Cost-effectiveness analysis

.

Which combination of tools?

.

The combination rationale

Some tools require the implementation of other tools prior to their use. Such is the case when a tool yields useful information for the implementation of another tool, or when bringing a different viewpoint to the analysis, it strengthnens or nuances the conclusions reached which another tool. Before taking the decision to implement complex tools (such as judgement tools), the evaluator should check whether preliminary information which is capable of improving their performance are available, and find the optimum tool capable of yielding such information.
The most frequent cases of combinations of tool are listed in the table below. For example, the first line shows that objectives diagrams and impact diagrams require or may require the implementation of interviews and focus groups.

Tools required by other tools

The table is indicative and other combinations can be developed in particular contexts. Usually, collection tools (interviews, focus group and surveys) are the most frequently combined with analysis and judgement tools, because the latter require specific information for their implementation. Thus, special care should be given to analysis and judgement tools, so as to ensure the homogeneity of the methodology and the maximum performance of the tools throughout the evaluation.

The tools' frequency of use

The table shows that the interview is the tool the most frequently used by other tools, including collection tools (which is not surprising). Conversely, case studies are seldom used and belong to the category of meta-tools requiring the support of all the collection tools and, if needs be, analysis tools.
Although presenting an exhaustive table of the tool's combinations is not possible, recommendations should be made to ensure the homogeneity of the tools performance all through the evaluation:

• Some tools are frequently used by other tools. For example, the interview is the collection tool the most often used by other evaluation tools.
• Most of the tools are multifunctional. The decision to use a specific tool should thus be based on its conditions of use and on the constraints related to the implementation of additional tools.
• Because of the polyvalent nature of the majority of the tools, criteria for rationalisation and optimisation must support their selection. For example, evaluation questions should be classified depending on the tools selected and vice-versa.

Example of combination

Testing of two country evaluation tools - the survey and the focus group - during the evaluation in Benin.
The test checked the possibility of providing elements to answer the following evaluation question through the implementation of a combination of two tools: "To what extent are the European Commission's sanitary strategy and interventions adapted to the fundamental needs of the population, and particularly the poor?"
The objectives of the test were to collect and analyse information and points of view from beneficiaries, in order to assess the global trends of the service provided by health centres. The test combined three surveys and four focus groups.
Three surveys:

• Two structured questionnaires, one addressed to households and the other to the users of the health centres, which resulted in a total of 660 respondents
• One open-ended questionnaire addressed to the local authorities, health professionals and people in charge of insurance companies and local associations, which resulted in a total of 100 respondents.

Four focus group investigations:

• Two focus groups of beneficiaries (countrywomen) in two different villages.
• One focus group with midwives and nurses.
• One focus group with doctors from the private and public sector.

The last 2 focus group investigations were set up to complement the outcomes of the 2 focus groups with beneficiaries and that of the surveys. The goal was to confront different perspectives on a same situation where a few dysfunctional elements could be raised.
The confrontation of the survey's outcomes with that of the focus group contradicted the hypothesis at the start of the reorganisation of the public health system implemented by the European Commission in Benin. It also asserted that the access to health care in the public sector was very limited for the poor population.

.

What specific constraints and requirements?

.

Limitations and risks depend on the constraints and requirements specific to each of the available tools, whatever their categories.

Possible constraints

Access to information and sources of information

Most of the tools require relatively straightforward access to baseline documentation (Objectives diagram and impact diagram, Problem diagram, Decision diagram, Case study, Context indicators). Some tools strongly depend on the representative nature of the interlocutors and stakeholders, on their availability and co-operation, such as the Interview, the Focus group and the Survey.

Quality of the collected information

The quality of the collected information is crucial, and can be influenced by:

• The insufficiency and/or approximate nature of the information (SWOT, Context indicators, Cost-effectiveness analysis, Multicriteria analysis, Cultural and social analysis)
• Bias in the opinions expressed (Survey, Focus group)
• Partiality in the judgements or bias in the analyses (Expert panel, Cultural and social analysis)
• Partiality in the analysis of the strengths/weaknesses (SWOT)
• An ill-founded generalisation (Case study)
• Answers difficult to compare (Interview)
• An inappropriate selection of context indicators (context indicators)
• An excessive level of abstraction in the analysis and too great a distance from reality (Cultural and social analysis)

Knowledge of specific techniques

A good knowledge of the tool is needed for all the tools, and especially for: the Survey, the Cultural and social analysis, SWOT, Cost-effectiveness analysis, Multicriteria analysis.
The availability and experience of acknowledged experts must be confirmed, particularly for: the Case study, the Cultural and social analysis, the Expert panel, the Focus group.
The processing of answers can quickly become unmanageable because of a lack of sufficient competence (Survey).
A lack of experience can lead to inappropriate choices in study cases, with no direct relationship with the evaluation's objectives (Case study).
The attempt at consensus seeking on the perception of threats and opportunities requires a specific knowledge (SWOT).

Prerequisites

Human resources

The availability of acknowledged experts must be guaranteed whatsoever. It is a key condition for: the Cultural and social analysis, the Expert panel, SWOT, Cost-effectiveness analysis, Multicriteria analysis and Focus group.
The evaluator's expertise must be assessed for the organisation tools: Objectives diagram and impact diagram, Problem diagram, and Decision diagram.
Expertise is required for the data processing analysis developed in Surveys.
Experience in meeting management and moderation is crucial for: the Cultural and social analysis, the Expert panel, the Focus group.

Costs

Implementation costs should be examined particularly for the following tools: Cost-effectiveness analysis, Multicriteria analysis, Survey, Case study.

Time span

An appropriate time schedule is important for all tools, and particularly for the implementation of: Cost-effectiveness analysis, Multicriteria analysis, Survey, Case study and Interview.

Impact of preparatory stages

When a tool requires a preparatory information collection, its time span, cost and human resources should not be under-estimated. If needs be, access to sites and beneficiaries availability must be checked (Case study).

.

Check list for the tool's implementation

.

Check list for the evaluation team

• Does the selection of evaluation questions allow the use of tools to efficiently provide answers?
• Will the selection of tool help in formulating an overall assessment?
• Does the implementation of the tools selected provide relevant answers to the evaluation's objectives?
• Can each tool be adapted to the constraints and opportunities related to the specific conditions of the evaluation?
• Does the organisation of each of the tools take into account all the prerequisites for their implementation?
• Were the prerequisites for each tool (such as the competences, number of working days, expenses) precisely assessed? Are they relevant with regard to the expected outcomes? To the outcomes achieved?

Check list for contract managers

• Are the answers provided for each of the evaluation questions supported by the implementation of an effective combination of tools?
• Does the use of appropriate tools support the overall assessment?
• Is the choice of each tool and their combination clearly and convincingly explained?
• Are the available resources for the tool's implementation (experts, budget, time span) being effectively used?

Toolbox and directions for use

The aim of the evaluation is to produce operational recommendations supported by solid conclusions which are based on clear judgement criteria, solid and concrete information and rational argumentation.
Conclusions and recommendations are a means of discusssing about the programmes or policies with their authors and the operators involved. They must result from a rigorous process (the expert's own point of view is insufficient).
The development of a rigorous ad hoc methodology and the use of appropriate tools are the crucial components of this process. This website presents a series of evaluation tools and directions for use of the toolbox which:

• Integrates the tools within the evaluation's methodology
• Presents selection criteria for the tools
• Gives guidance on an appropriate combination of tools

SUMMARY

.

Why are this tools used in evaluation?

.

The analysis of problems is a means to test the validity of the objectives of a project, a programme or a strategy. As a programme aims at solving a range of problems, the evaluation should be concerned with the validity of its analysis. The evaluators should therefore check:

• The validity of the procedure: how have the problems been identified and classified?
• The apparent coherence of the problems' linkage: are their causal links relevant?

Figure 1: Analysis of the problems in the project cycle and in the strategy policy co-ordination cycle

.

Programmes and strategies depend on the analysis of the situation. This analysis presents the primary data relative to the problems addressed by the programmes and strategies, and the information about the context (economic, social, cultural, institutional, environmental) in which they will be implemented.

This crucial analysis identifies:

• The problems to be addressed by the activities in order of priority
• The positive elements on which activities can rely
• The actors to be involved in the activities
• The structural and dynamic constraints which may hinder the success of the activities

Therefore, the evaluation team can be asked to assess the quality of the analysis, and the conformity between the analysis of the situation and the adopted strategy (or programme). The problem diagram, as an ex post construction, can be one of the tools used to check the coherence and the relevance of the analysis in respect of the main contextual problems.

.

What are the possible uses of these diagrams?

.

When it is impossible to directly establish objectives diagrams, problem diagrams play an essential role in the organisation stage of the evaluation. Problem diagrams present a summarised vision of the situation to be improved, partially at least, by the strategy. The classified objectives included in the strategy should be deduced from the problems diagrams.

.

Figure 2: The objectives deduced from the problems

.

The reconstruction of the problem diagram includes a step which differentiates between context problems and intervention problems. As a consequence, the diagram resulting from this selection should be completely convertible into a logically reconstructed objectives diagram, i.e. the higher-order overall objective corresponds to the core problem, and each row of subordinated objectives to its equivalent row of problems.

.

How is the problem diagram constructed?

.

Stage 1: How to identify the problems?

Record the references to teh problems found in the documentation: quotations from the evaluation's baseline documents are used to identify problems which may not be systematically depicted as such in the texts. Sometimes, problems are identified as assistance objectives or impacts targeted by these objectives. Problems are thus expressed as:

• problems (presented as such)
• objectives, whose goal is to resolve explicitly or implicitly a problem
• expected impacts of the CSP assistance

The problems directly targeted by the intervention may not be explicitly identified. Main problems and context problems can be intermingled, which complicates the construction of the diagram. The evaluator will only be able to differentiate the two types of problems after the completion of the diagram. 

The crucial stage of the process is the identification of the core problem among the variety of the selected problems. Three situations can be encountered:

• The core problem is mentioned in the documentation
• The core problem is not clearly mentioned, whereas the overall objective is. In this case, the evaluator can deduce the core problem from the higher-order objective.
• Neither the core problem, nor the overall objective are explicitly mentioned

In the last situation, the evaluator plays the role of the planning manager, and decides which of the problems will be the core problem. He/She may:

• Take the decision, on the basis of his/her rationale
• Benefit from the assistance of specialists during interviews or working sessions

The selection of the core problem should be conducted concurrently with the classification of the problems into levels. Indeed, the selection of the core problem should be supported and justified by the coherence of the whole diagram.

Stage 2: How to construct a problem diagram?

Figure 3: Stages involved in the drafting of temporary diagram

Usage highlights the fact that these two types of problems/causes are the easiest to identify, whereas intermediary causes are the hardest to determine and classify. Thus, it is recommended that the development of the diagram starts with its extremities at the same time.

Figure 4 : Problem diagram in the transport sector in Tanzania

Test the temporary diagram

Where possible, the authors of the documentation referred to above should test the diagram in order to validate the classification of the problems by rows and links. The aim is to check that the evaluator's interpretation reflects the authors' intention correctly. If the authors are not available or, in order to complement their original contribution, the evaluator should consider asking for the participation of other actors responsible for the drafting process. 

Contacting the authors is usually possible when the documentation is recent and the authors are still in their position or contactable in one of the services. This task is challenging when the documentation is old and its authors are not easily identifiable or contactable.

Develop the final version of the problem diagram

The final version takes into account the opinions collected during the testing of the temporary diagram. It is an accurate account of the initial intentions logically reconstructed of the European Commission, taken from official documentation.

.

What are the preconditions for its use?

.

The need for human resources varies with the tasks to be achieved and the situation encountered: reconstruction of the problem diagram with the analysis of the situation displayed in the documentation; retrospective reconstruction of the problem diagram as it should have been established to support the relevance of the strategy or programme under assessment. 

Most of the graphic problems can be solved with software like MS PowerPoint. 

The evaluator can also benefit from specific software designed to assist decision-making, such as MS Visio, TeamUp-PCM for Windows, and Palisade PrecisionTree.

What are the advantages and limitations of the tool?

.

SUMMARY

.

Why are these tools used in evaluation?

These diagrams are usually used as organisation tools. They provide a framework for the information collection and the undertaking of in-depth interviews during the assessment of the programme or strategy. They relate direct outcomes and field effects with the expected objectives and effects of the programme. The construction of the diagrams should therefore constitute one of the first tasks of the evaluation.

The objectives diagram illustrates the objectives classification, from the global objective to detailed operational objectives.

The effect diagram displays the classification of the results, outcomes and effects of what is intended from the implementation of the objectives system.

Figure 1: Rationale of the objectives diagram and the effect diagram

What are the possible uses of these diagrams?

In the evaluation context, diagrams are used to reconstruct the intervention logic of the European Commission's support to the country. This reconstructed logic will be shaped into one or more logical diagrams of effect. Prior to the preparation of effect diagram(s), the team will have to prioritise the stated cooperation objectives and translate these into intended effects. These intended effects will form the "boxes" of the diagram(s). Possible "holes" in the intervention logic will be indicated and filled on the basis of assumptions to be validated by the reference group. The effect diagram(s) will help to identify the main evaluation questions.

Figure 2: example of an objectives diagram

The faithfully reconstructed objectives diagram displays the objectives system and provides the evaluator with a first approach to the strategy and policies inner quality. Indeed, an unclear, incomplete or incoherent diagram means a lack of relevance in the resulting planning or a lack of faithfulness to the initial objectives system.

During the implementation of the strategy or programme, external events, such as the evolution of the world commodities market price, elections, political changes, or the conclusion of international agreements can influence the achievement of the objectives and expected outcomes. Comments dealing with the discrepancies between the expected and the observed outcomes should take these events into account. A would-be coherence
Diagrams establish logical links between objectives or effects. Each objective or effect is presented as logically dependent on a higher row objective or effect. The outputs of the activities implemented or scheduled by the programme appear as a contributions to the overall objective/effect and support the coherence of the objectives and effect system.

A strategy or a programme seldom address the full scope of the overall objective, which is limited to choices for each row of objectives or effects. Unless the evaluation can find an explicit explanation of the choices made in strategic documentation, it must provide an answer to series of questions:

• Do these choices represent the only possibilities?
• Are the objectives the most relevant ones for the row 'x'?
• What external factors have led to this selection?
• What reasons have led to the elimination of certain objectives?

How are they constructed?

After determining the evaluation scope, the evaluators should construct a diagram illustrating the objectives presented in the strategy and planning documents. The objectives diagram is drawn from this diagram. When the evaluation scope covers one or more strategy papers (geographical) or strategic policies (sector-based, thematic), it is recommended that one diagram per document is created (unless there is a logical continuity in the strategy or the policy). When a logical framework has supported the drafting of a programme, it clearly states the various levels of objectives targeted by the programme. The logical framework is consequently a reference point for the evaluation, as a consequence of the presentation of the objectives diagram's basic constituents. Although the establishment of result-based logical frameworks has not yet been generalised, the effect diagram can usually be deduced from the logical framework of the intervention's objectives. The objectives diagram and effect diagram can also be used in evaluations for projects and programmes whose rationale is not explicit in the logical framework. In practice, the objectives and intended effects of complex policies and strategies often lack explicit presentation and logical structure, whereas the justification for an evaluation is to be able to answer the following question:

• Have the objectives or intended effects of the policy or the strategy under assessment been achieved?

What are the preparation stages for the construction of the diagrams?

Stage 1: delimit the evaluation scope

The terms of reference include information about the timeframe and the financial tools to be assessed. Note that projects and programmes originating from previous documentation can also be implemented during this timeframe. 

Stage 2: identify the objectives and effects

Collect the documentation required for the establishment of the diagrams. It should comprise:

• Policies, strategies and programmes baseline documentation
• Complementary official documentation

An overall analysis of the break points in the global strategy should be undertaken, without including the objectives of the projects and programmes in progress and scheduled for a different timeframe from the one assessed.

Establish a list of the objectives recorded in this documentation. 

Stage 3: construct a faithfully reconstructed objectives diagram

A provisional classification could be carried out on the basis of the distinction between three levels of objectives:

• Higher-order objectives (goals)
• Short-term and medium-term intermediate objectives
• Operational objectives

La définition des relations logiques occupe une place centrale dans le classement. Elle est avant tout affaire d'expérience. 

Stage 4: convert the faithfully reconstructed objectives diagram into a logically reconstructed objectives diagram

Some faithfully reconstructed objectives diagrams may reveal logical defects in strategy or political papers, such as:

• Strategy or political papers involving a range of objectives without sufficient details to classify them
• No definition of the higher-order objective (goals)
• Weak relevance of the causal links between objectives
• The objectives of a given row do not result in any objectives

In order to prepare a comprehensive and coherent objectives diagram, the evaluation team will need all available documentation, its own expertise and, if required, that of experts.

Each of these rationalisation operations should be explained in a technical note. 

Stage 5: construct the effect diagram

Objectives diagrams and intended effect diagrams share the same rules of construction. The effect diagram is constructed from the conversion of each of the objectives into the corresponding intended effect:

• The higher-order objective corresponds to the higher-order effect
• Intermediate objectives correspond to intermediate effects or outcomes
• Operational objectives correspond to results

Computer devices

Most of the graphics can be addressed with software such as MS PowerPoint.

How are the findings presented?

Objectives diagrams are established during the organisation stage, where reports and notes should be provided. At this stage, the diagram's construction must be precisely described.

For the faithfully reconstructed objectives diagram, the sources of the objectives/effects (quotations, references to the original documentation) must be provided. References to documentation, interviews and expertise must support the objectives' location in the diagram, and the assumptions developed during the construction of the diagram must be explained.

Figure 3 : example of the EC strategy in Tanzania

The process through which the logically reconstructed objectives diagram has been extracted from the faithfully reconstructed objectives diagram must also be clearly explained. 

Verbal presentations

The evaluation team will need to present its work (methodology and findings) to different types of people (the evaluation reference group, participants in debriefing sessions). The objectives diagram and/or the effect diagram are very efficient tools for this purpose, providing that they are readable without being over-simplistic.

To do so, a main diagram, and several sub-diagrams developing fundamental sections of the main diagram, should be presented, each of them not exceeding 20 items (boxes).

What are the preconditions for its use?

What are the advantages and limitations of the tool?

SUMMARY

.

Why is this tool used in evaluation?

Strategies result from a procedure, taking into account global objectives, contextual elements as well as the position of the stakeholders. This procedure is usually empirical. The decision diagram sketches the strategy drafting process, particularly for the selection of key information, the participation of stakeholders in the process, and the management of the implementation arrangements. It highlights the choices that have been made when the strategy was elaborated as well as the consequences of the selected objectives and their forecast impact. The decision diagram is a useful complement to the objectives diagram

The documents which present co-operation strategies usually display a range of objectives among which one or occasionally more global objectives, operational objectives relative to development assistance projects, and a range of intermediary objectives at various levels can be identified. The selection carried out by the authors of the strategic and programming documents depends on various sources:

• Recommendations of the European institutions (Council, Parliament, Commission)
• Analyses of the external events (major events, country situation)
• Intervention of non-European Commission actors (partner governments, Member States, other donors)
• Lessons learned in previous programmes or projects

The aim of the decision diagram is to describe the impacts of such orientations, contextual data and analyses. Indeed, each box of the diagram's central column represents a choice (selected and rejected objectives), while the boxes on each side illustrate the flow of inputs which represents the external justification for these choices. 

What are the possible uses of these diagrams?

The decision diagram highlights:

• The range of options for the establishment of the objectives system (overall objective, selected and rejected intermediary objectives, etc.)
• The external events influencing the decision-making

The decision diagram facilitates the analysis of the strategy in terms of internal coherence (logical succession of the choices) and external relevance (contextual elements and position of the stakeholders).

When the terms of reference of an evaluation require an analysis of the partnership, the diagram is used to highlight the intervention of the main partners (governments, Member States and other donors) in the strategy design, the establishment of the programmes and the selection of the projects.

The diagram can perform the same role for the analysis of the 3 Cs (Coherence, Co-ordination, Complementarity).

How is the decision diagram constructed?

Figure 1 - Steps involved in the decision diagram's construction

The drafting process of the decision diagram continues in two steps:

• Determination of the points at which the decision-making took place (selection of the objectives)
• Identifying, collecting and analysing of the relevant information

What are the preparation stages for the construction of the diagrams?

Stage 1: determination of the points when the decisions were made

Figure 2 - Sketch of the moment of the decision-making: example of first row intermediary objectives

Usually, the establishment of the decision diagram follows the construction of the objectives diagram. The objectives diagram is the basic tool for identifying the points at which the decision-making takes place. Apart from the definition of the overall objective, each intersection of the diagram represents a decision-making point. It stimulates the following questions: Why have these objectives been selected? Why have others been rejected?

Stage 2: drafting of questions

The questions focus on the justification of the selection of objectives. The identification of the rejected objectives helps the accuracy of the wording of questions. How are these objectives identified? At each decision-making point, the evaluator may encounter four situations:

• Strategy and planning papers explain why certain objectives have been rejected and identify reasons to support the decision.
• Working papers (interim versions, notes and various correspondences) provide elements to pinpoint rejected objectives and may justify the choices made (in principle, they always include the positions of Member States and the response of the European Commission).
• The documentation available does not explain the choices made, but interviews with the decision-makers provide relevant information.
• The evaluators have not collected any written or verbal information relevant to the explanation of the choices made.

In the first three situations, the evaluators should investigate which of the objectives were planned but eventually rejected, and the reasons for rejection. In the last situation, would-be objectives should be identified.

Stage 3: collection of useful information

Before the construction of the diagram, the quality of the sources of information should be checked. If this verification reveals insufficient sources, the construction of the diagram should be abandoned.

Usually, written information can be found in four types of documents:

• Strategy papers and programmes can provide information about the lessons learned from previous implementations, the political, economic and social context, and the interventions of Member States and the main donors.
• The preparatory documentation (meeting reports, notes at the end of preparatory missions, correspondence, internal notes or notes between services) may provide explanations about the priorities chosen.
• Evaluations can be valuable in terms of lessons learned.
• The European Council and Commission's more general documents (conclusions, recommendations, reports), as well as the international treaties and agreements often display the contextual elements, lessons learned and priorities which are known to the designers of strategies and programmes.

The evaluation team should formulate its conclusions about the quality of the sources in terms of quantity, relevance, reliability and accessibility. This judgement is presented to the managers, who take the final decision.

Stage 4: constructing the temporary diagram

Preliminary selection of the relevant information is carried out on the basis of the objectives diagram. It requires:

• The establishment of one or more timelines, describing the successive events and/or information included in the drafting process of the strategy and the programme
• The selection of texts from the documentation collected, relative to the choice of the objectives and the scheduled assistance process, or relative to the factors influencing such choices
• The classification and the construction of an index for the texts

Provisional answers to the evaluation questions are formulated on the basis of the information collected. Some of the questions may not be answered at this stage.

The objectives diagram supports the decisions chain. It identifies four (sometimes five) decision-making points dealing with:

• The global objective
• The first row intermediary objectives
• The secondary row intermediary objectives
• Operational objectives

The drafting of strategies and programmes is not strictly and exclusively driven by such a rationale. Implicitly or explicitly, the designers of the strategy and programme begin with an overall objective. Thereafter, they examine which means at the disposal of the European Commission are able to achieve this objective. The successive choices can be synthesised in two points:

• The choice of the global objective
• The choice of the fields and the intervention processes

At the left of the sketch, a magnifier explains the decision's outcome:

• Selected objectives
• Rejected objectives.

A global fishbone shaped diagram is thus completed through detailed diagrams corresponding to each decision-making point.

Figure 3: Decision diagram of the Commission's strategy drafting in Tanzania 2000-2007

Stage 5: testing of the temporary diagram

The decisions and their explanation must be confirmed by the main actors responsible for the drafting of the strategy and the programming, including the European Commission's services (head office and delegations), the representatives of the other stakeholders (Member States, NGOs, etc.), the usual interlocutors in beneficiary countries and/or their government.

Stage 6: constructing the final diagram

The process of testing of the temporary diagram may question some of its parts when the justifications do not illustrate the real strategic and programming drafting process.

In this case, the information should be reviewed and augmented by another consultation round. The final and temporary diagrams have the same shape (a main diagram and the sketch of the point of decision-making). The final diagram includes an explanatory table about the analysis of the information collected.

What are the preconditions for the diagrams use?

What are the advantages and limitations of the tool?