Examples

.

This section is structured as follows:

.

EVALUATION AT COUNTRY LEVEL

.

For an evaluation at country level the questions concern, for example:

  • The strategy (and possibly the preceding strategies), its design, its implementation and the relevance of its objectives, taking into account the evolution of the context
  • The focal sectors
  • The integration of cross-cutting issues
  • Coherence, complementarity and co-ordination

The questions below are examples. Many are drawn from recent country evaluations undertaken by the Evaluation Unit.

Questions on relevance

  • How appropriate is the EC strategy as regards the objective of reducing poverty?
  • Have the development problems encountered by the country and their causes been sufficiently analysed and clarified to justify the choice of strategic priorities?
  • Was the strategy sufficiently concentrated and ranked?
  • To what extent has the strategy incorporated lessons learned from the experiences of the EC and other sponsors, concerning factors that facilitate or impede the success of aid in the partner country?

Questions on effectiveness

- Questions on sector-specific effectiveness

This site proposes a method for choosing questions specific to a particular aid sector, based on the analysis of the intervention logic. The method generates the following standard type of question:

  • To what extent has European aid in the sector contributed towards [generating effect X (specific to the sector)]?

These are questions on effectiveness (attaining a specific objective or obtaining an expected impact). Each standard question can be reformulated in many ways:

...by specifying the scope for example:

  • To what extent has European aid in the sector [and more specifically that implemented since devolution] contributed towards generating effect X?

...or by specifying the effect concerned, for example:

  • " To what extent has European aid in the sector contributed towards generating effect X [for the poorest population groups]?

...or by changing the evaluation criterion, for example:

  • Has European aid in the sector helped to generate effect X [sustainably]? (sustainability)
  • Has European aid in the sector helped to generate effect X [with a limited cost compared to …]? (efficiency or cost-effectiveness)

- Generic questions on effectiveness

  • To what extent have the Commission's interventions in the different sectors contributed to political dialogue focused on meeting the needs of the poorest population groups?
  • To what extent have the Commission's interventions in the different sectors helped to improve the management of public funds?
  • To what extent have the Commission's interventions in the different sectors helped to mitigate the negative impacts of economic reforms?
  • To what extent have the Commission's interventions in the different sectors contributed towards fair access to basic services of a high quality?
  • To what extent have the modalities of implementation of the strategy made it possible to meet the needs of the poorest regions as a priority?

Questions on efficiency

Questions on efficiency (also called cost-effectiveness) may be specific to an aid sector and are formulated as follows:

  • " Has European aid in the sector contributed towards generating effect X [with limited costs compared to …]?

It is also useful to ask generic questions on efficiency:

  • To what extent have the Commission's interventions made it possible to produce the expected effects with the lowest transaction costs possible?
  • Is there evidence of greater value added by Community aid in the case of actions that have mobilised the most resources?
  • In what proportion have EC resources been used to achieve the various strategic objectives?

Questions on sustainability

Questions on sustainability are generally specific to an aid sector and are formulated in one of the following two ways:

  • To what extent has the Commission's intervention helped to generate [effect X] in such a way that it lasts after the end of the aid?
  • To what extent has the Commission's intervention helped to generate [effect X] in such a way that it lasts even if the aid is limited [to investments only].

Questions on coherence and complementarity

  • To what extent does aid in sector X (e.g. roads) contribute directly or indirectly towards attainment of the objectives in sector Y (e.g. trade)?
  • To what extent has the design of the strategy taken into account potential interactions and conflicts with other EC policies?
  • To what extent has coordination with bilateral and multilateral interests made it possible to identify those objectives that the EC can meet better than others?
  • To what extent has the Commission contributed towards coordination between the different development partners, and to what extent has this allowed progress on the various roads to poverty reduction?

Questions on cross-cutting issues

The cross-cutting issues mentioned most frequently in recent evaluations are:

  • Gender
  • Environment
  • Good governance (decentralisation, efficient administration that is accountable and oriented towards serving the people)
  • Human rights, the rule of law, democracy, development of the role of civil society
  • Aids

Each of the following questions may concern one or more cross-cutting issues:

  • To what extent has the EC included [cross-cutting issue X] in the implementation of its interventions?
  • To what extent has the EC contributed directly or indirectly towards the development [of cross-cutting issue X]?

Questions on instruments and implementation

  • To what extent was the choice of different instruments and their combination made to generate the expected effects in the context of the partner country?
  • To what extent have the monitoring mechanisms been focused on the results and helped to improve them.
  • To what extent do the administrative procedures of allocating funds and managing actions facilitate or impede the adaptation of aid to beneficiaries needs?
  • To what extent has the implementation [of the new tool "Y' (e.g. financial support for a sector)] significantly improved the chances of obtaining the expected impacts?
  • To what extent has the design and implementation of the interventions allowed good take-up by local partners?

Example of a recent evaluation

The following example presents nine questions chosen in the case of a recent country evaluation (with several important simplifications).

  • To what extent does the Commission's health strategy meet the basic needs of the population and especially of the poorest groups?
  • Is the Commission's strategy in the private sector suited to the needs of firms?
  • To what extent have the Commission's interventions concerning transport helped to improve national and regional trade?
  • To what extent have the Commission's interventions helped to ensure that investments are maintained in the transport sector?
  • To what extent have European interventions helped to reinforce the process of regional and global cooperation and economic integration?
  • To what extent have the Commission's interventions contributed towards the decentralisation process?
  • To what extent has macroeconomic support helped to improve the management of public finances?
  • To what extent does the EC ensure that there is coordination, coherence and complementarity between its interventions and the programmes of other sponsors, especially member States?
  • To what extent is the cross-cutting objective of good governance integrated into all the interventions?

.

EVALUATION AT REGIONAL LEVEL

.

For an evaluation at regional level the questions concern, for example:

  • The strategy (and possibly the preceding strategies), its design and implementation, and the relevance of its objectives, taking into consideration the evolution of its context
  • The instruments of cooperation set up at regional level
  • Support for regional integration.

The questions below are examples. Many are drawn from recent country evaluations undertaken by the Evaluation Unit.

Questions on relevance

  • To what extent is the Commission's strategy suited to the priorities of the region's integration process?
  • To what extent has the Commission identified the needs/problems that justify support at regional rather than national level?
  • To what extent is the Commission's strategy concentrated on the institutions/partners in the best position to deal with regional integration problems?
  • To what extent does the Commission's regional strategy take into account the specific needs of countries in the region?
  • To what extent is the Commission's strategy suited to the changing needs of the region and the countries comprising it?
  • To what extent does the strategy incorporate the experience gained in comparable regions?

Questions on effectiveness

This site proposes a method for choosing questions peculiar to a particular aid sector, based on the analysis of the intervention logic. The method generates the following standard type of question:

  • To what extent does European aid in the sector contribute towards [generating effect X (peculiar to the sector)]?

These are questions on effectiveness (achieving a specific objective or obtaining an expected effect). Each standard question can be formulated in many other ways:

...by specifying the scope for example:

  • To what extent has European aid in the sector [planned in the framework of the regional strategy underway] contributed towards generating effect X?

or by specifying the effect concerned, for example:

  • To what extent has European aid in the sector contributed towards generating effect X [in the least developed countries]?

...or by changing the evaluation criterion, for example:

  • Has European aid in the sector contributed towards generating effect X [sustainably]? (sustainability)
  • Has European aid in the sector contributed towards generating effect X [with limited costs compared to …]? (efficiency or cost-effectiveness)

It is also useful to ask questions on generic effectiveness, that is, cutting across an entire intervention logic, for example:

  • To what extent have the Commission's interventions contributed towards reinforcing the process of regional integration?
  • To what extent have the Commission's interventions contributed towards improving the capacity of regional aid institutions?

Questions on efficiency

Questions on efficiency (also called cost-effectiveness) may be specific to an aid sector and may be in the following form:

  • Has European aid in the sector contributed towards generating [effect X (specific to the sector) with limited costs compared to …]?

It is also useful to ask generic questions on efficiency, for example:

  • To what extent have Commission interventions helped to produce the expected effects with the lowest possible transaction costs?
  • Is there evidence of greater value added by Community aid in the case of actions that have mobilised the most resources?
  • In what proportions have EC resources been used to achieve the various objectives of its regional strategy?

Questions on sustainability

Questions on sustainability are generally specific to an aid sector and are formulated as follows:

  • To what extent has the Commission intervention helped to generate [effect X] in such a way that it will continue after the end of the aid?

Questions on coherence/complementarity

  • To what extent does aid in sector X (e.g. roads) contribute directly or indirectly towards attaining the objectives in sector Y (e.g. trade)?
  • To what extent does the strategy design take into account potential interactions and conflicts with the other EC policies?
  • To what extent is the Commission's strategy at regional level complementary to the strategies defined in each of the countries of the region?

Questions on cross-cutting issues

The cross-cutting issues most frequently mentioned in recent evaluations are:

  • Gender
  • Environment
  • Good governance (decentralisation, efficient administration that is accountable and oriented towards serving the people)
  • Human rights, the rule of law, developing the role of civil society
  • HIV/AIDS

Each of the following questions may concern one or more cross-cutting issues:

  • To what extent has the EC integrated [cross-cutting issue X] into the design and implementation of its interventions?
  • To what extent has the EC contributed directly or indirectly towards the development [of cross-cutting issue X]?

Questions on instruments and implementation

  • To what extent has the choice of different instruments and their combination been appropriate for generating the expected effects?
  • To what extent have the monitoring mechanisms been focused on the results and helped to improve them?
  • To what extent have administrative procedures of allocating funds and managing actions facilitated or impeded progress towards the strategic objectives?
  • To what extent have the design and implementation of interventions allowed good take-up by regional aid institutions?

Example of a recent evaluation

The following example presents the nine questions chosen in the case of a recent evaluation in the MERCOSUR region (with several important simplifications):

  • How consistent is the Commission's strategy or support with the priorities of MERCOSUR's integration process?
  • To what extent does the EC strategy take into account the specific needs of the member States of MERCOSUR and their evolution during the evaluated period?
  • To what extent has the Commission's support helped to enhance MERCOSUR's ability to negotiate an inter-regional partnership agreement effectively?
  • How have the economic and commercial projects helped to reinforce MERCOSUR's integration process?
  • To what extent have the Commission's cooperation procedures (including mechanisms for implementing and granting funding) affected potential results?
  • To what extent has the EC coordinated its strategy and ensured its coherence with its other policies that have an international dimension, especially the environment?
  • To what extent has the European strategy contributed towards the continuation of MERCOSUR?

.

SECTOR OR THEMATIC GLOBAL EVALUATION

.

For an sector or thematic global evaluation the questions concern, for example:

  • The effects peculiar to the sector, and their sustainability
  • The integration of cross-cutting issues
  • The effectiveness or efficiency of instruments peculiar to the sector.

The following questions are examples, many of which are drawn from recent country evaluations undertaken by the Evaluation Unit.

Questions on relevance

  • To what extent are the Commission's interventions adjusted to the specific needs of countries (or regions) [category X]?
  • To what extent does the design of the Commission's interventions make it possible to obtain an appropriate combination of activities and instruments for meeting the objective [X] ?
  • To what extent does the Commission consider the evaluated sector/theme as a priority, and to what extent are these choices justified?
  • To what extent has the Commission's strategy concerning the sector/theme incorporated the lessons learned from other sponsors' experiences?

Questions on effectiveness

This site proposes a method for choosing questions from the analysis of the intervention logic. The method is generally based on the analysis of one or more strategic documents that define a coherent set of objectives. Such documents may not exist in the case of thematic or sector evaluation on a global scale, but that does not preclude the more informal reconstruction of the intervention logic. It is then possible to formulate questions with the following standard form:

  • To what extent did European aid help to [generate effect X (specific to the sector/theme)]?

These are questions on effectiveness (attaining a specific objective or one of the expected effects). Each standard question can be reformulated in many ways:

...by specifying the scope for example:

  • To what extent has European aid [specifically devoted to the sector/theme under evaluation] contributed towards generating [effect X]?
  • To what extent has European aid [implemented according to a particular procedure / using a particular tool] contributed towards generating [effect X]?

...or by specifying the effect concerned, for example:

  • To what extent has European aid contributed towards generating [effect X for the poorest categories of the population]?

or by changing the evaluation criterion, for example:

  • To what extent has European aid contributed towards generating effect X [sustainably]? (sustainability)
  • To what extent has European aid contributed towards generating effect X [with limited costs, compared to …]? (efficiency or cost-effectiveness)

It is also interesting to ask generic effectiveness questions, that is, questions cutting across the entire intervention logic, for example:

  • In which cases has European aid contributed most to developing institutional capacities and a governance suited to needs in the sector?
  • To what extent have the Commission's interventions made countries able to identify their needs and to plan appropriate actions?
  • To what extent have the Commission's interventions allowed it to respond as a priority to the people's needs and especially to those of the most underprivileged groups?

Questions on efficiency

Questions on efficiency (or cost-effectiveness) can be specific to the sector/theme. They are formulated as follows:

  • Has European aid contributed towards generating [effect X] with a cost comparable to that of good practice observed in similar countries?
  • In which countries (and circumstances) has European aid contributed towards generating [effect X] with a particularly favourable cost-effectiveness ratio? What are the reasons for this success?
  • In what proportions have EC resources benefited the various groups/regions concerned?

Questions on sustainability

Questions on sustainability are generally formulated in one of the following two ways:

  • To what extent has the Commission's aid contributed towards generating [effect X] in such a way that it will last after the end of the aid?
  • To what extent has the Commission's intervention contributed towards generating [effect X] in such a way that it will last even if the aid is limited [to investment only]?

Questions on coherence/complementarity

  • To what extent do interventions in the sector contribute directly or indirectly to the objectives of another sector?
  • To what extent do interventions in the sector have mutually reinforcing effects (or contradictory effects) with other EC policies?
  • To what extent are the various EC policies mutually reinforcing (or contradictory) as regards generating [effect X]?
  • To what extent does European aid in the sector take into account potential interactions or conflicts with other European policies concerning the sector?
  • To what extent has coordination with bilateral and multilateral interests allowed more productive dialogue as regards generating [effect X]?
  • To what extent has the Commission contributed towards coordination between the different development partners, and to what extent has this allowed progress on the various paths towards achieving [effect X]?

Questions on cross-cutting issues

The cross-cutting issues mentioned most frequently in recent evaluations are:

  • Gender
  • Environment
  • Good governance (decentralisation, efficient administration that is accountable and oriented towards serving the people)
  • Human rights, the rule of law, democracy, development of the role of civil society
  • HIV/Aids

Each of the following questions can concern one or more cross-cutting issues:

  • To what extent has the EC integrated [cross-cutting issue X] into the design and implementation of its interventions?
  • To what extent has the EC contributed directly or indirectly towards the development [of cross-cutting issue X]?
  • In which countries/circumstances have EC interventions contributed most successfully towards the development [of cross-cutting issue X]?

Questions on instruments and implementation

  • To what extent have interventions in the sector implemented [tool X] and with what effect?
  • To what extent do administrative procedures of allocating funds and managing actions facilitate or impede the achievement of [effect X]?

Example of a recent evaluation

The following example presents the nine questions chosen in the global evaluation of the 'Commerce' sector:

  • To what extent are the Commission's programmes affected by the specific needs of the country/region for the negotiation of future agreements?
  • To what extent are the Commission's programmes affected by the specific needs of the country/region for the benefit of international trade?
  • To what extent do diagnostic and analytical practices in the identification and programming stages of the project cycle make a combination of activities, contributing to a balanced achievement of the objectives, more likely?
  • To what extent do the modalities of implementation (including available instruments and resources) determine the achievement of objectives?
  • To what extent have Commission interventions enabled the countries to strengthen their institutional framework for regional commercial agreements?
  • To what extent have the Commission's programmes made the countries able to identify and articulate their needs and to improve their position in international trade negotiations?
  • In those countries and regions in which the Commission has financed the largest number of interventions for trade, to what extent have agents in the productive sector effectively used the outputs of those interventions?
  • In those countries and regions in which the Commission has financed the largest number of programmes for trade, to what extent have those interventions affected the volume, value and direction of trade?
  • To what extent have EC initiatives helped to counter-balance possible negative effects of trade on human rights, basic social rights and environmental issues?

Author

last update
14 January 2015

More actions