Navigation path

Left navigation

Additional tools

ECA/07/40
Luxembourg, 2007/12/29

Information note by the European Court of Auditors on Special Report No 9/2007 concerning "Evaluating the EU Research and Technological Development (RTD) framework programmes - could the Commission's approach be improved?"[1]

Research and technological development (RTD) is the main driver of scientific and technological progress and innovation. Being a major player in the field of research is important for the European Union and its Member States so that the high living standards of its citizens can be maintained or even further improved. The Treaty establishes the Communities’ role in supporting RTD, stating that "the Community shall have the objective of strengthening the scientific and technological bases of Community industry and encouraging it to become more competitive at international level, while promoting all the research activities deemed necessary".

Within the wide range of policies implemented in the European Union to strengthen innovation and competitiveness, the RTD framework programmes (FPs) are the most important financial instrument contributing to the Lisbon strategy and the Barcelona objective at the Community level. Through the FPs, the Community provides funding to researchers within the European Union, associated countries and beyond. Their budgets have increased significantly over the years, reaching 7 217 million euro per year under FP7 (2007 to 2013). In terms of budgetary appropriations, the FPs represent the largest area of direct centralised management within the Commission. They are implemented jointly by six Directorates-General, the so-called "research DGs".

The overall share of the FPs in total public RTD funding within the European Union and its Member States ranges between 4 % and 5 %. Nevertheless, its impact on what kind of research is carried out within Europe is significantly larger. This is because Community grants generally provide only part of the total funding for a project. In addition, when applying for funding, researchers need to demonstrate that their projects address the objectives specified in the calls for proposals. In this way, the FPs set incentives to orient RTD activities towards scientific and technological objectives agreed at the level of the European Union.

The Court's audit (which was carried out jointly with the EFTA Board of Auditors and the assistance of external experts) covered the evaluation and monitoring arrangements in place at the Commission since 1995 for the last three programming periods (FP4, FP5, FP6), and also gives an outlook for the current FP7 (2007 - 2013).

The audit addressed the basic question of whether the Commission's approach to assessing the results of the FPs was adequate. In this context, the Court checked whether the Commission met the legal requirements and ascertained whether its system for evaluation and monitoring met stakeholder expectations.

In particular, the Court asked whether

  • there was an underlying intervention logic to the FPs and their constituent programmes that could provide a sound basis for evaluation and monitoring;
  • the Commission had defined an evaluation strategy for the FPs;
  • the organisational structure of the Commission's evaluation system for the FPs ensured that evaluation was properly coordinated, adequately resourced and perceived by stakeholders as being sufficiently independent;
  • methodologies used for evaluation were satisfactory; and
  • evaluation and monitoring activities provided information and analysis which could be used to improve on-going programmes and prepare future FPs.

The Court's findings and conclusions were as follows:

  • the lack of an explicit intervention logic and the presence of poorly defined programme objectives and weak performance measurement undermined effective monitoring and evaluation;
  • the absence of a comprehensive evaluation strategy, agreed between the "research DGs" implementing the FPs, resulted in inconsistent approaches between the different Commission services. In particular, this is illustrated by an under-evaluation of significant parts of the FPs;
  • evaluation of the FPs was decentralised, the existing mechanisms for coordination among the DGs implementing the FPs were not effective, and the Commission's central services, in particular DG Budget, had no enforcement role. In general terms, however, the "research DGs" did comply with the Commission's formal requirements on evaluation. The use of panels composed of high-level external experts appointed by the Commission for major evaluation exercises was well-suited to the Community context. However, these temporary panels were not involved early enough in the process to influence evidence collection and had only limited possibilities to follow up relevant issues over time. By using only one overall FP expert panel for five-year assessments which covered all aspects and all programmes, insufficient attention was given to specific issues related to the different scientific areas within the FPs;
  • inadequate methodological guidance was provided, evaluators found difficulties in gathering relevant data and there were no evaluation studies that addressed the longer-term outcomes and impacts of the FPs, as opposed to short-term issues related to the monitoring of programme implementation; and
  • as a result of the above, and because the timing of evaluations was often premature, the Commission's evaluation system for the FPs was of limited usefulness to policy-makers, stakeholders or even the Commission itself. Ultimately, little is known about the achievement of programme objectives and the results of the FPs.

The Court recommends that:

  • intervention logic should be rendered explicit in future legislation. Underlying assumptions should be explained, the link between scientific and socio-economic objectives clarified and appropriate performance indicators developed;
  • a comprehensive evaluation strategy should be developed by (and agreed between) the "research DGs" implementing the FPs. In particular, this should entail a consistent approach with regard to the minimum level at which detailed evaluation must take place so as to take account of the specificities of each scientific field;
  • consideration should be given to setting up a joint evaluation office for co-ordinating the Commissions’ evaluation activities for the FP as a whole and creating a system of panels (and sub-panels) composed of external experts. These panels should be set up sufficiently early in order to provide effective assistance to the Commission, and continue to do so throughout the programming period and thereafter;
  • the data requirements for evaluation and monitoring should be analysed properly and more extensive use should be made of other existing sources of data. The Commission should also rationalise the reporting requirements for participants and, finally, it should draw up a comprehensive evaluation manual for the FPs and develop a broader range of methodologies and techniques to be used for evaluations in the field; and
  • the Commission should establish the type and scope of evaluation that can reasonably be expected for the dates specified in the FP7 legislation and clarify how evaluations can be used to adapt programmes ("learning programmes") and what contribution they can make to policy decisions.

Special Report No 9/2007:

http://eca.europa.eu/portal/pls/portal/docs/1/651520.PDF
European Court of Auditors - External Relations Department
12, rue Alcide De Gasperi - L - 1615 Luxembourg
Tel.: (352) 4398 45410 - Fax: (352) 4398 46430 – e-mail: euraud@eca.europa.eu


European Court of Auditors
Communication and Reports – Press Office
12, rue Alcide De Gasperi - L - 1615 Luxembourg
Tel.: (+352) 4398 45224 - Fax: (+352) 4398 46233 – GSM (+352) 621 55 22 24

e-mail: press@eca.europa.eu


[1] The sole purpose of this Information Note is to provide a summary. The Special Report, as adopted by the Court of Auditors, is available on the Court's website (http://www.eca.europa.eu) and will shortly be published in the C series of the Official Journal of the European Union.


European Court of Auditors
Communication and Reports – Press Office
12, rue Alcide De Gasperi - L - 1615 Luxembourg
Tel.: (+352) 4398 45224 - Fax: (+352) 4398 46233 – GSM (+352) 621 55 22 24

e-mail: press@eca.europa.eu


Side Bar