Pawe ł SAMECKI
European Commissioner responsible for Regional Policy
Evaluation of Cohesion Policy
Sixth European Conference on Evaluation of Cohesion Policy
Warsaw, 30 November 2009
Ladies and Gentlemen,
I would like to welcome you to this sixth European conference on the evaluation of cohesion policy. I would like to welcome those of you who come from Ministries and regions across the EU – who design evaluations and use the results. I would also like to welcome the evaluators. The conference provides us with an opportunity to exchange experiences. Beyond that, however, it is crucial to examine how evaluation can contribute to cohesion policy performance now and in the future.
It is not a coincidence that the Commission is organising this conference in Poland. First, Minister Bieńkowska kindly invited us. But Poland is also the largest beneficiary under cohesion policy, so we were particularly pleased to emphasise how important good evaluations in a flagship country are for the cohesion policy as a whole.
Let me seize this opportunity to congratulate the Polish authorities on successfully managing to establish within a very short period of time one of the most dynamically developing evaluation systems for cohesion policy in the EU. With more that 150 civil servants involved in evaluation at national and regional level, with nearly 450 evaluations undertaken since 2004 – this is a phenomenal effort. The Commission very much appreciates actions taken in Poland in developing evaluation plans, increasing the potential of the evaluation market, strengthening co-operation with academia, organising training seminars, postgraduate studies and annual evaluation conferences as well as promoting the use of evaluation results.
Let me now move to the main part of my speech.
As this conference takes place, the whole budget of the EU is under scrutiny. President Barroso announced that impact assessment and ex-post evaluation would be the cornerstones for policy formulation in the new Commission. This means that decisions concerning the budget for the next financial perspective should be based more on the performance of policies than has ever been the case in the past.
Cohesion policy, as the second most important envelope of the EU budget will be in the focus of this debate. I believe that we know more about the effects of our policy than is the case for many other Community policies. But to reply to a simple question: "What is the benefit from the cohesion policy?" we need to know even more and we need to explore to the maximum the opportunities given by modern evaluation techniques.
I see a four-fold role for the evaluation, which should always be combined with monitoring and strategic reporting. First, it should provide evidence on the effects of the policy. It should enhance its performance. It should help formulate lessons from the past so as to improve the new generation of the policy. And finally, it should provide the substance for a debate by European policy-makers.
No doubt we will be asked again the simple question I mentioned. That is why my ambition is that more than any other Community policy, ours should be the one which is recognised as being seriously preoccupied with performance and evidence based policy-making. This is not something the Commission can do alone. It must be a shared task of the Commission and Member States and regions, with expert input from evaluators and the academic community.
Why do we focus today on the methods?
The answer is that experience both in the Member States and the Commission over the last six years, since the last evaluation conference, has taught us much, in particular in what areas we need to improve. Let me mention two of the lessons we have learnt.
First, we now know that trying to evaluate large-scale programmes in their entirety from all angles at one point in time will not deliver credible evidence on effects and impacts. We need to unbundle big programmes for evaluation. Evaluating thematically creates an opportunity to go more in-depth and to adapt methods to intervention areas. It also provides the bricks for meaningful policy discussion. Only with knowledge across all priorities is a synthesis of the overall effect possible.
The unbundling of programmes for the purpose of evaluation gives us a possibility to apply more rigorous evaluation methods, be it case studies, participatory evaluations or quantitative methods based on counterfactual approaches or examples of good practice. In the past we sometimes tried to capture the effects of our complex policy in just one figure. We tried to reply in one sentence to the question I mentioned. Our experience shows that this is not credible. Let me say it clearly: there are no methods or indicators which would provide a fully fledged one-sentence answer to the question – what is the impact of the cohesion policy?
Second, we have learned that we need to identify the data to be collected very carefully and early in the process. We need a firm commitment to collect data in view of specific evaluation methods and increased use of more rigorous methods – both quantitative and qualitative.
We must not create unnecessary bureaucracy. We must not create data graveyards. It would be not rational or responsible to have interventions without any assessment. In this context, the necessity for evaluation plans to structure the evaluation effort is clear.
Evaluation as One Input to the Policy Debate
As I mentioned earlier, one of the roles of evaluation is to provide food for a political debate about the results of the cohesion policy. My aspiration would be that already in this programming period we can start a dialogue not just between the Commission and Member State, but a dialogue between Member States about the performance and problems of the operational programmes. This process could start with the strategic reporting of the current period, reporting to the Council every 3 years about progress towards achieving EU priorities of promoting competitiveness and jobs.
In the future, I would see the need for annual political debates in the Council and the European Parliament focused on the effectiveness of the cohesion policy and reasons behind success and failure. Such debates would have the potential to shift attention from the pure degree of financial absorption to performance in the various stages of the policy process. They should allow Member States to identify appropriate solutions and good practice, promote mutual policy learning and increase visibility of cohesion policy.
To this end, I would propose to make the core indicators obligatory and allow for the assessment of their cost-effectiveness. This would facilitate comparisons between Member States and programmes. Core indicators could become an ingredient for peer review through high level political debate. It might also be helpful to make setting annual targets for these indicators.
Incentivising Performance in Cohesion Policy
The more we generate evidence on the performance on the policy, the more insights we gain into not only "what works", but also "what doesn't work" and "what could work an awful lot better". We know there are some weaknesses in the policy. We have questions about what really changes in some of our regions, in our cities, as a result of expenditure of cohesion policy resources. We know that in some regions, structural weaknesses can persist over successive programme periods despite significant injections of Community tax-payers' funds. Why is this so? Insights from our evaluations suggest a need for a more rigorous design and implementation of programmes, with clear ideas of how change will be effected, greater concentration of resources and better project planning.
I am very interested to explore mechanisms which could incentivise the performance focus of the policy. The political debate is one of them. But I am convinced that we should look for best ways to introduce incentives into the policy mechanisms. The issue is quite complex. Evaluation evidence tells us that today the capacity to establish ex ante realistic targets for indicators is still not sufficiently developed to build conditionalities on this basis, in the sense of allocating additional funds or withholding funds. The Commission itself has difficulties to judge if targets have been correctly set, taking into account the diverse contexts in which the policy is implemented. In addition, linking resources to the achievement of targets without being sure that they are realistic, but ambitious can lead to perverse behaviour, such as setting them artificially low. Another challenge is the subsequent "audit" of the achievements reported. This needs to be changed radically before the next programming period. I am convinced that building the capacity to incentivise the performance of the cohesion policy should be a top priority for both the Commission and the Member States in the years to come.
At least for the most important priorities, the Commission and the Member States or regions should agree in advance on what will be achieved by when, with incentives for good performance. What we learn in the current programming period – both in relation to indicators and evaluation approaches – will provide a crucial input to the future design of this aspect of the policy.
We should not of course neglect non-financial incentives. Awards, publication of good practices and public ranking of projects have the potential to enhance the quality of cohesion spending. Improving transparency through the publication of details on final beneficiaries for major projects together with costs and benefits could also result in improved project quality.
Objectives for this Conference
I hope that in the various workshops today and in the discussions tomorrow will help to develop these findings, or indeed challenge them.
It is clear to me that we need to invest again in evaluation methods. If we agree that evidence on the performance of the policy needs to be generated at the level of types of intervention, then clearly we need to reflect and provide guidance on the methods needed for different areas.
What we propose is that following this conference we should bring together a group of evaluation experts to review our guidance and help us to develop it further. I am aware of the wealth of knowledge and experience which is accumulated here in this room, so we also want to invest in exchanges of experience and good practice among those responsible for evaluation across the EU. Directorate General for Regional Policy will hold seminars highlighting and disseminating examples of good practice.
If we succeed in doing all this, we will be able to move away from "spending the money" being perceived as the objective of the policy, as sometimes was the case in practice in the past. This would mean nothing less than a management revolution. From top to bottom, our policy must focus more on achieving its objectives: supporting the development and structural adjustment of regions and Member States. Evaluation can help us in this task, but only if we shift the focus throughout programme design and implementation also towards achieving objectives.
I wish you all a successful conference and, even more importantly, an effective follow-up in the months and years ahead.
Thank you very much for your attention.