This section is structured as follows:
- Why and when is a cost-effectiveness analysis undertaken
- How is a cost-effectiveness analysis undertaken
What is a cost-effectiveness analysis?
A comparative methodology for economic evaluations
Cost-effectiveness analysis identifies the economically most efficient way to fulfil an objective. Focused on the targeted major result of the activity(for example, the number of jobs created), the tool associates the effectiveness of a programme with its cost.
The costs are the expenses planned for the implementation of a programme. The effectiveness of an evaluation lies in the relationship results and objectives. The objective of cost-effectiveness analysis is to estimate the cost of the result's implementation. If we take the above example, the analysis estimates the cost of each job generated by a specific measure.
A comparative tool
The tool compares policies, programmes, or projects (for example, the costs of different programmes with similar impacts - which requires the identification of situations sufficiently similar to allow relevant comparisons). Conversely, in activities with similar costs, results and outcomes are compared. The tool presents alternatives, in order to identify the most appropriate one to achieve an result at least cost.
Fields of application
Cost-effectiveness analysis is especially used in the context of the assistance to decision-making in public policies and programmes, and in public and private investments. In fields such as health, education, environment, employment, and road safety, cost-effectiveness analysis is frequently used to compare complex policies, although the tool is appropriate for other fields.
Its implementation may be challenging, particularly when an activity yields different types of outcomes at the same time.
Analysis of activities
The tool is generally more appropriate for the analysis of clearly defined activities (in which costs and impacts are easily identifiable), than for the collection of information supporting the analysis of activities, whose sources are varied. The tool is not suitable for the analysis of multiple impact activities, whose effects are varied and difficult to rank.
What are the specificities of cost-effectiveness alalysis?
Its place among economic evaluation tools
Cost-effectiveness analysis belongs to the economic analytical tools which assess per se or comparatively the effectiveness or efficiency of a policy or a programme.
Various economic analysis tools available for comparing intervention programmes
In addition to these tools, some authors include cost-feasibility analysis which assesses the feasibility of a policy with the estimation of the costs it generates.
Effectiveness measured with a single outcome
In order to measure the effectiveness of a policy or a programme, the tool focuses on the main expected impact of the intervention. The analysis does not consider other impacts of the policy or programme, whether they are targeted impacts or not.
Because the tool focuses on a single dimension, the effectiveness of the intervention's results can easily be measured. This is appropriate for analyses dealing with an intervention whose primary objective and results are clearly identified, or obvious. Conversely, the tool may be inappropriate or too partial for broad scope interventions, whose implementation is characterised by various impacts (causing various and concurrent outcomes).
The determination of the single variable is one of the difficulties with the analysis.
Determination of a relevant outcome
A measure based on a physical basis, not a monetary one
The other specificity of cost-effectiveness analysis is to measure the effectiveness with indicators highlighting results and outcomes. Contrary to cost-advantage analysis, cost-effectiveness analysis does not evaluate the monetary value of the outcomes. Thus, it is an economic (and not a financial) analysis methodology.
The analysis yields the unit costs of a result. For example, a heath programme focusing on the decreasing number of ill people:
Which questions can a cost-effectiveness analysis answer?
Cost-effectiveness analysis may contribute to answers to the following questions:
What are the possible uses of cost-effectiveness analysis in evaluations?
In the context of a co-operation assistance policy evaluation, cost-effectiveness analysis can be used as an evaluation tool dedicated to projects, programmes or sectors. It is an economic tool which analyses operational objectives at various levels. Conversely, it is not adapted to country strategy analyses, except when the strategy focuses on a single objective, whose implementation targets a clearly identifiable result.
Thus, this tool can only contribute to the analysis of the coherence of the assistance.
Possible uses of cost-effectiveness analysis depending on the type of evaluation
At the start of the evaluation
It can be useful to choose the programmes or projects which will be the subject of a cost-effectiveness analysis at the start of the evaluation. Indeed, selecting the programmes with prioritised co-operation objectives, and impacts which are measured in homogeneous units, avoids making distracting analyses of efficiency and impact during the evaluation.
This selection, when conforming with the preconditions of cost-effectiveness analysis implementation, eases the identification of the priority result, to which a simple and quantifiable indicator corresponds. Thus, it orients the evaluation on the major questions and priority goals of the co-operation strategy.
An analysis centred on cost-effectiveness considerations should determine the priority goals of co-operation strategies and the programmes whose quantitative outcomes can be measured precisely.
To assess the choices in the allocation of resources
The process of drafting and negotiation of national co-operation strategies require the distribution of limited resources between programmes. Various factors influence this distribution, such as questions of sensitivity, acceptability, equity and effectiveness.
In this context, cost-effectiveness analysis is useful for the comparison of resources allocation strategies by comparing the costs and outcomes of the main interventions.
To determine strategy planning priorities
Cost-effectiveness analysis is a valuable tool for defining the strategy planning priorities. It enables decision-makers to determine priorities in relation to the costs and consequences (in terms of simple quantitative results) of the various possible initiatives needed for the achievement of a pre-established objective.
To conduct a complex debate
Determining the priorities among the results, quantifying and homogenising indicators of outcome constitute the crucial stages of a country evaluation.
When the priority results (quantifiable and measurable) are highlighted by cost-effectiveness analysis, the evaluators can debate with decision-makers and other stakeholders on the priorities of the intervention strategies and the expected or achieved key impacts.
In country evaluations, such debates with decision-makers can be as useful for the evaluation as the analysis conclusions.
Within which type of evaluation can it be conducted?
Undertaking cost-effectiveness analysis in the context of an ex ante, intermediary or ex post evaluation generates various methodological challenges and a complex information collection operation during the implementation stage.
In ex ante evaluations
Cost-effectiveness analysis is primarily used to support decision-making. It determines whether a decision to intervene is economically well grounded, or which intervention among others is the most efficient. Thus, this type of analysis is particularly designed for ex ante evaluations, to judge the economic relevance of a measure to be taken.
In ex post evaluations
In ex post evaluations, the analysis can contribute to the measurement of the economic efficiency of an intervention already carried out. For the same intervention, ex post and ex ante analyses can yield different outcomes, for example, because unexpected costs have become necessary for the achievement of the objectives, or because the outcomes are different from the expected results.
In intermediate evaluations
The analysis can be very useful for intermediate evaluations as a means to update the ex ante outcomes and choose which options should be selected for the continuation of the intervention.
What are its advantages and limitations?
An ex ante evaluation tool
Cost-effectiveness analysis is a simple and effective tool which compares different measures or programmes with identical objectives. It is a valuable tool for ex ante evaluations, because it facilitates the selection of various interventions on the basis of their cost.
An educational and communicational tool
Cost-effectiveness analysis summarises the outcomes using a single quantifiable indicator. It expresses in a simple and factual way a single priority objective in an intervention, which has informative and communications value.
Visibility of the intervention's effectiveness
Cost-effectiveness analysis presents the costs of the results of the intervention to beneficiaries and decision-makers, who often appreciate such a simple and factual illustration of the intervention's objectives.
Cost-effectiveness analysis focuses on the main direct outcome of the intervention. Focusing on the main impact means neglecting indirect impacts. Indeed, the analysis compares costs to immediate outcomes, and neglects long-term impacts.
Thus, this analysis is limited to the comparison or the evaluation of "straightforward" outcomes programmes, i.e. programmes which are relatively homogeneous and characterised by a clear and quantifiable priority. Yet, in practice, there are few programmes and interventions centred on a direct and straightforward priority impact.
When an intervention generates secondary and/or indirect results, the use of cost-effectiveness analysis may be irrelevant or counter-productive. Because these impacts are neglected, the costs generated by these impacts may be under-estimated, which can lead to a misinterpretation of the situation.
Analysis of the effectiveness, not the relevance
Cost-effectiveness analysis can determine whether or not an intervention is economically efficient and, more specifically, if it is comparatively more efficient. Conversely, the analysis does not determine whether the intervention's implementation is relevant or appropriate.
Difficulties of the ex post evaluation
In the context of an ex post evaluation, cost-effectiveness analysis requires reliable data on the real costs of the intervention and on its outcomes.
Depending on the diversity of the resources spent during an intervention, collecting the information about the costs required for the implementation of the intervention can be difficult or even impossible. In addition, if the observation of the results was not scheduled and regularly carried out during the monitoring of the programme's implementation, the collection of the information can be complex and approximate.
Whatever the case, ex post situations are more challenging than ex ante situations because the implementation of the intervention generates unexpected costs and impacts. The data collection for these costs and impacts is difficult.
With which tools can it be combined?
With a cost-advantage analysis
The outcomes analysis is not limited to the study of direct positive outcomes. The evaluator should also quantify negative impacts, as well as indirect and multiplier impacts, which are difficult to measure precisely. Yet, an estimation may still be valuable, to give a more accurate picture of the effective outcomes. Cost-effectiveness analysis can produce these estimations.
With a multicriteria analysis
To assess programmes or national (or regional) co-operation strategies characterised by numerous and complex impacts (direct and indirect, positive and negative), the evaluator should use weighting systems on the results. These compound methodologies (such as multicriteria analysis) can be combined with cost-effectiveness analysis or used on its own.
In this context, cost-effectiveness analysis may constitute a useful introduction to multicriteria analysis.
STAGE 1: Define the conditions for its use
Conditions for the implementation of cost-effectiveness analysis
Check the relevance of the analysis to the objectives of the programme
This stage checks whether cost-effectiveness analysis constitutes a relevant analytical tool for the programme under study.
If the outcome of a programme cannot be defined as a priority outcome, or if homogeneous and quantifiable units cannot be determined, the use of cost-effectiveness analysis should be avoided.
Identify the availability and reliability of the data
Before undertaking cost-effectiveness analysis, the availability and reliability of the data relating to the costs and the quantitative indicator on the results and outcomes should be checked. The analysis requires reliable data, i.e.:
Determine the effectiveness criteria and develop the relevant indicator
Selection of the effectiveness criteria
Once the feasibility of cost-effectiveness analysis is confirmed, the choice of the effectiveness criteria depends on the main objective of the intervention.
In an intervention where the objective is clearly determined, the identification of the effectiveness criteria is straightforward. Conversely, when the intervention's objective is broad (for example, an intervention improving the efficiency of basic education), the identification of the intervention's main objective should be discussed (for example, choosing between the reduction of illiteracy, the increase in the average level of the basic knowledge provided in primary school, or an increase in the number of primary school pupils).
The quality of the answers produced by cost-effectiveness analysis depends on the selection of the criteria. This selection could be done, when required, through the participation of stakeholders in a focus group. In a national programme, the selection should be the responsibility of the specialists of the area under study, gathered together in an expert panel.
Example of the selection of effectiveness criteria
Selection of a relevant criteria
The development of criteria depends on several factors. In many countries, the scarce and limited reliability of statistical sources limits the range of choices. Thus, finding means of observation required for the use of an indicator is crucial.
For example, when an intervention aims at improving the effectiveness of the basic education, the effectiveness criteria could be the increase in the average level of the primary school's basic knowledge. This increase can be measured through the evolution of the grades obtained in all the courses followed by the students, or in the two courses deemed as the most important (for example, reading/writing and mathematics), or through the organisation of a single examination for all primary school students.
In most of the interventions, the selection of the indicator can strongly influence the conclusions of the analysis. Thus, it is recommended that this selection should be conducted during a focus group or an expert panel session. The selection of the effectiveness criteria should be conducted likewise.
Although a single focus group is enough to determine the criteria and the indicator, complex cases require more time to determine the criteria and organise a second working session for the indicator selection.
STAGE 2: Evaluate the total cost of the programme
Add direct costs
In the context of the analysis of the development assistance intervention, the simplest and more commonly used methodology consists of adding all the public resources used in the programme. In this type of calculation, only the direct costs invested in the intervention are considered. These costs are often financial: subventions, financial transfers, decreases in taxes, financing of projects and activities, etc.
Direct costs calculation
Examine indirect costs
Indirect costs, such as the costs of human resources required for the management of programmes, indicate the value of civil servants' work in charge of monitoring the programme or intervention. They should be taken into account whenever they constitute an important part of the intervention's costs.
However, estimating indirect costs can be complex and it is recommended that the relative importance of these costs in relation to the intervention's direct costs is assessed first, in order to decide whether the indirect costs are likely to be important for the analysis.
Indirect costs calculation
Examine other types of costs
Cost calculations can include the loss of earnings and benefits due to the fact that public financing have been attributed to a specific objective (this is called the lost opportunity cost).
This methodology for cost estimation is justified when alternatives for the use of resources are clearly identified, which is seldom the case.
Calculation of other costs
STAGE 3: Measure the impact of the programme
This is the most difficult stage of the process.
Preconditions: nature of the information
The study of the programme's impact depends on the nature of the information at the disposal of the evaluator.
Ex ante evaluations
In ex ante evaluations, the evaluator must forecast the quantitative results of the programme. Depending on the complexity of the intervention, the use of simulation techniques may be required.
Ex post evaluations
In ex post evaluations, the evaluator can use empirical techniques if the primary data at his/her disposal are sufficiently numerous and reliable. If not, the evaluator needs to estimate the actual quantitative results from secondary data.
The analysis may be undertaken, as follows:
STAGE 4: Establish a cost-to-effectiveness ratio
The analysis requires stable elements to support the comparison between:
Comparison of programmes with identical outcomes
When the analysis compares different programmes with identical outcomes, the study is straightforward because the chosen parameter is the cost comparison criteria.
Comparison of programmes with identical costs
When, for the same objective, the analysis compares different types of interventions with identical costs, the study becomes challenging.
The comparison between indicators from different quantitative outcomes will be partly supported by qualitative elements. This is the case, for example, when the study compares the number of people cured, and the number of infections averted, with a view to assessing the relative cost-effectiveness of curative and preventive measures in the health field.
Decision-making is usually political. The evaluator uses cost-effectiveness analysis as a decision-making assistance tool designed to facilitate the decision.
Develop and lead the process of thinking
The tool can be used:
Combine cost-effectiveness analysis with multicriteria analysis
Cost-effectiveness analysis is not a sufficient tool to conduct a debate between the representatives of various interests and help them define priorities together. It must be supported by a multicriteria analysis. In this context, cost-effectiveness analysis constitutes the first stage of the multicriteria analysis.