Detailed presentation

 

This section is structured as follows:

 

DEFINITION

.

What is a multi-criteria analysis?

Objectives

Multi-criteria analysis guides decision-making on the basis of common criteria. It is mainly designed to facilitate the understanding and resolution of decision-making issues. It is undertaken to make a comparative assessment between projects or heterogeneous measures. It can therefore be appropriate for evaluations.

In accordance with selected criteria, the analysis helps decision-makers to integrate into a prospective or retrospective framework different opinions about a project, in order to formulate a judgement.

The tool requires the participation of stakeholders (decision-makers, technicians, beneficiaries, etc.) and yields operational advice and recommendations.

Its goal is to suggest a solution through the simplification of the issue, taking into account the preferences of the stakeholders.

Origin

This methodology was first developed in the economic sciences and industrial engineering fields.

The use of multi-criteria analysis, also called "multi-criteria decision-making", quickly spread in the second half of the 1970s and is now considered a scientific tool.  

Main practices

Practices have evolved with the development of the tool. The analysis is now used to assist decision-makers in the resolution of issues which comprise several (and sometimes competing) points of view.

Conditions for its use

  • A set of criteria, allowing assessment of the activities
  • A table showing the performance of the activities for each criteria
  • Aggregated findings, classifying the activities by preference

Multi-criteria analysis may rely on both objective and subjective information.

What are the different types of multi-criteria analysis? 

Four types of methodologies can be developed for the implementation of a multi-criteria analysis.

Without compensation

Methodologies without compensation classify criteria and define binary thresholds for each criterion. They systematically test the activities under study with the selected criteria: if the activity ranks low against the first criterion selected, it is excluded; but if it ranks high, the activity is selected and goes through a test with another criterion, etc.

For example, the evaluation of the candidates' offers to the European Commission's calls for tenders uses the same methodology in its preliminary stages: the criterion for exclusion, then the criterion for selection, etc.

Complete aggregation

In complete aggregation methodologies, a synthesised index aggregates all the criteria. Thus, criteria must be measurable and preferences mathematically rational.

In this type of multi-criteria analysis, compensations can be made and all the choices are comparable. It is used in straightforward and determined situations.

Complete aggregation methodologies include the weighted average, the additive utility (RAU), Goal Programming, the Analytic Hierarchy Process (AHP) and the Multi Attribute Utility Theory (MAUT).

The weighted average is the easiest methodology to carry out. For example, it can analyse the average grade of a student during an examination. Criteria correspond to the subjects, weights to the coefficients, and performances to the grades for each subject.

Partial aggregation

When situations cannot be compared, partial aggregation methodologies can develop a system of preference.

In accordance with an outranking index, the analysis is based on the comparison between activities which are ranked in pairs.

The advantage of these methodologies is their ability to compare complex situations, which are not usually comparable because of the inclusion of very different criteria (objective and subjective).

Partial aggregation methodologies comprise well-known methodologies, such as Electre, Promethée, Oreste, Macbeth, etc.

Local aggregation

Local aggregation methodologies rely on an iterative process based on the decision-makers' preferences.

This type of methodology selects a variant, suggests a series of alternatives, and repeats the process.

These approaches only involve a limited number of activities and require a significant involvement by decision-makers. Among these methodologies are the multiple linear programming, PREFCALC, UTA interactive, etc.

.

WHY AND WHEN SHOULD MULTI-CRITERIA ANALYSIS BE UNDERTAKEN?

.

How can multi-criteria analysis be used? 

 

Planning assistance

Above all, multi-criteria analysis is a tool that assist decision-making. It is often used for planning assistance and in planning-related ex ante analyses.

The analysis compares several choices (roads, choices related to national and regional development, public procurement tenders, etc.), or various measures within a programme.

In evaluations

In ex ante evaluations

In the evaluation field, multi-criteria analysis is usually an ex ante evaluation tool, and is particularly used for the examination of the intervention's strategic choices. It may also be used as a negotiation tool supporting debates about the intervention's strategic choices. 

In ex post evaluations

In ex post evaluations, multi-criteria analysis can contribute to the evaluation of a programme or a policy through the appraisal of its impacts.

How should multi-criteria analysis be undertaken in country evaluations? 

Prerequisites

When the tool is used in challenging situations, where stakeholders have divergent opinions, it requires experienced expertise.

In addition, the time span and the cost of such a high level of analysis may be unsuitable to the timescales and budgets usually agreed for an evaluation.

Thus, in country evaluations where situations are often challenging, multi-criteria analyses should use simple methodologies. The analyses should be limited to the comparison of straightforward activities, and conducted with a limited number of criteria.

Possible uses

In ex ante or intermediary evaluations

In ex ante and intermediary evaluations, multi-criteria analysis can be useful:

  • To evaluate the ability of various activities of a programme to fulfil a given objective (for example, in a programme targeting the improvement of the population's health). This assessment can take place ex ante or during programme implementation, and collect the opinions of decision-makers and beneficiaries about the effectiveness of the activities
  • To structure the views of project or programme managers about on-going activities
  • To discuss the content of the programmes, and the funding of various activities during the drafting of CSPs and NIPs
In ex post evaluations

The use of this tool as an analytical methodology for complex objectives can be instructive in ex post evaluations, where the strategies and stakeholders interests are diverging.

In beneficiary countries, interventions in fields such as poverty alleviation, maintaining security, immigration control, or trade development can benefit from this type of analysis, as long as it is adequately funded.

Advice on its use

In country evaluations, the analysis requires action:

  • To identify the topic of the analysis and to identify or reconstruct the intervention rationale
  • To form a judgement group, composed of informed stakeholders holding different views about the topic
  • To develop a limited number of criteria (see Stage 5: determine judgement criteria) based on the most important components of the programme's objectives
  • To obtain the assistance of these stakeholders during the weighting of criteria (done individually and collectively) and the rating of the performances of activities per criteria (see Stage 7: judgement per criteria)
  • To develop the performance matrix, following the judgement group's conclusions

What are the advantages and limitations of multi-criteria analysis?

Advantages

Find a solution in complex situations

The most important advantage of multi-criteria analysis is its capacity to simplify complex situations. In practice, beyond a certain number of criteria, decision-makers cannot take account of the totality of information into their judgement. Multi-criteria analysis breaks down the components of complex situations and structures them, in order to progressively find a solution in a transparent way. 

Understandable methodology

Even though mathematical or cartographic tools can be complex, the bases on which they choose criteria and rate performance are straightforward, understandable, and drafted by the group in charge of the analysis. Thus, key stakeholders have good visibility over both the process and the progress towards successive decisions. 

Rational methodology

Multi-criteria analysis develops a homogeneous and simultaneous approach of a range of components during the evaluation, which creates good overall understanding of the various components of the analysis. In this sense, it rationalises the decision process. 

Useful negotiation tool for complex debates

Because of its advantages, multi-criteria analysis is often used for the resolution of complex issues, and in conflicting situations, such as in national and regional development.

The simplicity of the process eases the debate and fosters communication between stakeholders. It is a useful negotiation tool for debates among users.

Limitations

Prerequisites

The analysis can only be undertaken when stakeholders agree on a minimum range of points. For example, the multi-criteria analysis of the programme's operational objectives can only be undertaken if stakeholders agree on the overall objective and, if possible, on the programme's specific objective. For example, stakeholders must first agree on the necessity of improving the vehicular traffic before working together on the choices for a road project. 

Lengthy debates

The practical difficulties of choosing the activities or the variants to be studied, to determine comparison criteria, and to produce grading grids, should not be underestimated. The debates organised to settle these crucial points may sometimes be very long and complicated. 

Availability of the data

The lack of reliable data over a period of time sufficient to organise and validate the methodologies can be a handicap. 

Time factor

The time span allocated to the analysis (and its cost) is often the limiting factor in an evaluation context. Indeed, multi-criteria analyses are often based on slow and iterative processes, which may include protracted periods of negotiation.

Technical aspects of the methodology

The technical proficiency required for the effective use of the methodology is self-evident. In addition to the knowledge of computer applications, evaluators should have skills in mathematical concepts and data aggregation methodologies (see stage 8), in order to avoid poorly based conclusions or undertaking the analysis in the absence of a structured process.

Subjective dimension of the analysis

Although multi-criteria analysis rationalises the approach of complex problems, and takes into account objective and subjective data, its opponents nevertheless consider it a subjective tool.

With which tools can multi-criteira analysis be combined?

Although multi-criteria analysis is a self-standing tool, it can be combined with collection and analytical tools, depending on the specific needs of the situation. The following combinations are not exhaustive.

In the evaluation field, these combinations are often limited because of the costs related to the application of these techniques.

Cost-effectiveness analysis

A cost-effectiveness analysis can be undertaken for one of the criteria used in a multi-criteria analysis.

SWOT analysis

Prior to a multi-criteria analysis, a SWOT analysis can be undertaken to support the determination of criteria.

Expert panel

On a very technical topic, an expert panel can assist the negotiation group to improve its judgement on the topic, to participate in discussions on weightings, etc.

Observation tools

Case studies, questionnaires and focus groups provide basic information which can be useful to the negotiation group in charge of the selection and rating of criteria.

What are the preconditions for undertaking a multi-criteria analysis?

The time span

The analysis usually takes time, except for very simple situations or when the goal is to collect opinions retrospectively.

In ex ante decision-making assistance, multi-criteria analyses usually last several months.

Human resources

Apart from very simple cases, multi-criteria analysis requires the participation of several categories of stakeholders. 

The negotiation (or judgement) group and the mediator

Multi-criteria analysis mainly needs the following stakeholders:

  • The negotiation group (in the planning field) or judgement group (in the evaluation field): people who bring their knowledge and perspectives to the topic under study.
  • The mediator who assists the negotiation/judgement group during the various stages of the analysis. He/she will have a good understanding of the methodology and can moderate group debates in the event of strong opposing views
The technical group

Depending on the scale and technical aspect of the mission, other stakeholders can participate the analysis, for example:

  • A technical assistant, who should be expert in the use of computer software needed for the implementation of certain parts of the analysis (for example, multi-criteria analysis software or geographical information systems). He/she should also be responsible for the production of data for non-specialist readers.
  • Experts in charge of the data collection required by the negotiation group to complete the information during the course of the analysis.

Financial resources

Financial resources can vary greatly. In the context of a development assistance evaluation, multi-criteria analysis should be easy to undertake and cost-effective.

However, in ex ante evaluations, and dependent on the allocated budget, the undertaking of multi-criteria analysis can become as difficult as its use in urban development programmes which include planning assistance or decision assistance.

.

HOW SHOULD MULTI-CRITERIA ANALYSIS BE UNDERTAKEN?

.

STAGE 1: Choose the field of application and determine the intervention rationale

Selection of the field of application

In evaluations, multi-criteria analysis is seldom used for the whole range of the topics under study. Thus, the fields where the analysis will be undertaken should first be determined.

Identification of the intervention rationale

Once the evaluation team has defined the field of application, the logical framework of the intervention should be identified or reconstructed. Indeed, as various members will judge competing activities, the intervention rationale must be fully understood by everyone in the team.

STAGE 2: Choose the negotiation/judgement group (s)

Selection of group members

Multi-criteria analysis is based on the rating and preference of members of the negotiation group (in a planning process) or judgement group (in evaluations).

The group must be selected by the evaluation team. This choice is crucial and the members can be selected from two categories:

"Simple" stakeholders

The "simple" stakeholders are the people concerned with the impacts of the programme (for example, beneficiaries or victims). They can also be project technicians, public administration managers, etc. 

Representatives of the "simple" stakeholders

They can be elected representatives (local or professional), leaders of NGOs (such as associations of stakeholders, associations for the environmental protection, consumer protection, for the defence of women's rights), public administration managers, donor representatives, etc.

Members of the group are usually selected from this category.

Skills of group members

Group members should be sufficiently informed about the topic under study and aware that their points of view will need to be presented to the evaluators and debated.

Fair representation of group member's views

The evaluation team must ensure that the various types of stakeholders and all the viewpoints relating to the activities being compared are fairly represented within the group.

Possibility to appoint several groups

Some evaluators appoint two groups: a group of specialists, who rate the activities using judgement criteria (for example, the ability of an activity to have a sustainable outcome), and a group of decision-makers in charge of the weighting of judgement criteria. In country evaluations, this type of specialisation in groups is not wide spread because it is complicated and expensive in resources.

Necessity for an agreement on the overall objective between members

The group will be effective only if its members agree on the intervention's overall objective (for example, to improve the protection of the natural environment of a specific area). If the objective is not agreed unanimously, working on specific objectives will be problematic (for example, to protect forest biodiversity), and even more challenging for operational objectives and activities at the core of these objectives (for example, to promote the certification of sustainable forest management, to create nature reserves, etc.).

STAGE 3: Choose the technical team responsible for supporting the judgement team group

The number of staff appointed to the technical team will depend on the complexity of the study.

The mediator

 

Main mission

The mediator is the key actor in multi-criteria analysis. Its role is crucial because a group cannot usually formulate judgements alone. His /her main tasks are:

  • To focus the analysis scope and clarify the problems and issues under study
  • To assist members with the reconstruction of the logical framework of the project or programme under study
  • To participate in the definition of activities, choices and scenarios which will be incorporated within multi-criteria analysis
  • To assist the group in the determination of judgment criteria and the drafting of weighting rules
  • To assist the group in the rating of activities per criterion
  • To decide, in agreement with the group, to ask experts to carry out additional studies, in order to assist the group in its decision-making
  • To be responsible (if necessary) for the management of the technical assistant whose computer skills should be adapted to the specific multi-criteria analyses
  • To co-ordinate the analysis process by maintaining the group's cohesion and fostering the participation of everyone
Required skills

The mediator should be selected for his/her skill in this type of intervention. Good knowledge of the field under study is an asset. The mediator should be aware of potential bias in the methodology, in order to avoid presenting misleading findings. Also, mathematical operations should not be opaque to the members of the group (for example, different classifications between two activities provided by the weighted sum and the weighted product method).

In country evaluations, the mediator should be familiar with the context of the development assistance.

The technical assistant and the experts

 

Technical assistant

The technical assistant should have a full knowledge of the software required for the undertaking of specific multi-criteria analyses.

Frequently, wide-scope multi-criteria analyses require the use of calculating and even cartographic software packages. The proper use of these tools requires specific knowledge. In addition, he/she should be able to present the findings in a clear and understandable way. 

Experts

The group may need additional information during the analysis to carry on its work (for example, details about the impacts of a specific activity on the environment). In this case, one or more experts specialised in the relevant field will be required.

Their assistance may be limited to providing advice, but they may also be asked to undertake in-depth studies.

STAGE 4: Establish the list of competing activities to be included in the analysis

Selection of the activities to be compared

Depending of the objectives, multi-criteria analysis helps comparison of:

  • Scenarios or potential solutions in a planning or ex ante evaluation (for example, measures to alleviate poverty)
  • Choices for land-use planning (for example, selection of the location of a harbour, layout of a road, etc.)
  • Activities implemented in a programme (for example, advice on various activities scheduled within an education programme)

The choice among these options is crucial and must be understood by all the members of the group. Some options can be presented visually (two different roads layouts drawn on a map), while others are more conceptual (various types of business assistance).

The mediator should therefore make sure that members of the group have fully understood the content of each activity.

Production of a list of activities 

At the end of stage 4, a list of activities, scenarios and choices relevant to the analysis should be produced. 

Definition of the content of the activities

In some methodologies, this list can be completed progressively, through an iterative process of thinking. This technique facilitates the identification of the optimal solution (or, depending on the case, the consensus solution), and progressively describes the content of these activities.

The definition of activities may be long and sophisticated. Yet, in ex ante evaluations, for example, the evaluator often proceeds to the selection of activities of a programme after the definition of the activities.

STAGE 5: Determine judgement criteria

This is the core stage of multi-criteria analysis. It can be very technical and it is defined by specific rules.

Rules for the selection of criteria

Characteristics of the criteria

Basic rules apply to the definition of criteria:

  • Criteria should be defined by rules recognised and accepted by all, prior to the undertaking of the analysis.
  • They should integrate all the points of view expressed by the members of the group (for example, economic interests for some members, environmental ones for others).
  • They should be unique (for example, a particular interest should not be measured by several criteria).
  • They should constitute a coherent whole, resulting in plausible and non-disputable findings.
Example: recruitment advertisement in an enterprise

When an enterprise needs to recruit a employee of a given skill level, the head office publishes an advertisement and uses the following criteria:

  • Minimum grades in key subjects in examinations.
  • Motivation and experience to be evaluated during a professional interview.
  • Level of salary expected by the candidate.

In this example, criteria are easily understandable and acceptable to everyone. They are established prior to the selection, in order to remove the risk of preference or bias from members of the enterprise. The head office ensures that all views about the recruitment process have been taken into account.

Criteria should be unique, although the criteria for the expected salary and the level of experience are likely to be related. They should be coherent: if two candidates obtain the same rating in two criteria out of three, the third criterion should distinguish them without giving rise to complaint.

This voluntary simplistic example illustrates the rationale of the criteria's construction in ordinary situations.

Categories of criteria

Four categories of criteria

Criteria can be very varied. In the usual bibliography of multi-criteria analysis, four categories of criteria are often encountered, which are easily applicable to all varieties of fields:

  • Economic
  • Environmental
  • Social and organisational
  • Legal and political

Technological criteria may be added to these categories, if necessary, such as in reliability studies on specific technologies, and the programme's utility criteria. 

Development assistance criteria

The following are examples for each category of criteria in the field of the development assistance:

  • Economic criteria: cost of the activity; capacity of an activity to create sustainable enterprises and employment, to attract foreign currencies, to alleviate poverty, etc.
  • Environmental criteria: capacity of an activity to improve water quality, to encourage waste recycling, to protect specific species, etc.
  • Social criteria: social acceptability of the activity; acceptability to local customs; impact of the activity on minorities; capacity of an activity to improve the literacy level, maternal health, the ratio girl to boy at school, etc.
  • Legal and political criteria: political risks inherent to an activity; conformity of the activity with the national rules, etc.

How should criteria be constructed?

The mediator often carries out this stage. He/she ensures that the rules for the criteria's selection are respected, and that the proposed system is coherent and comprehensive.

Techniques for the criteria's selection

The techniques are very varied, from brain-storming to a list prepared in advance by the mediator or the technical working parties.

At the end of the study, which may sometimes take a long time, all points of view must be incorporated within the criteria system.

Except where the group does not have a full knowledge of the topic and overlooks important components, the process of the criteria selection is judged to be comprehensive when the group does not have any further complaints. This often happens when each member of the group sees one of his/her preferred activities get the highest score.

Use of criteria categories

The evaluator can refer to the categories of criteria to ensure comprehensive use of the criteria.

Example: criteria selected for the multi-criteria analysis of an urban waste management programme

The following criteria could be chosen for the ex ante evaluation of a series of activities included in an urban waste management programme in a developing country:

  • Economic criteria: cost of the collection and treatment utilities, transports costs, employees and maintenance costs, etc.
  • Environmental criteria: sensitivity to pollution from storage sites, control of the chemical releases, area needed for the project, etc.
  • Social and organisational criteria: necessity to move populations, participation of beneficiaries, necessity to resort to technical assistance, etc.
  • Legal and political criteria: conformity of the future utilities with the country's laws and regulations, local job opportunities, acceptability of the utilities to the local population, etc.

If required, each of the criteria can be divided into sub-criteria. For example, the criterion relating to the participation of beneficiaries can be divided into willingness to pay, willingness to carry out their part of the work as counterparts, etc.

STAGE 6: Determine each criterion's relative weight

Methodology for the weighting of criteria 

Objectives of the weighting

Once the criteria are established, one of the rules in multi-criteria analysis is to weight these criteria, in order to measure their relative importance for the members.

In secondary school, the coefficients allocated to each subject during the evaluation of the students' work is an example of criteria's weighting. When one subject is judged to be more important than another, its weight within the average is higher than the others.

In certain multi-criteria analyses, the criteria's weights are based on a consensus (for example, the tender evaluation grids for public calls for tenders, where the judgement rules determine in advance the weight for each criterion). The judgement group, which in this case is the tenders evaluation panel, only attributes scores per criterion for each offer.

In most cases, the criteria's weight is not the subject of consensus. Each member can give a different weight to each criterion.

This stage can therefore reveal important divergences of opinion between members, similar to the selection criteria stage. 

Weighting coefficients method

The most well-known example of the use of coefficients is the evaluation of students during their school years.

In the simplest cases, the group can be asked to put the criteria on a scale, and allocate coefficients to each of them (for example, each criterion is given a coefficient scoring from 1 to 5, which reflects its weight).

The distribution of a pre-determined total of points between the various criteria by each participant (for example, a total of 20 points to be allocated among 6 criteria) can also be a simple solution. If the weighting coefficients method is adopted, a criterion cannot be given a 0.

Weighting matrix of 6 criteria using coefficients

This matrix shows how members can distribute a maximum of 20 points between various criteria. The right hand column of the matrix shows the level of consensus (or opposition) between participants for a criterion.

"Playing cards" method

In complex situations, various methods have been developed to improve the organisation of the weightings (for example, the Macbeth method, resistance matrix, etc.).

Among these methods, one of them is simple enough to be understood by everyone and sufficiently sophisticated to take account of notions such as equality, preference and net preference. This method, called "playing card", is used as follows:

  • All the criteria are copied on cards such as the ones used in playing cards. Some of them can be blank (without any criterion)
  • Each member of the group receives a series of cards for each criterion and as many blank cards as he/she wishes
  • The rule stipulates that each participant must rank the criteria (usually, in a decreasing order). The method offers two possibilities: the participant can get a draw for several criteria; or insert several blank cards between two criteria, in order to indicate his/her preference between two criteria.
  • Two basic rules must be agreed by everyone at the start: one deals with the total number of the cards accepted (for example, no more than 8 cards is authorised for the classification of the criteria); the other deals with the way to rate criteria (for example, the gap between the best and the worst score should not exceed a multiple of 5).

The outcome is a classification of criteria which precisely weights the notion of preference.

'Playing cards' method

 

In this example, the scores of the various levels reflect the preference gap among criteria: the 8 levels have been given scores from 5 to 1. The participant x sees the criteria C1 and C5 as the most important, but has no preference between them. C7 also has a good score but the participant indicates the difference of preference with the first two criteria by inserting a blank card between them. And so on, until C6, which is the least important criterion for this participant (so much so that he/she separates it from the other criteria with three blank cards).

Establishment of veto, indifference and preference thresholds

 

Definition of veto thresholds

Some criteria may have such importance that they have to be singled out. This is the case for criteria determined by a veto threshold. These thresholds can be absolute or have relative values, intervene alone or in series, etc., depending on the choices made by the group.

Example

An examination can require an average mark from the candidates higher than 10 and marks higher than 5 for each subject to pass the exam. In this example, two veto thresholds intervene in series.

Relative values may be used, for example, in the different costs in two projects or two offers. Whatever the difference of quality between projects, the chosen criterion can be that the cost of the best project should not be higher than double of the cost of the next cheapest project.

Veto threshold imposed by the regulation

Veto thresholds corresponding to regulation rules can also be used. If, for example, in the study of various motorway alternatives, detailed studies undertaken during the process of definition of the route show that one choice destroys a rare biotope, regulation/rules can act as a veto threshold and reject this choice.

Definition of preference and indifference thresholds

Preference and indifference thresholds also need to be defined, especially for long and complex analyses.

Actors without a common understanding about the subjects under study may not apply the rules the same way, which can lead to distortions.

For example, if these thresholds have not been clearly defined in advance, two actors with very similar opinions may rank two activities differently: one may put them at the same level, and the other at different levels, because preference and indifference thresholds had not been sufficiently defined.

For each criterion, the thresholds must be clearly understood by everyone. They are usually determined in relation to the degree of the detail yielded by data measuring the criterion.

Sensitivity analysis

Objectives

When the group has finished constructing the system for the preparation of the analysis, it is important to test its sensitivity. This test examines the impact of modifications to the parameters selected by the group on the findings of the analysis.

Thus, all the rules established by the group can be tested, in order to see, for example, if variations in the rating of performances, in the weighting, in the aggregation of sub-criteria within a criterion, in the definition of a threshold, etc. have an important impact on the analysis findings.

In long and complex analyses, tests for sensibility can be carried out at various stages of the process. 

Usefulness of a software

These tests can be relatively easy to conduct if the analysis is undertaken with a software package which provides the findings of the simulations immediately. 

Advantage of the test

In addition to checking the smooth process of the system, these tests, which should preferably be carried out with the members of the group, are informative and demonstrate the reliability of the system.

STAGE 7: Formulate a judgement per criterion

Once the criteria are established, each member should judge each activity and compare them using the criteria.

Study of the impacts of the activities based on criteria

In ex ante evaluations

In ex ante evaluations, the study is prospective. It can be based on the experts' advice, statistics, or, depending on the complexity of the subject, pre-established models.

At this stage, values based on criteria are given to each activity's impact (for example, the criterion "impact of the activity on road safety" calls for an answer in terms of a decrease in the number of road accidents).

In ex post evaluations

Ex post evaluations permit the same process as ex ante evaluation but using the findings. Sometimes, the statistics are not collected and the evaluator needs to reconstruct them from the experts' advice. 

Example: impact study undertaken by experts

Table of the findings of the impact study on the activities measured with different type of criteria

This evaluation can be quantitative (for example, the number of employments created), as well as qualitative (the answer to the criterion "impact on the employment market" can be presented in a grid rating the answers "weak, fair, or strong").

The evaluation can also be relative. For example, the answer to the criterion "social acceptability of an activity" can only suggest that the activity A is less acceptable than the activity B from the perspective of the stakeholder.

The findings of the impact analysis can therefore have different characteristics, be derived from technical calculations or from the advice of experts.

Activities' rating and judgement per criterion

The rating and judgement of each activity per criterion is the result of the criteria-based study of the activities' impacts.

The group has responsibility for the judgement, whereas the technical assistants are in charge of the study of the impacts, prior to the group's task.

This stage aims at providing each activity with a rating for each criteria. Comparisons between activities and between the opinions of stakeholders for the same activity can be made using this rating. 

Conversion of qualitative judgements into values

If specialised software packages which increases the potentialities of multi-criteria analysis are not used, qualitative judgements need to be converted into values, in order to facilitate the calculation. However, this may impoverish the quality of the information.

Example of the rating of activities

After studying the impact of two activities with the support of 3 criteria (number of jobs created, impact on the labour market, risk of opposition from the population), their conversion into values can cause some difficulties.

The number of jobs created can be organised into a system of rated groups: 1 for no job created, 2 for the creation of 1 to 50 jobs, and so on up to a maximum decided by the group (for example, 5).

The rating relative to the population's opposition is difficult to convert into a value. One way to avoid this difficulty is to sample the opinions of the population about the perceived difference between the two given situations.

If the risk of opposition is stronger for A than for B, a way to rate this situation can be to answer to the question: is it possible to describe the degree of this difference (low, medium, high, very high, etc.)? 

Use of the relevant software

The increasing use of computer tools to assist with multi-criteria analysis facilitates the integration of notions of preference, to which no absolute value can be stated.

STAGE 8: Aggregate judgements

Risks in data coding methods

This crucial component of the analysis is also the most challenging. 

Checking the possibilities for aggregation

After the activities rating per criterion, only evaluators experienced in the use of such methods should carry out mathematical operations to compare activities. Otherwise, the risk of presenting incoherent or non credible outcomes could be great.

Evaluators should first ensure that all the data are understood the same way in terms of preference by the members of the group (for example, the surface area occupied by a building is preferable when it is small). The use of ratings of a criterion based on a defined group (such as the group for the occupied surface area) or on the rank (for example, the activity which comes first in a criterion) can diminish the differences between criteria.

However, the risk of getting unsatisfactory findings is still great. At this stage, it is important to check whether several ways to carry out the operation yield similar or inconsistent findings (such as the difference on the scoring scale of an activity ranking first in a grid and ranking last in another because a parameter has been changed).

Methods for aggregation of judgements
 Weighted sum method

The use of the weighted sum method is appropriate for measuring real comparable values, but it is not adapted to values of different types.

If the weighted sum method is used as a tool to aggregate findings, all types of compensations between criteria are possible.

 Weighted product method

In spite of the mathematical risks, the weighted product method is often used to take account of extreme findings. It includes the components judged to be barely or non-compensatory. For example, in this system, a veto is expressed with a value equal to zero in a criterion.

 Example of the findings' analysis using the aggregation method

The table below illustrates the findings of a simple analysis on 3 activities and 3 criteria graded from 0,05 to 1 by using the weighted sum and weighted product methods.

Table of comparison between the findings of a multi-criteria analysis using the weighted sum and the weighted product methods

 

This table shows that the findings of the analysis depend on the method chosen.

The introduction of the value 0 in the weighted product method would result in the elimination of the activity, and would act as a veto.

Even though these methods are commonly used, this table highlights their relative weakness (such as in the calculation of the students' grades, or in the evaluation of tenders).

Other mathematical approaches can be adopted, with a view to avoiding the bias previously exposed, yet, their use can be very technical and may require the assistance of a specialist, especially to explain any limitations in their use.

Outranking method

Basic concepts to characterise the preference of the members

Specialists in multi-criteria analysis can encounter difficulties when competing activities are barely or not comparable (such as in a health programme, a vaccination action, or an information action).

Consequently, three basic concepts have been developed in the most recent multi-criteria analyses to describe preference:

  • Preference, which clearly indicates the selection of one activity over another made by the stakeholder
  • Indifference, when the stakeholder is not able to sufficiently distinguish two activities
  • Incomparability, which highlights the difficulty for a stakeholder to compare two activities. The notion of outranking is based on the concept that when a stakeholder encounters two non-comparable activities, he/she should decide whether "the activity A is at least as good as, or, not worse than the activity B".
Implementation
Use of software packages

Methods using very precise concepts (essential for the analysis of preferences) depend on software packages dedicated to facilitate the development of the findings, the sensitivity test, the iterative approaches, etc. These software packages also avoid complicated calculations (What are the advantages and limitations of multi-criteria analysis?).

In evaluations

In ex ante evaluations, such tools can be used to support the evaluation of the financial aspect of the programme's planning.

In the evaluations of the external assistance strategies, which are neither ex ante or ex post evaluations, and where timescales are often very short and resources limited, the use of these tools seems problematic.

Construction of a performance matrix

Whatever the methods selected to undertake the calculations and the aggregations, multi-criteria analysis should yield one (or more) performance table(s) summarising the findings per activity in each criterion (and possibly for each stakeholder).

Consensus weighting of criteria

If the study to be undertaken happens in a consensus group (such as in the company's head office) working with criteria of identical weight (such as for the professor grading his/her students), the performance table represents the findings of the multi-criteria analysis.

Performance table for an identical weighting of criteria

Different weighting depending on stakeholders

If multi-criteria analysis is undertaken with a specific weighting for each stakeholder, the performance table must be interpreted, and the evaluator should turn to the group's members to find new elements to support the progress of the analysis. This could be done through:

  • The presentation of the judgements of each member in front of the group, when multi-criteria analysis does not require any negotiation (such as in evaluations).
  • The search for the best solution(s) for the group. Various techniques can be used: elimination of the worst activities (the ones with the lowest rate or the least preference), or selection of the best ones (best preference). These techniques, which are often iterative at this stage, seek consensus.

BIBLIOGRAPHY

  • European Commission: MEANS Collection (1999)
  • B. ROY, D. BOUYSSOU: Aide multicritère à la décision : méthodes et cas
  • L.Y. MAYSTRE, D. BOLLINGER: Aide à la négociation multicritère. Pratique et Conseils, Presses polytechniques et universitaires romandes, 1999.
  • A. SCHARLIG: Décider sur plusieurs critères, panorama de l'aide à la décision multicritère, Presses polytechniques et universitaires romandes, 1990.

In addition, several software packages such as Electre, Prométhée, support multi-criteria analyses.

Author

FC
Former Capacity4dev Member
last update
7 December 2022

More actions