Why is this tool used in evaluation?
The implementation of case study reviews of one or more actual examples, in order to gain an in-depth knowledge of the topic and, if possible, to learn about the entire evaluation programme.
In complex situations, case studies are the preferred evaluation tool when "how" and "why" questions are being posed, They allow a detailed examination of the actual elements in line with the evaluation goals. The purpose of the case study is to provide a picture, which is often more reliable than the outputs from other tools in context of the scarcity of basic data (which is often the case in country evaluations).
|Figure 1- Case study's components
If case studies include the analysis of documents, statistical and implementing data, they are mostly known as a field observation tool and a means to interview people directly involved in the programme, such as the officials and stakeholders.
How is a case study carried out?
What are the conditions for the use of this tool?
To ensure that a case study is credible and yields satisfactory results the geographic evaluations specific context needs to:
- find an effective local partner, who should be neutral to the topic under evaluation and competent both in the theme to be studied and in evaluation techniques
- plan for supervision procedures of the international and local working teams
- keep control of the selection of participants
- have a fair distribution between interviews with officials and beneficiary representatives
|Figure 2 : How is a case study carried out?
Steps involved in case study implementation
How is the instance selection undertaken?
This selection is crucial because an incorrect basis for selecting an instance can lead to a flawed evaluation outcome and can jeopardise its generalisation.
The United States General Accounting Office suggests 3 possible keys for instance selection:
- Purposive samples
Example of the country's selection criteria for the European assistance evaluation in the water sector:
Be included in the list of the country's main beneficiaries of the watsan European assistance
2. Have a representative in every region of the world
3. Have water sanitation as a priority intervention sector
4. Not being influenced by the last evaluations conducted by the European evaluation Unit
How is the case study organised and planned?
A modus operandi defining how to carry out one or more case studies is always useful.
|Figure 3 : A modus operandi defining how to carry out one or more case studies is always useful.
Among other advantages the carrying out a pilot case study:
- validates the methodology with a field test
- determines more precisely the categories of respondents and the basic bibliography
- completes the interview guidelines and the questionnaires
- reviews the questions asked to the evaluators, criteria and indicators, in the light of what is available on-site
- designs a standard report to be followed by the other evaluators
How are the data collected?
To ensure that the case study findings are reliable, a number of fundamental elements should be carefully taken into account:
- The data collection should include adequate longitudinal data, i.e. data covering a sufficiently long period of time in order to avoid taking an anomalous situation as a reference point.
- The data collection should be based on a principle systematically adopted in the evaluation: information verification through triangulation.
- The evaluator must ensure that the information collected is thoroughly used, and that nothing important has been overlooked. It is essential to obtain as much information as possible, especially when opinions differ among the people interviewed.
- By definition, the case study is open to any possible discoveries throughout the course of its implementation. Thus, the evaluator must know how to identify key features during the case study implementation and focus on them, even if they were not expected or scheduled in advance.
- The field stage leads to first-hand observations of what is occurring. The evaluator must also note them down carefully.
How are the results analysed and interpreted?
This is the most challenging stage of the case study. Its goal is to analyse the data that have been collected during the fieldwork and to link as far as possible the effects of the observed facts to their causes. This analysis is difficult to conduct because it is less structured than at the conception and the collection stages.
The analysis overlaps with the data collection stage, and this is particularly true for case studies in which:
- The data collection stage includes a pre-established hypothesis test that may partly modify the study content during its implementation.
- The study is large enough to allow the evaluator to review and refine his criteria for the next data collection as a result of the initial findings.
What are the preconditions for its use?
|Figure 4 : the preconditions for its use
The time span
||Preparation: 15 to 20 working days
Field mission: 3 to 10 working days
Data analysis: 3 to 10 working days
Evaluators' training (in case of multiple sites case studies): 2/3 working days
Analysis and conclusions from multiple sites case studies: 10 to 30 working days.
||Qualified people who fully understand the problems associated with the evaluation, have sufficient experience of interview techniques and speak the language of the people interviewed.
||A minimum budget of €15,000 should be fixed and allocated to the multiple case studies preparation stage.
A budget of at least €5,000 to €7,000 should be planned for each case study, not including long-distance transportation.
What are the advantages and limitations of the tool?
|Figure 5 : The advantages and limitations of the tool
||Its richness, made possible by detailed qualitative information and the context of implementation precisely described.
Its relatively straightforward use.
Its flexibility, making continuous adaptation to various situations possible.
Its implementation stage is compatible with that of a country evaluation.
The opportunity to obtain and understand information at a sufficiently deep level. This tool allows evaluators to become familiar with the logic of action of the various actors.
||The difficulty of identifying the appropriate targets.
The difficulty of identifying cases, setting boundaries and linking them to problems as broad as those commonly addressed in a country evaluation.
The difficulty arising from generalisation to a global level (for example, a country) of themes that were studied at a local level.
The tool's cost.
The fact that this tool rarely allows statistical interpretation of data.
The fact that this tool relies on the judgement of one or more evaluators can lead to partiality, even with the most careful use of case study methods.