Data collection

For the purpose of answering questions, the evaluation team collects data that are already available (secondary data) and applies data collection tools with a view to obtaining new information (primary data).

.

USING AVAILABLE DATA

.

What are they?

To avoid duplicating efforts, running up unnecessary costs and tiring the informants, it is recommended wherever possible to rely on existing information (secondary data), such as administrative data, research, previous evaluations, management and monitoring databases, statistics.

This information can be obtained at a lower cost. It can help to provide partial answers to some of the questions asked.

Why is it important?

• Existing (secondary) data are cheaper and quicker to gather than primary data.
• If the evaluation team does not acknowledge relevant existing data, this will undermine the credibility of the whole evaluation.

Main channels for identifying and gathering secondary data

• Managers, implementing agencies, operators and partners
• Experts in the domain under consideration
• The Internet
• Statistical offices and monitoring bodies
• Scientific and professional literature

Reliability problems

Before making use of secondary data, particular attention should be paid to the following points:

• Definitions: do the data measure/reflect what is to be addressed?
• Missing data
• Accuracy: are the sources and measurement method reliable?
• Age: when did measurement actually take place?
• Reliability over time: did definitions or measurement methods change in the period covered?
• Comparability: are definitions and measurement methods consistent from one place to another?
• Aggregating data: are pooled data consistent with one another?
• Disaggregating: is it possible to break down data into sub-territories or sub-sectors while keeping a large enough statistical basis?

Recommendations

• Where possible, multiple sources of evidence should be used so as to follow cross-checking.
• Be aware of vested interests when using secondary data. Those responsible for their compilation may have reasons for showing an optimistic or pessimistic picture. For instance, it has been reported that officials responsible for estimating food shortages exaggerated the figures before sending aid requests to potential donors.

.

MOST FREQUENT DIFFICULTIES WITH DATA COLLECTION

.

What are they? How to cope with them?

Even if the data collection programme has been wisely prepared, the evaluation team often encounters problems during its field work. The most frequent difficulties occur with:

- Access to informants

- Cultural gap

- Lack or weakness of data

Recommendations

• Any evaluation creates a feeling of uncertainty, which makes some stakeholders reluctant to co-operate, if not hostile. In such cases keep a positive attitude, emphasise the intended use of the evaluation, promise impartiality, and focus on facts rather than opinions.
• If an information source is not accessible or if a survey technique is not manageable, change the data collection work plan in order to collect similar information through other sources.
• Pay sustained attention to biases and risks of unreliability. Strive to understand them. Report on them.
• Avoid relying on one single information source in order to facilitate cross-checking at the analysis stage. This will also make it easier to manage if one of the sources cannot be used.

.

RELIABILITY OF COLLECTED DATA

.

What are the risks?

While gathering information, the evaluation team faces various risks of biases which may undermine the reliability of collected data.

Why should biases be considered carefully?

• For improving the reliability of data collection
• For assessing the quality of the evaluation
• For understanding the limitations of conclusions which draw on unreliable data

Most frequent biases

- Confirmation bias

This risk is a threat to all data collection approaches. It results from a tendency to seek out evidence that is consistent with the intervention logic, rather than evidence that could disprove it.

When subject to this bias, the evaluation team and informants tend to focus on intended effects and systematically to overlook external factors, unintended effects, negative effects, interactions with other policies, outside stakeholders, alternative implementation options, etc.

This bias is avoided by relying on independent and professional evaluators.

- Self-censorship

In some instances, informants may be reluctant to freely answer questions, simply because they feel at risk. They tend to rigidly express the views of their institution or their hierarchy.

This bias is combated by guaranteeing confidentiality and anonymity in the treatment of answers. The interviewer should also insist on factual questions and avoid collecting opinions.

- Informants' strategy

Those who have stakes in the intervention may distort the information they provide, with the aim of obtaining evaluation conclusions closer to their views.

This bias will be reduced if the whole range of stakeholders is included in the data collection work plan and if various sources of information are cross-checked.

- Unrepresentative sample

This bias may be a matter of concern if the evaluation team generates quantitative data through a questionnaire survey. It should also be considered when using secondary data obtained from a questionnaire survey.

In this instance, the evaluation team should verify that the sample of surveyed informants is large enough and representative of the population as a whole.

- Question induced answers

This bias and the following ones are frequent in interviews and questionnaires. The way in which questions are asked by interviewers or the interviewer's reaction to answers can generate a bias which is either positive or negative. Even the order of the questions in a questionnaire may change the substance of the answers. This bias will be limited by having questionnaires designed and tested by experienced professionals.

- Empathy bias

Interviewees may not have a pre-determined opinion about the questions put to them. They try to make up their mind in a few seconds when responding to the interviewer or to the questionnaire. While doing so, they may be strongly influenced by the context.

Especially in the case of interviews, the evaluation team has to create a friendly (empathetic) atmosphere, at least for the sake of achieving a high rate of answers and fast completion of the survey.

The combination of the two introduces a systematic positive bias in the answers, which tends to overestimate the benefits of the intervention and to underestimate the role of external factors.

This bias is prevented by relying on properly trained interviewers.

- Sample selection bias

People who agree to be interviewed may not be representative of the overall target audience.
This bias could be controlled by undertaking a special qualitative survey on a few "non-respondents", although this exercise brings additional costs.

Recommendations

• Rely on an independent and professional evaluation team in order to limit confirmation biases.
• Systematically mix positive and negative questions in order to reduce empathy bias and question bias.
• Be highly credible when promising confidentiality and anonymity in order to limit respondents' self-censorship - and keep such promises strictly.
• Never rely on a single category of stakeholder (e.g. programme managers, beneficiaries) in order to reduce strategic bias.