Survey

SUMMARY

.

Why is this tool used in evaluation?

A survey is an observation tool which quantifies and compares information. Data are collected from a sample of the population targeted by the evaluation.

A survey questionnaire is a schedule of questions collecting information needed for the evaluation. Respondents are not supposed to answer the essential issue under investigation: in a good questionnaire, questions derive from the evaluation questions and are sufficiently basic for the respondent to answer them.

Questionnaires often combine both types, with a preference for structured items and a few open-ended questions (yielding information which is more diverse and/or precise, but less amenable to statistical analysis). 

Structured questionnaires

Structured items are questions which respondents must answer in a specific way by choosing from a limited and predetermined set of answers. The questionnaire format is designed to obtain information about facts, to find out whether respondents agree to a suggestion, to record their opinions on a set of assertions, etc. 

Open-ended questionnaires

In open-ended questionnaires, respondents answer a precise question and interviewers take notes. Thus, open-ended questionnaires are similar to structured interviews, as open-ended items allow a variety of approaches and depth in response.

What use can be made of the survey in evaluation country?

A survey is the best tool for collecting information from the population, and to compare and quantify the various opinions expressed. Its use is particularly relevant to know the final beneficiaries' degree of satisfaction concerning a policy.

Structured questionnaires have the advantage of allowing a cost-effective statistical analysis. Yet, they can be unfitted during the survey's implementation course when the evaluator needs to refine some questions.

Through a daily check with the interviewers, the evaluator can decide to develop or add questions during the interviews, with a view to undertake a more precise analysis.

How is a survey carried out?

Figure 1 : Steps involved in the use of survey
Figure 2 : What are the conditions for use of the tool in country evaluation?
Concerning the question wording Be familiar with the context and the programme's stakes for the beneficiaries prior to the evaluation.

Be provided with any relevant information to cover the survey's scope.

Concerning the sample design Be provided with the minimum statistical data.

Be sure of the physical and logistical access of respondents.

Concerning the survey's organisation Identify a solid local partner, able to provide for human and logistical resources.

Spend enough time to carry out all the preliminary stages dedicated to the questionnaire and sample design without which the findings may be disappointing.

An efficient time management is a prerequisite for the survey. During the tool's testing mission in Benin, the local partner was identified a month and a half prior to the field mission; the sample and questionnaire design, and interviewers training were organised on-site 2 weeks before the arrival of the testing team.

How is the questionnaire developed?

Relevant questions for the evaluators
  • What is required?
  • Is each question strictly necessary?
  • Will a single question be sufficient to obtain this information?
  • Is the respondent in a position to answer the question?
  • Will the respondent provide the evaluator with accurate information?
Structuration of the questionnaire
  • Define the investigation's topic and design the questionnaire precisely
  • Ask overlapping questions in order to check the relevance and coherence of the answers
  • Formulate clear and short questions for the respondents; incite them to take sides

How is the survey carried out?

Design the samples

The methodology selected depends on the determination of the population that constitutes the target group of the survey. This determination is linked to:

  • The purpose of the survey
  • The working hypotheses selected
  • The nature of the available documentation
  • The evaluation constraints
  • The degree of homogeneity of the population
  • The scope of the area to be surveyed

Various types of sampling can be developed: simple (random sampling), stratified, cluster sampling, probability proportional sampling, progressive, etc.

Questionnaires among households during the testing mission in Benin:

" The method to collect data is the itinerary method with which the interviewer can identify the survey's targets by counting the households while covering each street of the scoped area, register them in a household counting slip, proceed to the selection of the household samples and interview the head of family an/or his wife.

The counting should start from the chief of the village's house and progress clockwise, segment after segment, so as to cover the whole village and reach the number of households to be surveyed.

The number determining the first household to be surveyed is random and given by the survey's centre. The numbers of the other households to survey will be determined by the drawing number established by the survey's centre.

For example: if the first number is 3 and the drawing number is 5, the first household to be interviewed will be the third of the survey's list. The other households will therefore have the number 8, 13, 18, 23, etc. "

Conduct a pilot questionnaire

A good quality survey relies on the clarity of the question wording, the ease of response, the questionnaire's length and flow, problems encountered by the interviewers, etc.

How is the questionnaire conducted?

The choice of a specific type of survey depends mainly on the context:

  • In developed countries, questionnaires by telephone, email or Internet are increasingly conducted
  • In developing countries, face-to-face questionnaires remain the surest tool to obtain the information required. However, other types of questionnaire or tools may be used, dependant on the stakeholders' categories (for example, local agents for development)

The evaluator should keep in mind that, because of linguistic reasons (mostly with final beneficiaries) and time spent in the preparation of the survey, a local partner is strongly advised. The latter should be able to provide the evaluator with human resources - interviewers, statisticians, demographers, etc. - and material resources - transportation, IT, demographic data, etc. - all of which the evaluator may not have available on-site.

How are the findings analysed and treated?
 

  • Coding questionnaires: assign a code value to each response, which forms the input for data processing. Responses are gathered in relevant categories.
  • Counting findings and correlations: always refer to the working hypotheses.
  • Analysing findings: in the evaluation field, simple treatments are often sufficient for the analysis stage, even though establishing correlations between items enriches the analysis.

The analysis of structured questionnaires may not yield the expected results. During the testing mission in Benin, although the local partner was equipped for statistical analysis, the analysis was disappointing and less thorough than if it would have been directly managed by the evaluator.

What are the preconditions for its use?

Figure 3 : the preconditions for its use
The time span Carrying out a survey requires great care at the preparation stage, and an allocation of time in proportion to the importance of the survey, the extent of sampling and field difficulties.

The elaboration of the questionnaire dedicated to specific groups requires sufficient data and hypotheses, which means that the survey cannot take place at the start of the evaluation.

Human resources Where cultural and linguistic specificities are important, it is better to have locally recruited interviewers.

Specialist organisations may sometimes be able to support the evaluator's recruitment process.

The evaluator should organise one or more training/debriefing days for the interviewers.

Financial resources Remuneration of the interviewers.

Transportation expenses.

.

What are the advantages and limitations of the tool?

Figure 4 : The advantages and limitations of the tool
Advantages Yields quantified and reliable data 

Useful to identify changes and make comparisons between opinions 

Enables the evaluator to survey a large number of final beneficiaries 

Enables the evaluator to work on a target population and on a limited scale 

Identifies the outcomes of programmes and policies

Limitations Requires implementation delays exceeding the average time scale of an evaluation mission 

Requires important resources and logistics provided by a reliable local partner 

Requires pre-existing data on the initial situation 

Requires a large number of staff to conduct the survey and analyse the findings 

May present difficulties during the development of representative sampling

Author

FC
Former Capacity4dev Member
last update
7 December 2022

More actions