Why is this tool used in evaluation?
A survey is an observation tool which quantifies and compares information. Data are collected from a sample of the population targeted by the evaluation.
A survey questionnaire is a schedule of questions collecting information needed for the evaluation. Respondents are not supposed to answer the essential issue under investigation: in a good questionnaire, questions derive from the evaluation questions and are sufficiently basic for the respondent to answer them.
Questionnaires often combine both types, with a preference for structured items and a few open-ended questions (yielding information which is more diverse and/or precise, but less amenable to statistical analysis).
Structured items are questions which respondents must answer in a specific way by choosing from a limited and predetermined set of answers. The questionnaire format is designed to obtain information about facts, to find out whether respondents agree to a suggestion, to record their opinions on a set of assertions, etc.
In open-ended questionnaires, respondents answer a precise question and interviewers take notes. Thus, open-ended questionnaires are similar to structured interviews, as open-ended items allow a variety of approaches and depth in response.
What use can be made of the survey in evaluation country?
A survey is the best tool for collecting information from the population, and to compare and quantify the various opinions expressed. Its use is particularly relevant to know the final beneficiaries' degree of satisfaction concerning a policy.
Structured questionnaires have the advantage of allowing a cost-effective statistical analysis. Yet, they can be unfitted during the survey's implementation course when the evaluator needs to refine some questions.
Through a daily check with the interviewers, the evaluator can decide to develop or add questions during the interviews, with a view to undertake a more precise analysis.
How is a survey carried out?
An efficient time management is a prerequisite for the survey. During the tool's testing mission in Benin, the local partner was identified a month and a half prior to the field mission; the sample and questionnaire design, and interviewers training were organised on-site 2 weeks before the arrival of the testing team.
How is the questionnaire developed?
Relevant questions for the evaluators
Structuration of the questionnaire
How is the survey carried out?
Design the samples
The methodology selected depends on the determination of the population that constitutes the target group of the survey. This determination is linked to:
Various types of sampling can be developed: simple (random sampling), stratified, cluster sampling, probability proportional sampling, progressive, etc.
Questionnaires among households during the testing mission in Benin:
" The method to collect data is the itinerary method with which the interviewer can identify the survey's targets by counting the households while covering each street of the scoped area, register them in a household counting slip, proceed to the selection of the household samples and interview the head of family an/or his wife.
The counting should start from the chief of the village's house and progress clockwise, segment after segment, so as to cover the whole village and reach the number of households to be surveyed.
The number determining the first household to be surveyed is random and given by the survey's centre. The numbers of the other households to survey will be determined by the drawing number established by the survey's centre.
For example: if the first number is 3 and the drawing number is 5, the first household to be interviewed will be the third of the survey's list. The other households will therefore have the number 8, 13, 18, 23, etc. "
Conduct a pilot questionnaire
A good quality survey relies on the clarity of the question wording, the ease of response, the questionnaire's length and flow, problems encountered by the interviewers, etc.
How is the questionnaire conducted?
The choice of a specific type of survey depends mainly on the context:
The evaluator should keep in mind that, because of linguistic reasons (mostly with final beneficiaries) and time spent in the preparation of the survey, a local partner is strongly advised. The latter should be able to provide the evaluator with human resources - interviewers, statisticians, demographers, etc. - and material resources - transportation, IT, demographic data, etc. - all of which the evaluator may not have available on-site.
How are the findings analysed and treated?
|Figure 3 : the preconditions for its use|
|The time span||Carrying out a survey requires great care at the preparation stage, and an allocation of time in proportion to the importance of the survey, the extent of sampling and field difficulties.
The elaboration of the questionnaire dedicated to specific groups requires sufficient data and hypotheses, which means that the survey cannot take place at the start of the evaluation.
|Human resources||Where cultural and linguistic specificities are important, it is better to have locally recruited interviewers.
Specialist organisations may sometimes be able to support the evaluator's recruitment process.
The evaluator should organise one or more training/debriefing days for the interviewers.
|Financial resources||Remuneration of the interviewers.
What are the advantages and limitations of the tool?
|Figure 4 : The advantages and limitations of the tool|
|Advantages||Yields quantified and reliable data
Useful to identify changes and make comparisons between opinions
Enables the evaluator to survey a large number of final beneficiaries
Enables the evaluator to work on a target population and on a limited scale
Identifies the outcomes of programmes and policies
|Limitations||Requires implementation delays exceeding the average time scale of an evaluation mission
Requires important resources and logistics provided by a reliable local partner
Requires pre-existing data on the initial situation
Requires a large number of staff to conduct the survey and analyse the findings
May present difficulties during the development of representative sampling