Evaluation methodological approach

Why is this tool used in evaluation?

A survey is an observation tool which quantifies and compares information. Data are collected from a sample of the population targeted by the evaluation.

A survey questionnaire is a schedule of questions collecting information needed for the evaluation. Respondents are not supposed to answer the essential issue under investigation: in a good questionnaire, questions derive from the evaluation questions and are sufficiently basic for the respondent to answer them.

Questionnaires often combine both types, with a preference for structured items and a few open-ended questions (yielding information which is more diverse and/or precise, but less amenable to statistical analysis). 

Structured questionnaires

Structured items are questions which respondents must answer in a specific way by choosing from a limited and predetermined set of answers. The questionnaire format is designed to obtain information about facts, to find out whether respondents agree to a suggestion, to record their opinions on a set of assertions, etc. 

Open-ended questionnaires

In open-ended questionnaires, respondents answer a precise question and interviewers take notes. Thus, open-ended questionnaires are similar to structured interviews, as open-ended items allow a variety of approaches and depth in response.

What use can be made of the survey in evaluation country?

A survey is the best tool for collecting information from the population, and to compare and quantify the various opinions expressed. Its use is particularly relevant to know the final beneficiaries' degree of satisfaction concerning a policy.

Structured questionnaires have the advantage of allowing a cost-effective statistical analysis. Yet, they can be unfitted during the survey's implementation course when the evaluator needs to refine some questions.

Through a daily check with the interviewers, the evaluator can decide to develop or add questions during the interviews, with a view to undertake a more precise analysis.

How is a survey carried out?

Figure 1 : Steps involved in the use of survey

An efficient time management is a prerequisite for the survey. During the tool's testing mission in Benin, the local partner was identified a month and a half prior to the field mission; the sample and questionnaire design, and interviewers training were organised on-site 2 weeks before the arrival of the testing team.

How is the questionnaire developed?

Relevant questions for the evaluators

• What is required?
• Is each question strictly necessary?
• Will a single question be sufficient to obtain this information?
• Is the respondent in a position to answer the question?
• Will the respondent provide the evaluator with accurate information?

Structuration of the questionnaire

• Define the investigation's topic and design the questionnaire precisely
• Ask overlapping questions in order to check the relevance and coherence of the answers
• Formulate clear and short questions for the respondents; incite them to take sides

How is the survey carried out?

Design the samples

The methodology selected depends on the determination of the population that constitutes the target group of the survey. This determination is linked to:

• The purpose of the survey
• The working hypotheses selected
• The nature of the available documentation
• The evaluation constraints
• The degree of homogeneity of the population
• The scope of the area to be surveyed

Various types of sampling can be developed: simple (random sampling), stratified, cluster sampling, probability proportional sampling, progressive, etc.

Questionnaires among households during the testing mission in Benin:

" The method to collect data is the itinerary method with which the interviewer can identify the survey's targets by counting the households while covering each street of the scoped area, register them in a household counting slip, proceed to the selection of the household samples and interview the head of family an/or his wife.

The counting should start from the chief of the village's house and progress clockwise, segment after segment, so as to cover the whole village and reach the number of households to be surveyed.

The number determining the first household to be surveyed is random and given by the survey's centre. The numbers of the other households to survey will be determined by the drawing number established by the survey's centre.

For example: if the first number is 3 and the drawing number is 5, the first household to be interviewed will be the third of the survey's list. The other households will therefore have the number 8, 13, 18, 23, etc. "

Conduct a pilot questionnaire

A good quality survey relies on the clarity of the question wording, the ease of response, the questionnaire's length and flow, problems encountered by the interviewers, etc.

How is the questionnaire conducted?

The choice of a specific type of survey depends mainly on the context:

• In developed countries, questionnaires by telephone, email or Internet are increasingly conducted
• In developing countries, face-to-face questionnaires remain the surest tool to obtain the information required. However, other types of questionnaire or tools may be used, dependant on the stakeholders' categories (for example, local agents for development)

The evaluator should keep in mind that, because of linguistic reasons (mostly with final beneficiaries) and time spent in the preparation of the survey, a local partner is strongly advised. The latter should be able to provide the evaluator with human resources - interviewers, statisticians, demographers, etc. - and material resources - transportation, IT, demographic data, etc. - all of which the evaluator may not have available on-site.

How are the findings analysed and treated?
 

• Coding questionnaires: assign a code value to each response, which forms the input for data processing. Responses are gathered in relevant categories.
• Counting findings and correlations: always refer to the working hypotheses.
• Analysing findings: in the evaluation field, simple treatments are often sufficient for the analysis stage, even though establishing correlations between items enriches the analysis.

The analysis of structured questionnaires may not yield the expected results. During the testing mission in Benin, although the local partner was equipped for statistical analysis, the analysis was disappointing and less thorough than if it would have been directly managed by the evaluator.

What are the preconditions for its use?

What are the advantages and limitations of the tool?