Evaluation methodological approach

WHAT IS A SURVEY?

What are the reasons for using a survey?

A survey collects data over broad populations, by means of a standardised set of questions.
A survey questionnaire is a schedule of questions collecting information needed for the study. It standardises the information collected in survey. Thereafter, this information is used to assess a change in a given situation. The selection of respondents depends on various methodologies aimed at developing a representative sample of the whole segment of the population considered.

Are there different types of questionnaire?

There are two main types of questionnaire:

• Structured questionnaires
• Open-ended questionnaires

Usually, questionnaires combine both types, with a preference for structured items and few open-ended questions (yielding information which is more diverse and/or precise, but less amenable to statistical analysis).

Structured questionnaires

Structured items are questions which respondents must answer in a specific way by choosing from a limited and predetermined set of responses. The questionnaire format is designed to obtain information about facts, to find out whether respondents agree to a suggestion, to record their opinions on a set of assertions, etc. Structured items do not collect detailed information, but as the categories of response are predetermined, they provide the basis for efficient statistical analyses.

Types of questions in structured questionnaires

In structured questionnaires, the respondent may be inclined to respond in a way which does not accurately reflect his opinion, because:

• He/She thinks that his/her answer will please the interviewer or put him/her in a favourable light
• He/She has chosen the 'least worse' answer at his/her disposal
• He/She has been subconsciously attracted by an answer, for example, expressing an average opinion

Open-ended questionnaires

In open-ended questionnaires, respondents respond to a precise question and interviewers take notes. Thus, open-ended questionnaires are similar to structured interviews, as open-ended items allow a variety of approaches and depth in response.

Types of responses in open-ended questionnaires

Limitations of its use

The survey is an outstanding observation tool for the collection of information from a large number of final beneficiaries.

Although structured questionnaires can measure the relative weight of the opinions expressed, they must be conducted with a carefully constructed sample so as to be statistically valid. Thus, facilities and means are required, such as:

• A sufficient number of interviewers to survey a large number of people without whom the collected data would loose its representativeness
• Technical means, such as computer devices to undertake statistic analyses
• Enough time to carry out the survey's three stages (drafting, implementation and analysis). The less the evaluation team knows the context (topic under study and surveyed population), the longer it takes to draft a relevant and useful questionnaire
• Financial resources allocated to the organisation of a long field mission, the recruitment of a interviewer and a team responsible for the data analysis

The use of open-ended questionnaires can be an alternative to structured questionnaires when the evaluation does not need a statistical analysis of the data collected. Past experiences have shown that the information collected by this type of questionnaire eases the understanding and analysis of the population's reactions, and provides answers to the evaluation's requirements.

WHERE DOES THIS TOOL COME FROM?

Its origins

The use of surveys and sampling techniques emerged after the First World War, during electoral forecasting. Polls institutes were the first to use surveys. Nowadays, surveys and sampling techniques represent the social sciences' reference tools. The demands from quantitative sociologists have led to the development of surveys and sampling techniques to be able to benefit from the statistical data.

Its use in other fields

In addition to opinion polls institutes, the marketing field is a significant user of surveys. In anthropology and ethnology, observation surveys are designed as semi-structured interviews.

This type of survey can be used in some case studies, but is seldom used in evaluations due to constraints on time and budget. Indeed, these surveys are more useful for researchers, although their findings may provide the evaluator with insights into the general context and the socio-cultural dynamic of the region under evaluation.

WHEN AND HOW SHOULD A SURVEY BE USED?

In which situations is this tool appropriate?

An observation tool collecting quantitative and qualitative information

The survey is an observation tool which collects:

• Stakeholders' needs, opinions, etc., especially in ex ante evaluations
• Information about expected changes or those which have occurred during the programme. This type of survey is carried out in mid-term or final evaluations.

In developed countries, opinion pollsters and market researchers often conduct surveys because they yield quantitative data, whereas in developing countries, they are less common because of constraints of costs and difficulties in implementation. As a general rule however, surveys remain an appropriate tool to collect information in studies and evaluations.

Surveys require good definitions, and careful management of the information under investigation and of the people from whom the information is to be collected. Surveys should be implemented after a piloting stage.

Theoretically, surveys can provide good indications of change, if carried out with the same sample and within a period of time sufficient to yield information about the impact of a policy or a programme. In practice, however, these conditions seldom prevail in developing countries.

Warning about its use in evaluation

The evaluator will experience fewer difficulties designing a precise set of questions in project and programme evaluation than in country/region evaluation, where the process of identifying real beneficiaries and stakeholders can be difficult.

What type of information does a survey yield?

In evaluation, surveys are mainly used to assess the impact of policies and programmes on various categories of people.
Surveys collect:

• Information about the socio-economic situation of various groups of people, which may support the design of typologies
• Information about changes of situation and practice (provided by economic and social indicators)
• Opinions, judgement and perceptions of a policy or a programme (provided by precise assessment criteria)
• Analyses dealing with the causes of change, and with the impact of policies and programme upon these changes. The analyses start with questions supporting the preliminary hypotheses.

Surveys are carried out:

• Face-to-face with the respondent
• By telephone
• By mail
• By email and on the Internet

What are the advantages and limitations of a survey?

Despite the tool's advantages (especially comparing with other observation tools), limitations in the time span, the financial resources and the technical means can limit its use in evaluations.

Can a survey be combined with other tools?

Structured questionnaires bring complementary quantitative information to other information collection tools (which only yield qualitative data), such as:

• Interviews
• Focus groups

These tools can be used during the exploratory stage of a survey (to identify or detail working hypotheses) to ease the drafting of the questionnaire with:

• The listing of themes to be studied
• The determination of relevant questions

A series of interviews or focus groups can be organised prior to a structured questionnaire, in order to highlight the context of the outcomes, and to support the understanding of the data and its interpretation (for example, to describe a custom, or explain an individual perception).

Surveys' findings also support other tools which are more complex, such as:

• Case studies
• Socio-anthropological observations
• Benchmark studies

WHAT ARE THE PRE-CONDITIONS FOR ITS USE IN EVALUATION?

The time span

Carrying out a survey requires great care at the preparation stage, and an allocation of time in proportion to the importance of the survey, the extent of sampling and field difficulties.
Generally, surveys are conducted after the start of the evaluation, when the evaluator has enough information and when his working hypotheses are sufficiently developed to support a questionnaire dedicated to specific groups of the population.

Human resources

Setting up a survey necessitates the appointment of interviewers.

Required skills for interviewers

• Ability to quickly become familiar with the purpose of the survey
• Ability to establish a relationship based on trust
• Ability to detect possible misunderstandings and impediments
• Ability to complete the questionnaire in a clear and reliable way
• If necessary, speak a dialect and be aware of local customs

The recruitment

• Where cultural and linguistic specificities are important, it is better to have locally recruited interviewers (such as development operators, teachers, technicians, etc.)
• Specialist organisations may sometimes be able to support the evaluator's recruitment process
• The evaluator should organise one or more training/debriefing days for the interviewers. Before the start of the survey, this training should ensure that the survey's objectives and the questionnaire are clearly understood, and a "team spirit" has been established. After implementation, a debriefing on the survey's process, the difficulties encountered, possible bias, respondents comments, etc., should be held. The output of the training will provide the evaluator with insights which will improve the analyses of the findings.

Financial resources

Most of the costs of surveys are remuneration of the interviewers and transportation expenses, if the survey is carried out using widely dispersed interviewees. It is not possible to specify exact amounts because expenses vary greatly with the type of country in which the survey is implemented.

However, the transportation budget is critical in countries where transport is scarce. Generally, in assessing the financial requirements, the following should be taken into account:

• The type of survey which is being implemented (face-to-face, by mail, by telephone, by email)
• The size of the samples
• The remoteness of the area surveyed
• The number of interviewers to be recruited and the duration of their contract, which depends on the elements noted above

Surveys belong to the category of tools which are the easiest to use, allowing the evaluator to obtain information from a large number of people. Surveys by telephone are cheaper than face-to-face surveys. Surveys by mail are the least expensive, although the response rate is usually low and limits information about the respondents profile, especially in countries where computer networks are poorly developed.

WHAT IS THE ROLE OF A SURVEY IN EVALUATION?

Why should it be used in evaluations?

Although open-ended questionnaires require skilled resources for drafting, the organisation of implementation and their analysis, the time spent in data collection on site is limited in most evaluations for financial reasons.

Yet, in certain cases, the evaluation questions require the collection of information from final beneficiaries. In this context, the questionnaire is crucial because it collects information about the viewpoints and practices from a large range of people.

When the completed structured questionnaire is analysed statistically, it quantifies the opinions surveyed, which can be supportive of an impact analysis, or the drafting of evaluation or programme indicators. This type of analysis is however difficult to undertake and requires more resources than in any other analyses.

What are the requirements for the use of a survey?

• The evaluator's knowledge of the context and the programme's impact on the beneficiaries, so as to formulate relevant questions for the questionnaire.
• An available statistical database on the social structure of the population, which should be sufficiently reliable to support the design of samples with a minimum risk of errors.
• The identification of a reliable partner for the recruitment of interviewers, statisticians, and demographers, and the procurement of material resources (transport, computer devices, demographic data, etc.).
• A context allowing for the easy selection and interviewing of a sample of respondents. The evaluator should also have access to respondents, suitable meeting locations, and means of grouping respondents together to maximise the use of time.

WHY IS THE SURVEY USED IN COUNTRY EVALUATIONS?

As surveys need specific conditions for their implementation in country/region evaluations, they have proved to be more appropriate for countries at a high level of development.

In developing countries, however, the evaluator may survey target groups to obtain information (such as quantitative data) which cannot be collected by other means.

In country/region evaluations, surveys have a selective purpose and focus on geographically defined categories of stakeholders. Consequently, the evaluator needs to closely supervise the interviewers, or delegate the supervision to a reliable local partner.

The evaluator can be confronted with continuing problems relating to the collection and speedy selection of the required information from abundant sources. In addition, it may be difficult to reach final beneficiaries. But, by interviewing a large number of beneficiaries, surveys can provide the evaluator with part of the information on the policy under evaluation, such as:

• The progress of programme implementation
• How the needs are taken into account
• Lessons learned
• Impediments, etc.

Currently, surveys and questionnaires are rarely used in country/region evaluations but as surveys can provide useful analyses of impact relating to various groups (and particularly beneficiary groups), the number carried out in the future should increase.

HOW IS A QUESTIONNAIRE DEVELOPED?

What are the relevant questions for the evaluator?

What is required?

As the design of the questions depends on the target population, and on the information, the analyses and opinions which are sought by the evaluator, the questionnaire should start with a list of data about the study's topic, including:

• The purpose
• Working hypotheses
• Target groups
• The type of information which is required
• The geographical setting of this information
• The period of time in which the survey is implemented

This list organises the stages of the survey and the drafting of the implementation timetable.

Is this question strictly necessary?

Asking this implies that the purpose of a study can only be fulfilled if the question is posed. Therefore, if the answer is 'no', the question should not be included in the questionnaire.

Will a single question be sufficient to obtain this information?

Depending on the complexity of the information required, several questions may be necessary, which is particularly important for elements that seek to examine causality.
The evaluator must also be careful about asking questions with two parts. In these cases, the question should be broken into two separate and straightforward constituent parts.

Is the respondent in a position to answer the question?

Three main reasons may account for a lack of response:

• He/She does not understand: the vocabulary is not appropriate, the question is too theoretical, etc.
• He/She does not know: the questionnaire is not adapted to the target group, or the respondent has been poorly selected
• He/She has forgotten: the evaluator should distinguish between the situation where the respondent can find the information and give it later, from the situation where the respondent does not have access to this information

Will the respondent provide the evaluator with accurate information?

This issue mainly deals with information whose nature may be sensitive, such as:

• Socially sensitive topics, in which certain attitudes are likely to be rejected or changed (for example, this can lead to an under/overestimation in the survey)
• Topics closely related to the respondent's private life
• Topics demanding extra effort and time consuming inquiries

In these cases, respondents may be tempted to alter or avoid a topic. The evaluator may anticipate or avoid this reaction by the skilful choice of questions or by structuring the questionnaire.

How should a questionnaire be structured?

The questionnaire's basics

• Determine the topic under investigation and focus the questionnaire's elements on the subject
• Do not be over-ambitious about the questionnaire's capacity to obtain information: it should be short and straightforward, and the wording of questions should be simple and precise
• Allow for overlapping questions, in order to check the reliability and the coherence of the responses

Introductory and transitional documents

These documents are necessary for the clarity and the natural flow of the questionnaire. They must be adapted to the survey's type, its topic, and the socio-cultural characteristics of the sample.
These documents must include:

• "Indications relating to the topic under evaluation: 'the questionnaire's purpose is to determine…, to identify…, we will now study the issue of…, now that we have listed all the relevant points of…'
• Technical indications: 'please tick from the list below any reasons that apply to you, check one box for each row, please write short answers …'
• Polite phrases: 'your opinion is critical…, thank you for your kind collaboration'
• Confidentiality/usage statement: 'this questionnaire is strictly confidential…, this survey will only be used for scientific purpose'
• Short and understandable comments: they should be introduced only when necessary" (extracts from "Surveys, a handbook for practitioners". Jauveau C. 1992. Brussels University Edition)

Sequence of questions

• Place the most sensitive questions at the centre of the questionnaire, to prevent avoidance, fatigue or loss of attention by the respondent if placed at the end of the questionnaire
• Regroup attribute questions to the beginning or the end of the questionnaire. If they are very personal (dealing with income, religious practice or political opinion), place them in the core of the questionnaire.
• Start with easy questions which give respondent confidence
• Insert easier questions between difficult or sensitive questions, in order to make the respondents more comfortable. Their purpose is to facilitate progress through the questionnaire.
• Plan transitional statements between topics which are unrelated
• Group questions dealing with the same topic together
• Rank questions:

• Incorporate non-responses into categories such as "do not know", "no opinion", "refuse to answer". Beware: the respondent may favour these options to avoid responding to the question. The evaluator should determine a coding scheme for this type of response which must be classified as a regular answer.
• Balance open-ended questions and close-ended questions. Open-ended questions provide developed answers, but are difficult to analyse statistically. The evaluator must consider when open-ended questions should be replaced by several close-ended questions.
• Prior to its drafting, plan how the questionnaire will be used: its wording influences its structure (open-ended questions, close-ended questions, correlations between questions, etc.)

What is a typical format for a questionnaire?

Presentation of the interviewer and the survey

This stage, sometimes overlooked, is useful to introduce the questionnaire to respondents.
The evaluator may conceal or slightly change the description of the study's topic if he/she considers that it would otherwise provoke the respondent's negative reaction (such as bias, superficial response, concealment, fear, refusal, etc.). The topic of the study may need to be rephrased into simpler wording, and presented in a less "institutional" format.

Introductory questions

Introductory questions aim at arousing the interest of respondents and establishing a rapport in the case of a face-to-face survey. They convey the general scope of the topic and are formulated to obtain a positive response, even though this response may not be directly useful for the survey.

Background questions

Background questions are used to check whether a respondent has the required information or, if necessary, give the interviewer contact details of an informed respondent. Background questions often take the format of dichotomies.

Warming-up questions

Warming-up questions contribute to progressively focusing the questionnaire on the topic of the study. Usually, they take the form of straightforward questions (for example, behaviour questions).

Specific questions

Specific questions represent the core of the questionnaire: responses to these questions must provide the study with critical and focused information. At this stage, the respondent should be completely at ease with the interviewer and concentrate on the topic. The interviewer may ask him/her personal questions, or press him/her to reflect upon his/her motivation, opinions and judgements.

Attribute questions

With the help of responses to attribute questions, the evaluator describes respondents and incorporates them into sub-groups. These questions yield two types of information:

• Socio-demographical variables (gender, age, profession, etc.)
• psychological information (values, opinions, etc.)

How should a questionnaire be written and presented?

The questionnaire's basics

• Short, straightforward, clear and unambiguous questions
• Question wording must not influence the response, for example by implying a positive or negative answer
• Question wording should take into account possible translation, which requires the quality of the wording to be high

The vocabulary

The style

Ways of introducing the questionnaire to respondents
This stage influences the quality of the survey's findings and should not be overlooked. The most effective way to introduce the questionnaire to each respondent of the sample is to send a letter or an email which should state:

• The nature of the body and the department/service conducting the survey
• The disclosed purpose of the study
• Inducement to participate: interest for the community, research, etc.
• Instructions about how to return the questionnaire if it has been mailed (closing date, contacts, etc.)
• The signature of an official of the body conducting the survey

The cover letter should be carefully prepared, especially its general appearance.

HOW ARE DESK-BASED SURVEY TECHNIQUES CARRIED OUT?

Why are desk-based survey techniques carried out?

The evaluator develops a line of questioning supported by his working hypotheses, with a view to designing the questionnaire. In many cases, pre-field techniques are used to validate these hypotheses and the resulting questions. More generally, they are carried out to improve the questionnaire, prior to the start of the survey.

Techniques for the validation of the questionnaire

Complementary techniques may be used to validate the questionnaire and the whole drafting process. These techniques include:

• Studying the available documentation about the subject under evaluation
• Setting-up a group discussion (a brainstorming meeting) with field test interviewers and, if necessary, people having insights into the subject but not involved in the survey process
• Conducting a focus group, with a view to interviewing competent people about the line of questioning
• Conducting a debriefing session, with a view to collecting critical information for the questionnaire wording

These techniques enable the evaluation team to agree on the same definition of the study's key concepts. This stage is necessary for the questionnaire wording process.

HOW ARE SAMPLES DESIGNED?

The constraints of the population selection

The selected methodology depends on the determination of the population that constitutes the target group of the survey.
This determination is linked to:

• The purpose of the survey
• The selected working hypotheses
• The nature of the available documentation, such as lists, updated or not, etc.
• Constraints such as financial and human resources, delays, need for translation, transportation, etc.
• The degree of homogeneity of the population
• The scope of the area to be surveyed

Basics for the sampling technique

Sample quality is key to the reliability of the findings. The sample should reflect as closely as possible the whole population. It should be a scale model of the population.
To be analysed statistically, a sample should gather at least 100 people. The less the evaluator knows about the population under study, the more numerous the sample should be in order to be representative.

The evaluator's dilemma

In programme or co-operation policy evaluation, samples are often hard to design: registry documents, professional index, etc., relating to the population under study are seldom available, and information about the population's characteristics are often non-existent, unavailable or unreliable. The situations in which the survey is to be conducted are often very difficult.
In this context, the evaluator must implement surveys based on numerous samples, which can be very expensive. Otherwise, he must restrict the survey's field of inquiry to very specific categories of people, which leads to problems of reliability.
The evaluator may conduct several very focused surveys, providing that the analysis of the group of stakeholders has been correctly carried out during the pilot stage.

The various types of sampling

How is the size of the samples determined?

Basics: the findings' accuracy increases with the number of respondents.
The size of the sample is a compromise between the required precision and the available budget.
An equilibrium must be found between:

Image

Therefore, more extensive sampling than the survey's requirements must be scheduled, so as to anticipate possible refusals, non-attendance of a session, disruptions, etc., which often occur during the implementation of a survey.

The respondent list

This list should include the respondent's name, profession, contacts, and, if required, his selection characteristics.

HOW IS THE PILOT QUESTIONNAIRE CONDUCTED?

Why is the questionnaire piloted?

A good quality survey relies on the piloting of the questionnaire and of the survey's methodology, in order to identify problems relating to the clarity of the question wording, the ease of response, the questionnaire's length, problems encountered by the interviewers, etc.
A limited number of people (usually 10 to 30) is selected to be the "sample for the sampling" and to test the questionnaire before its use.

The content of the pilot questionnaire

• Understanding of the terms used: if sentences have to be explained, the evaluator should rephrase them
• Complexity of questions: should they be split up? Should repetitive or investigatory questions be included in the questionnaire?
• Sequence of questions: does it provoke negative reactions (such as surprise, suspicion, boredom, avoidance, refusal, etc.)?
• Question wording: does it lead to the expected answers?
• The questionnaire's length: is the questionnaire tedious, irritating?
• The questionnaire's flow: are introductory and transitional texts sufficient and efficient?

Because surveys are expensive and difficult to administer (constraints of time), they are not usually systematically piloted during an evaluation, although this could add a qualitative dimension.

HOW IS THE SURVEY CARRIED OUT?

The different types of survey

The choice of a specific type of survey depends mainly on the context:

• In developed countries, questionnaires by telephone, email or Internet are increasingly conducted
• In developing countries, face-to-face questionnaires remain the surest tool for obtaining the information required. However, other types of questionnaire or tools may be used, dependant on the stakeholders' category (for example, local agents for development).

The various types of questionnaire and their particular characteristics

Advice

The carrying out of surveys with beneficiaries may pose logistical problems, especially in far-reaching areas. Among other distortions, the evaluator must be careful to avoid an over-representation of the most accessible areas (for example, nearby main roads), to the detriment of remote areas which may never be selected in field inquiries. If these elements are overlooked, the evaluator may miss specific elements of social organisation, such as access to equipment and utilities.

HOW ARE THE FINDINGS ANALYSED AND TREATED?

Coding questionnaires

After the receipt of the completed questionnaires, the code development stage can begin. This stage is necessary for open-ended questions which call for statistical analysis. Codes should reflect the type of analysis which has been selected.
The coding of findings consists of assigning a code value to each response, which forms the input for data processing. Codes present a straightforward interpretation of a free response (to an open-ended question) in numerical terms, and the responses may be a fact or a judgement.
The treatment of open-ended questions requires a very careful work. Responses must be grouped in relevant categories.
The determination of the way to code responses falls to the person or the team responsible for the analysis of findings.

Counting findings and correlations

After coding the questionnaires and entering the codes in a data processor, the evaluator can process the responses which should yield findings corresponding to the working hypotheses.

The treatment of responses

Analysing findings

The analysis of the findings relates to the working hypotheses. Pre-field techniques and documentation collected are also used during this analysis stage. Going back to the original responses may sometimes be useful to investigate certain elements (for example, to make additional "overlapping" of responses).

The different types of analysis 

Numerous statistical methodologies can be developed, whose usage is more or less complex. In the evaluation field, simple treatments are often sufficient for the analysis stage, even though establishing correlations between items enriches the analysis.

Margin of errors

Errors almost always occur in the treatment of findings (the survey's conditions, sampling errors, constraints of time, costs, etc.) and a margin of errors should not be overlooked during the analysis of findings.
Thus, findings should be interpreted as indicators of a tendency rather than precise statistical measurements.

Developing the survey report

The survey report must include an analysis of basic findings. It may contain, for example:

• A title page
• An index
• An executive summary
• An introduction
• An overview of the methodology
• The analysis of the findings
• The conclusions
• Recommendations interpreting the conclusions in terms of strategic choice, and providing advice about the types of action to implement
• The annexes

The evaluator must ensure as a minimum that the survey report has:

• Clarity
• Readability
• Accuracy
• Simple presentation with tables and diagrams
• A summary of the information ranked in accordance with its importance

Sample/sampling

Samples are designed to gather people who are to be effectively surveyed. People are selected from the evaluation's target population, and this selection depends on the methodology chosen for the design of the samples (see How are samples designed?).

Working hypotheses

Apart from assuming an empirical approach for the evaluation (which is almost impossible), scientific researches usually: (1) develop hypotheses, and (2) check them. This principle supports the survey's design. The survey's findings enable the evaluator to check the hypotheses at the core of the survey's development, to reinforce them and/or to correct them.

Target group

The target group represents the part of the population on which the survey focuses. A typology or descriptive variables determine it.

Typology

The typology is a methodology dividing a population into types or homogeneous sub-groups with specific criteria (economical, social, demographic, etc.). Within each sub-group of the typology, respondents share similar characteristics (corresponding to the typology's criteria), or even the same values for certain predetermined variables.

• Examples