Evaluation methodological approach

This section is structured as follows:

• Why and when?
• How the interview is carried out
• In country/region evaluation

 

WHY AND WHEN?

 

WHAT ARE THE REASONS FOR USING INTERVIEWS?

What is the purpose of an interview?

The use of interviews in evaluation is inevitable. This tool collects information and points of view, and analyses them at each stage of the evaluation.

The interview usually takes the form of a face-to-face discussion between the evaluator and the interviewee.

Where does this tool come from?

Interviews are used in many fields, such as psychology, ethnology, sociology, anthropology, and philosophy.

In psychology, this tool focuses on motivation, or the reasons for a person's behaviour. The interview is thus used to study human actions and to help with the understanding of the human psyche.

In ethnology, the interview aims mainly at collecting direct observations. In sociology, the interview is used in all activities.

Three types of interviews and their contribution to the evaluation

Unstructured interviews

The interviewee expresses himself/herself freely and can discuss unplanned topics, because there is no predetermined set of questions. The evaluator intervenes only to generate and develop questions relating to the interviewee's comments.

This type of interview is particularly interesting at the start of an evaluation, in order to get a global view of the subject, and identify the major topics and issues.

Semi-structured interviews

The evaluator modifies the interview guide's instructions with additional questions, in order to develop useful areas of inquiry during the interview.

This type of interview is the most frequently used, particularly when the evaluator knows sufficient about the aims and the main questions to pose during the evaluation.

Structured interviews

The evaluator follows strictly the interview guide's instructions. He asks different interviewees the same set of questions, in the same order, and using the same words. The evaluator avoids generating and developing additional questions, and the interviewee is not given the opportunity to express himself/herself freely. Answers to each question tend to be short. Structured interviews are seldom used in evaluation, where the evaluator needs to adapt to the situation. However, they can be used to classify points of view and information about the impact of a project/programme by categories. Thereafter, the evaluator can use the results of these interviews to design a questionnaire, with a view to analysing the impact of the project/programme.



Semi-structured interviews are the most commonly used tool in evaluation and are the subject of further guidance.

 

WHEN SHOULD THE INTERVIEW BE USED?

What kind of information does the interview collect?

The interview may be used as a quantitative collection tool; however, it is mostly a qualitative device.

In evaluation, the use of interviews is simple, quick, and affordable.

The interview collects:

• Facts and information for the verification of facts
• Opinions and perspectives
• Analyses
• Suggestions
• Reactions to the evaluator's hypotheses and conclusions

However, a series of interviews is not sufficient to quantify an event, because the answers are not standardised. This is the main difference between the interview and the questionnaire.

The added value of the interview

Among other advantages, this tool is essential for the development of analyses because it collects information taken directly from the context. Thus, it provides a good indication of what motivates stakeholders to act, their various perceptions of the programme's aims, problems encountered and effective outcomes.

Can the interview be combined with other collection tools?

The interview is a useful device for developing hypotheses and analyses. It can highlight the programme's aims and dynamics, the stakeholders' rationale, and the organisation of the various opinions and perceptions of the programme.

Depending on the type of observation tool used and on the stage of the evaluation, the interview can be used in combination with other tools:

• To test the main questions to address at the beginning of the evaluation with a view to preparing a series of focus groups.
• To be the main observation tool, and be supported by a questionnaire (if the country's general context allows it) or, for example, a focus group of beneficiaries.

How should the interview be conducted?

Usually, the interview takes the form of a face-to-face discussion. This arrangement is particularly effective, as the relationship builds on trust and interviews supplement written information.

However, beyond verbal information, other elements should be taken into account, such as the context, the general mood of the session, the people likely to influence the interviewee, etc, as well as the interviewee's reactions: hesitation, silence, eye contact, etc.

From whom does the interview collect information?

The interview is a suitable tool for collecting information, analysing and forming conclusions from a limited (but essential) number of respondents, such as:

• Partners and people in charge of the evaluated policy or programme
• Strategic institutional stakeholders
• Main operators and people in charge of the programme implementation
• Representatives of beneficiaries

 

AT WHAT STAGE OF THE EVALUATION SHOULD THE INTERVIEW BE CONDUCTED?

Types of interview appropriate to various stages of the evaluation

Stages of the evaluation	Type of interview	The interview's contribution to the evaluation
Desk phase: implementation of the methodology and preparation of the mission to the country	Preparatory interview used for the design of questionnaire grids and the selection of respondents. At this stage, interview guides should be flexible and aimed at highlighting the topics on which the interview is based.	It strengthens the basis for the choice of major topics and issues, and completes the questionnaire grid. Its structure should be flexible.
Collection of information in the country and from the European Commission	The interview is designed to collect information and perspectives. Several interview guides should be developed to correspond to the different categories of respondents, and to the major topics and issues.	It collects information on the programme's objectives and outcome from the people in charge of the programme, operators, stakeholders and beneficiaries.
Analysis and preparation of judgements	In-depth interviews for presenting and investigating the issues, used to collect reactions to the evaluator's findings and analyses. At this stage, the evaluator can test the relevance and the feasibility of his/her conclusions.	It collects feedback from respondents. It gives in-depth information and reformulates old questions.

 

WHAT ARE THE ADVANTAGES AND THE LIMITATIONS OF THE INTERVIEW?

The advantages

The interview is appropriate for project evaluation, as well as for more complex evaluations, such as sector evaluations and country/region evaluations, where it should be combined with other tools.

The limitations

 

WHAT ARE THE PRE-CONDITIONS FOR THE USE OF THE INTERVIEW IN EVALUATION?

The time span

The preparation for the interview does not take long.

One expert will not be able to conduct many interviews per day and, therefore, the number of interviews which can be carried out during the mission is limited. In practice, at the interviewee's request, the expert may conduct an interview with several respondents at the same time. Thus, this particular usage of the interview increases the opportunity for collecting the information required in a relatively short time.

Human resources

Interviews must be conducted by a trained professional. The necessary skills are:

• Thorough knowledge of the major topics and issues addressed in the evaluation
• Excellent interviewing skills:

• The ability to quickly understand the respondent's perspective (his/her interest in the interview, whether he/she has expressed himself freely, whether he/she has committed himself) in order to be interactive and, where appropriate, modify the questions

Financial resources

Costs depend on the number of interviews and their physical location in the country. However, apart from professional fees and transportation costs, the interview itself does not lead to substantial costs.

 

HOW THE INTERVIEW IS CARRIED OUT?

 

HOW IS THE INTERVIEW PREPARED?

Stage 1: list the categories of stakeholders and interviewees

The evaluator should define the categories of stakeholders he needs to meet, and then select a sample group for each category.

Examples of stakeholders' groups

• Agents in charge of the operational strategy / the implementation
• Actors specialised in a sector of intervention
• Operators working at a local level / at a national level
• Technical operators: representatives / on-site operators
• Public sector agents (working for the state, the administration, etc.) / private sector agents (working for the civil society, NGOs, etc.),
• Beneficiaries

The list of the stakeholder categories will evolve as the evaluation progresses. Thus, additional interviews should be anticipated.

Some choices are obvious (for example, the agent in charge of the implementation of a nationwide programme), but others are more complex.

Moreover, the evaluator must consider how he will meet the interviewee, which depends on:

• The respondent's accessibility (will the evaluator meet him in person or only a representative)
• The respondent's availability

What are useful questions to bear in mind for the selection of interviewees?

• Within the relevant institution, who knows the programme/the politics well?
• Who is in charge?
• Who is in charge of implementation, of monitoring?
• Of which aspects of the programme is the respondent in charge?
• Who will be able to trace the programme's evolution, in the context of a rapid turnover of the implementation team?

During the interviewees' selection, the evaluator should be careful to distinguish between direct witness' testimonies and second-hand testimonies.

Stage 2: design the questionnaire grid

Questionnaire grids, and interview guides derived from them, vary with the categories of respondent and the latter's links with the evaluated issue.

Grids should include all instructions, themes and questions which the evaluator wants to discuss with the respondents.

The stages required to organise the grid

 

The questionnaire grid is an intermediary between the evaluation study's design and its implementation.



Design of the interview guide

The evaluator should allow the interview to remain as close to an open discussion as possible and accept discursive answers, inconsistencies, pauses, hesitations, etc.

The questions can be written in the questionnaire grid, or be asked spontaneously, in order to complete, to probe, to give new insights and to challenge the respondent's answers.

 

Stage 3: schedule the interviews

This organisational stage depends on the evaluation stage within which the interview is implemented, and on the respondent's availability.

The selection criteria must clearly indicate the priority of the people to be met. Furthermore, the evaluator must be careful about:

• The risk of key officials being unavailable
• His/Her short mission span
• Travelling time
• Scheduling too many interviews, leading to the cancellations of appointments due to poor planning or unexpected events
• Respecting his/her appointment, such as the time, although conventions vary from one country to another
• Conducting the interview's introductory stage slowly, in order to establish rapport and respect traditions and customs
• Unplanned and additional interviews which have to be organised because a respondent has suggested an "interesting" informant who may be interviewed, or because a key interviewee is now available
• Tiredness: too many interviews conducted in a day may affect the capacity of the interviewer to listen

The evaluator must find a balance between:

 

Scheduling 4 to 6 interviews a day seems to be a good compromise between the two visions.

Make the appointments

This stage constitutes the first contact between the interviewee and the evaluator. Therefore, the choice of the person in charge of this task is very important.

It is often better to schedule appointments in advance. However, time should also be allocated for unplanned interviews, particularly in the context of country/region evaluation.

The evaluator must know how to resist pressure to meet as many respondents as possible. Organising focus groups can solve this problem.

Send interview guides prior to the appointment

This stage is optional, however, it often eases the interview's course by giving the respondent the opportunity to:

• Understand the topics on which he/she has to talk
• Collect all the necessary documentation
• Ask the evaluator whether the participation of other respondents during the interview is possible

It will save time for both the respondent and the evaluator, and help them feel at ease. Most of the time, the quality of the interview depends on this.

Conduct additional interviews

They are often necessary after the first elements are collected and during the progress of the evaluation. Thus, they must be programmed into the planned timetable and budget.

HOW IS THE INTERVIEW CONDUCTED?

Stage 1: establish a rapport

The interview is easy to conduct but its findings and value vary greatly with the way it is conducted.

The evaluator's duty

• To be aware of and respect local habits and customs (such as rules of politeness, silences, wording of questions, time devoted to conversation prior to core questions, etc.)
• To anticipate any language difficulties (codes, interpreter's competence, meaning of terms such as "development", etc.)
• To adjust himself/herself to the interviewee, his/her role and his/her hierarchical rank in the institution (for example, the evaluator may be asked not to reveal some information)
• To take into account the interviewee's material, social and hierarchical environment which may greatly influence his/her attitude
• To explain the purpose of the interview, how the respondents have been chosen, and the intended use of the information
• To establish (and sometimes to negotiate) the rules, such as the interview's length, the recording of the interview, etc.
• To inform the interviewee at the start whether his answers will be used anonymously
• To ensure that the interviewee has understood the aims of the interview and is willing to respond

To find out more:

• What are the tasks to be completed during the preparation for the interview in country/region evaluation?

Stage 2: Adjust the respondent's answers to the interview subject

The evaluator must adjust to his interlocutor's role and hierarchical rank in the institution. As a consequence, he/she must be aware of the specificities of the respondent's answers, such as:

• The way he/she understands the questions and deals with them
• The possible difficulty in expressing points of view, or criticisms
• Ideas he/she might want to point out with the evaluator's mediation
• Considering the evaluator as an auditor

The evaluator must also adjust to the interlocutor's attitude: his/her personal perspective, points of interest not planned in the interview guide, etc.

As a consequence, the evaluator's flexibility is the key to successful data collection in an interview. During the interview, however, he/she must control its progress by staying within the bounds of the subject and avoiding dwelling too much on one topic.

Stage 3: follow the interview guide and deepen the questioning

Flexibility and control are two elementary rules in an interview. The respondent's information is being analysed simultaneously through the evaluator's capacity to listen. Thus, the evaluator should not express himself/herself in a way which is detrimental to the interview, nor let the interviewee talk without limit. In practice, the evaluator's ability to react should provide him/her a balance between flexibility and control.

Three types of reactions

• Contradiction (the evaluator highlights the respondent's contradictions or expresses the contrary views of other respondents, whose identity may have to be concealed)
• Notification (the evaluator informs the respondent that he/she is about to address a new theme or a new question)
• Clarification (the evaluator asks the respondent to develop a specific point)

Types of clarification

Questions	Repeat verbatim the respondent' s comments in the interrogative form
Repetition	Return to items which have already been discussed
Interpretation	Summarise the content of what has been said, check its interpretation with the respondent (and correct it, if necessary), and move on to the next point

 

Intervene directly in the interview

Although the interview guide provides a useful structure for the interview, the evaluator must be free to generate and develop questions beyond it.

Making direct observations during the interview enables the evaluator to collect specific information about the respondent's attitude, behaviour and customs, and / or about the people around them. With these observations, the evaluator can detect discrepancies between the respondents' attitude and their words.

To find out more:

• Three types of interviews and their use in evaluation

The nature of the collected information

The evaluator must distinguish reliable facts from points of view, personal analyses and opinions. To do so, he/she should press the respondent to support his/her allegations with facts or actual examples which the evaluator can check, and which reinforce the respondent's comments.

Avoid asking difficult questions

During an interview, questions should be answerable and should avoid discouraging respondents. Thus, interviewees should be spared from having to provide information stemming from archives or voluminous report.

Control the information

In an interview, the first level of control occurs with the triangulation of questions. This methodology is based on the principle that three different sources are needed to validate the reliability of the information. Triangulation is used in the interview in two ways:

• In interviews, the evaluator asks systematic and similar questions to at least three different respondents. Prior to this triangulation, the information is not considered to be reliable.
• In a single interview with one respondent, the evaluator asks a question in three different ways, in order to check the information and observe the possible fluctuation in the respondent's analyses and interpretations of an event.

Stage 4: Conclude the interview

Conclude the interview

The evaluator should close the interview with positive reflections or an open conversation, and maintain a polite approach. He/She should also decide whether another appointment with the respondent is necessary.

Keep track of all the information

The evaluator should read his notes shortly after the interview, structure them and add, if necessary, non-verbal details such as the respondent's behaviour, trouble, silences, interruptions, his relationship with him, the atmosphere, and especially the respondent's suggestion of other people to be interviewed (who will have to be contacted) and read the specified documentation, etc.

These elements are invaluable; they are not detectable in audio tapes, nor short notes. Thus, the evaluator's impressions at the end of the interview must be considered a valid source of information.

Protect the confidentiality of the interview

If the respondent's willingness to respond depends on the evaluator's assurance of confidentiality, this principle must be respected, along with respect for the respondent's private life.

If necessary, validate the content of the interview report with the respondent

Depending on the intended use of the respondent's answers, this validation can be important if, for example:

• the respondent's verbatim comments are cited in the report
• They stand as evidence on their own
• The evaluator finds them ambiguous and is afraid of making a poor interpretation, or worse, a misinterpretation
• The respondent has an official or sensitive duty
• The respondent's willingness to respond depends on this validation

Anticipate what will have to be done after the interview

• The intended use of the information collected
• Its analyses
• The types of debriefing, which should meet the client's and respondents' expectations (sometimes, interviewees request feedback about the use of their answer)

 

IN COUNTRY/REGION EVALUATION

 

WHAT ARE THE TASKS TO BE COMPLETED DURING THE PREPARATION FOR THE INTERVIEW?

The hypotheses and questionnaire grid development

In country/region evaluation, the evaluator needs to define the strategic questions quickly, in order to be efficient during the information collection.

 

The evaluator's tasks during this first stage

Select the types of information to collect

The evaluator should:

• Organise the information to be collected on strategic questions between key and secondary interest
• Devise a methodology to limit the collection of documentation. As country/region evaluation covers so many diversified fields, the evaluator risks dissipating his efforts if this stage is neglected

Develop and organise the questions

The evaluator should classify key questions by their aims and categories of interviewees.

Determine the list of the respondents

Settle the order in which respondents will be met

THE TYPES OF INTERVIEWS?

Different types of interviews can be distinguished during the two main stages of country/region evaluation

During the first stage

Preliminary interviews setting the programme's purpose and boundaries

These consist of meeting people who have a global vision of the policies implemented, their objectives and impact, and who belong to relevant institutions. These interviews can avoid bias in the process of an evaluation arising from reliance on officials in charge of policy definition and monitoring.

Interviews with different categories of stakeholders

During these interviews, the evaluator meets the stakeholders who have been participating in the process of drafting, definition, negotiation and the operational choices for the implementation of policies.

Thus, the relevance and impact of geographical strategies rely on what different stakeholders have done in the process of drafting and negotiation.

During the second stage

Interviews with operators and beneficiaries

The use of interviews to analyse the impact of a programme on beneficiaries in country/region evaluation is not usually advisable because of the large number of people involved, representative problems, etc. However, interviews can be conducted prior to the implementation of other data collection tools, such as focus groups and questionnaires.

Thus, interviews should focus on particular categories of beneficiaries and/or precise questions dealing with various hypotheses.

HOW ARE RESPONDENTS CHOSEN?

In country/region evaluation, this issue is crucial. It is less a question of knowing how to collect information, than of determining what the information's content should be, and from whom it can be obtained, in the context of short missions and wide fields of inquiry.

Choosing the categories of respondents is particularly important:

• To identify people who have benefited from the implemented policies
• To identify those who have played a strategic role in the programme
• To identify those who have played an intermediary role
• To identify those who could have been behind obstructions or unplanned effects (such as groups with diverging interests, intermediary groups intervening in the middle of the process, target groups of the policies / the programme, etc.)
• To analyse current strategies and the various rationales considered

Distinguish between the different roles of officials from institutions

The list of officials who should be met can be long because of the nature of the country/region evaluation, which can cover numerous programmes and projects.

Thus, a selection must be made consistent with the budget allocated to the evaluation, and especially when the mission is very short. The evaluator should make a distinction between key interviews and "etiquette" interviews; he/she should also think of the quickest and cost-effective way to conduct them.

Once the roles of officials are identified, the evaluator must locate the people who have agreed to be interviewed, corresponding to this typology.

This selection belongs to the evaluation process and becomes clearer as the first hypotheses are being developed.

Go beyond formal statements

In country/region evaluation, the evaluation's political dimension and economic implications tend to complicate some interviews with institutional authorities and respondents concerned with their hierarchy. The evaluator should be able to "decode" the information he/she is given, by taking into account the context in which the interview is conducted.

A way to avoid formal statements implied by this situation is to carry out an informal interview (for example, away from the institution to which the respondent belongs).

Balance the interviews between officials and beneficiaries

The evaluator must be careful to avoid devoting all the time in his/her mission to interviews with officials, to the detriment of beneficiaries. He/She must check the interviewee's "representativeness", which is always relative and sometimes self-proclaimed. Several questions may help find the respondent's interest in the evaluation and his/her "representativeness":

• To which group does the respondent belong?
• What is his/her professional background?
• What is his/her influence on the programme's strategy?
• Does he/she have direct or indirect interests in the programme? Are they explicit or implicit?
• What is his/her opinion about the policies / the programme?
• How did he/she become a representative (has he/she been elected, designated)?
• Is he/she representative of the entire group he/she stands for?
• How many respondents should be interviewed to collect representative information and perspectives?

HOW DOES THE EVALUATOR SCHEDULE THE APPOINTMENTS?

The pre-conditions for appointment making

During the preparation for his/her interviews, the evaluator should be practical and:

• Fix an order of importance for the collection of data and, after developing the first set of questions, select who should be interviewed
• See: What specific tasks have to be completed during the preparation of the interview in country/region evaluation?
• Identify quickly on-site informants and stakeholders. They often hold a position at the centre of many sources of information. Thus, they can advise the evaluator, and suggest "good" respondents, who will be able to give accurate answers.
• Identify respondents who would be better interviewed in a group
• Consider the possibility of interviewing people on the telephone, which could save time
• Avoid overestimating the number of interviewees that can be met during a short mission

How to schedule the appointments?

The particular context of country/region evaluation implies taking additional precautions, in addition to the usual precautions, in the making of appointments.

The evaluator should:

• Schedule in advance at least part of the interviews that he/she needs to conduct on-site
• Ensure that the evaluation's clients (for example, the European Commission Delegation or the national authorities), or the local consultants organise the first set of interviews on his/her arrival
• Make time available for unscheduled appointments made necessary because of new perspectives identified during his mission. As a consequence, the evaluator's hypotheses can be extended or modified.

With a view to being more efficient, the evaluator could:

• Use the appointment making process as an opportunity to communicate to the respondents all the information they need to know: the purpose/ of the evaluation, what questions they should prepare for, and the interview/ duration. This is also the appropriate moment to check the respondent's profile against the planned content of the data collection.
• Distribute an interview guide prior to the interview. This is appreciated by all parties because it is time-saving and directs the respondent's information gathering to specific documentations.