Evaluation methodological approach

WHAT ARE THE REASONS FOR USING A FOCUS GROUP?

The evaluators use focus groups for two different reasons:

• This tool collects information from a group of participants
• It facilitates the collective expression of analyses, perspectives and suggestions

In the evaluation field, focus groups are used to achieve different objectives, at different stage of the evaluation and with various participants.

For what purpose should a focus group be used?

A focus group can be used as:

• A means to collect information and points of view from a group of stakeholders, operators or beneficiaries
• A means to investigate hypotheses and analyses with groups of stakeholders, operators and beneficiaries
• A means for various groups of stakeholders (whether or not they belong to the same social and institutional background) to express divergent points of view and their analyses
• An analytical tool focusing on the programme's effects, particularly at a local level
• A debriefing tool, at the local level, disclosing the evaluator's observations and his/her first conclusions of the on-site analyses
• A means to prompt and validate various suggestions and recommendations at an institutional and local level

Les fonctions de l'entretien collectif

For what outcomes?

Focus groups are used in order to:

• Collect qualitative data, and more rarely quantitative information
• Examine analyses and perspectives (the participants interact during the session, which facilitates a spontaneous exchange of views)
• Provide the participants with insights into perpectives from other groups, and examine the diversity of analyses developed by different groups of stakeholders. For example, using a focus group for a sector-based evaluation with the various operators in the sector usually yields good results.
• Develop and test working hypotheses at the beginning or during the evaluation
• Develop and test conclusions, recommendations and suggestions with stakeholders and beneficiaries for a mid-term evaluation

WHERE DOES THIS TOOL COME FROM?

The focus group comes from sociology, anthropology and the marketing field.

A controversial issue

Some schools of thoughts refuse to include focus groups in the "family" of group interviewing. As there are several techniques and methodologies available to design a group and conduct sessions, each member of the "family" has its own scientific basis. The choice between these various methodologies depends on the focus group's purpose.

Specific examples

• In the marketing field, focus groups are frequently used out to develop and validate advertisements or analyse new products.
• In developing countries, the social communication field uses focus groups to design understandable messages (for example, in awareness campaigns, dealing with aids prevention).

WHAT IS THE ROLE OF A FOCUS GROUP IN EVALUATION?

In what kind of evaluation?

Focus groups are frequently used in project or programme evaluations, and especially in field studies focusing on the beneficiaries' situation.
This tool could also be successfully used in sector and country/region evaluations, where the selection of objectives and types of participants is broader.

What type of focus group for which stage in the evaluation?

Preparatory focus groups

They provide grounds for the selection of the major topics and issues, and complete the questionnaire grid.

Focus groups for information collection

At the heart of the evaluation, these focus groups collect information and analyses:

• From the people in charge of the project/programme/policies, about the objectives and outcomes
• From stakeholders and operators, about the policies implemented
• From beneficiaries, about the effects

In-depth focus groups

During the evaluation, in-depth focus groups gather operators and beneficiaries together to analyse, collect and examine their reactions to the first draft of conclusions and recommendations.
In-depth focus groups should not be confused with sessions between the evaluator and clients which are designed for the validation or presentation of the final conclusions.

With which tools can the focus group be combined?

Focus groups can complement interviews, and be a useful alternative to surveys (which often appear to be difficult to plan and manage, and expensive).

With case studies

Focus groups can be used to conclude case studies in sector and country/region evaluations. For example, the evaluator can set up focus groups to compare opinions, and focus groups which encourage reactions to the evaluator's suggestions.
These types of focus groups concentrate on geographical sites, sectors or economic activities outside the scope of the case studies. Using the information and opinions collected by these focus groups, the evaluator will be able to assess the specificity or the general view of particular findings from case studies.

With interviews

In this case, focus groups aim at examining and challenging elements of individual analyses and opinions which have been expressed in earlier interviews.

WHAT ARE ITS ADVANTAGES AND LIMITATIONS?

The advantages

A means to interview numerous respondents

By definition, the number of respondents from whom information, analyses and opinions are to be collected, is greater in focus groups than in individual interviews. Consequently, the focus group's reference sample is bigger.

• With focus groups, the evaluator can investigate and develop information and analyses provided by interviews.
• The evaluator can also ensure that a point of view individually expressed can represent the opinion of a group of stakeholders.

However, participants in a focus group do not constitute a representative sample of the programme's reach, and findings cannot be interpreted statistically because the information collected is qualitative.

A tool which facilitates the study of the programme's effects

This tool is cost effective and can also provide the evaluator with valuable information about the effects of policies / programmes on specific groups of beneficiaries or stakeholders. Considering effect analyses, the evaluator can use this tool to collect the opinions of strategic interest groups about the aims of the policies / programmes under evaluation.
The more precise the preliminary identification and the grouping of strategic interests, the better the quality of the information collected.

A source of creativity, if the group dynamic is controlled

Focus groups facilitating the expression of divergent points of view provide opportunities:

• To analyse the stakeholders' reactions to other perspectives
• To examine the argumentation, perceptions and analyses of each group of stakeholders, through exercises involving confrontation and debate

Whilst not designed for this purpose, these sessions, can overcome obstacles and contradictions which are often linked to misunderstandings rather than contradictory interests.

The limitations

Complex organisation and preparation

Prior to the organisation of focus groups, the evaluator usually needs to conduct preliminary analyses on strategic groups which have been formed within the various stakeholders.
In developing countries:

• When the physical location of beneficiaries is scattered, sessions are difficult to organise and are time-consuming
• Local authorities may have difficulty in appointing national and local professionals to prepare and organise focus group sessions with the evaluator.

Restricted participant expression

Sometimes, the evaluator fails to elicit the participant's views due to:

• The public nature of the session
• The inhibiting presence of certain participants
• Political and social issues, which are more significant in a group situation than in interviews
• The natural tendency within a group to conform with the majority view and to avoid expressing minority opinions

Time allocation and costs (which can easily increase)

Conducting several focus groups can be problematic in terms of time allocation and costs, especially due to:

• The preparation, which is often time-consuming
• Compensation paid to participants
• Transportation costs
• Facilitators' salaries
• Information processing, which is often time-consuming

.

.

HOW IS A FOCUS GROUP SET UP?

Choose the type(s) of focus group

For each type of focus group, the evaluator must specify:

• The objectives of the focus group (to collect information, to analyse, to challenge opinions, to suggest, etc.)
• The categories of participants to be gathered
• The preliminary tasks to be completed in preparation for the focus group
• The most appropriate way to conduct the sessions

Schedule the implementation of the focus group

Two elements must be taken into account:

• Preparing focus groups may be time-consuming, especially for the selection of participants, the planning, and the organisation of the session. This is particularly true for focus groups of beneficiaries.
• Focus groups relating to an impact study of a policy or a programme should be implemented only after the evaluator has developed his/her initial hypotheses and analyses, which implies significant preparation.

Within the allocated time, the evaluator may have difficulty in dealing concurrently with the methodological quality of the focus group and implementation delays imposed on him/her.

Identify the strategic interests groups

Preliminary documentation analyses and interviews are necessary before the focus groups can be set up.

This work helps the evaluator to determine the various strategic interests groups among categories of stakeholders and beneficiaries. These groups usually identify themselves by their expectations, their points of view, their social perceptions and their specific strategies relating to the policy or the programme under evaluation.

Select the participants

Once the strategic groups are identified, the evaluator must select from them the people who will be invited to take part in the focus group. To do so, the evaluator needs a mediator or a local key informant, particularly when representatives of populations living in remote areas, or ones who only speak in dialects are invited.

The selection of the participants depends on their availability, the equipment and human resources (an interpreter, for example) at the disposal of the evaluator.

In addition to the moderator's skills, which play a crucial role in the overall conduct of the session, the group interaction must also be considered. It can be fostered by one or more leaders who often intervene in the debate and stimulate the other participants. The moderator and the key informant should thus ensure that such leaders participate in the session.

The participants' motivation to become actively involved in the focus group should have been stimulated by an in-depth presentation of the topic of the session. They should understand the principles underlying the session's process, and think about the topic before the session. This is particularly recommended for focus groups with users and beneficiaries, who are not familiar with this type of approach.

Construct the moderator's guide

The moderator's guide should be designed sufficient in advance to allow the moderator time to study it. The evaluator should specify the topics under study and the themes to be developed during the session, although participants are free to develop other questions linked to the suggested themes. Suggestions should therefore be highlighted in the text (including the information which is to be collected by the moderator).

Advice:

• Summarise the questions: the list of questions should be short. The moderator's guide should be a memento more than a questionnaire. The moderator should adapt the formulation of the questions to the reactions and statements of the participants. Thus, some questions prepared in advance by the evaluator will not need to be submitted to the participants specifically, because they will have discussed these questions spontaneously.
• Avoid questions expressed in an alternative form. Experience has shown that only the first part of the question is taken into account.

Plan the focus groups meetings

This stage mainly depends on the availability of:

• The people invited to join the focus groups
• The evaluation staff
• The key informants

Planning the focus groups also requires sufficient progress in the preparatory work relating to the focus group implementation.

WHO IMPLEMENTS FOCUS GROUPS?

Usually, the moderator (or the facilitator) implements the focus groups. He/she should be selected for his/her capacity to create and maintain the group interaction. The quality of the focus group depends on this capacity, as well as a focused and relevant exchange between participants. He/she should therefore be well informed of the evaluation's topics and goals.

What are the necessary skills?

Focus groups must be conducted by a skilled professional who:

• Thoroughly comprehend the evaluation's major topics and issues
• Is familiar with all the techniques relating to group interaction
• Speaks the language of the participants
• Is sufficiently aware of the participants' social and cultural characteristics, in order to:
  • respect their traditions and customs,
  • understand any implicit chains of command among them,
• Avoids power games or social in-fighting between the participants

How should the moderator be trained?

Although moderators are selected for their abilities to moderate a group, few of them will really know how to moderate a focus group. They should be introduced to the topics and goals of the evaluation, and be trained in the type of techniques which will ensure that the focus group process is successful. A training day should be scheduled.

This training day includes the presentation of the moderator's guide, indications about the goals of the focus group and how the evaluator wants it to be conducted. Before the session, the evaluator should check with the moderator that all the terms and expressions used in the moderator's guide are understood.

Advice to the moderators:

• React to statements, in order to obtain more information
• Ensure that all participants have sufficient time to express their opinions
• Present a summary for each topic developed, and ask the participants to react to the summary
• Reformulate the questions in case the participants have not properly understood them
• Take body language into account (nod of the head, laughs, etc.)

What are the moderator's tasks?

Guide the sessions

• Describe precisely the issue(s) to be discussed without suggesting solutions
• Provide details about the original situation
• List all the hypotheses
• Focus the discussions on the major points from the interview guide

Supervise session progress

• Make a general review of the group's findings
• List the issues raised by the discussion by order of importance
• Rephrase ideas which are not clear
• Maintain the momentum of the debate

Encourage the discussion

• Establish the reasons for any impediment to free expression within the group
• Avoid tension
• Control "leaders" and encourage "shy" participants
• Take a full account of group silences and hesitations

What are the tasks of the local key informant or intermediary?

The participation of key informants during the selection of participants is often helpful to the evaluator.

The interaction within the group depends on its composition, and of the presence of one or two very active participants, who can foster the debate. The key informant can ensure that such individuals are included in the group.

The key informant can also prepare the participants by explaining the organisation of the session, the topics and the objectives of the focus group.

What are the observer's tasks?

In addition to the moderator's activities, an observer can be invaluable to the evaluation by keeping track of the opinions expressed (shared and contradictory) during the session.
The observer can also replace the moderator in his note-taking task and allow him more time to concentrate on his core duties.

HOW ARE THE PARAMETERS OF THE GROUP DETERMINED?

How are the groups designed to match the evaluation's purpose?

Determining the group's composition in support of the evaluation's purpose

When this tool is used to help a group articulate its of points of view, two possibilities can be considered for the composition of the group:

Groups constituted by people sharing similar opinions and professional activities

When the group comprises socially homogeneous members, participants can speak freely. The debate, however, can be stimulated when participants do not have identical views on the questions. Differing analyses, viewpoints and experiences can be challenged on the basis of a common understanding.

A focus group with a homogeneous composition is used to probe information and points of view expressed by stakeholders sharing the same strategic interests (in terms of social and professional status). Thus, the focus group provides the evaluator with an overall view of the experiences, the opinions, the behaviour and the needs of participants.

Groups constituted by people having different opinions and professional activities

In this case, the evaluator expects that the participants' analyses and points of view about the issue will be divergent. Consequently, the focus group is used as a means to express these divergences publicly. Each category of participants is given the opportunity to explain and defend his/her opinions. The focus group is a means to collect statements about perspectives on an issue. The evaluator can draw the participants' attention to the similarities, the differences and the complementarities of their various points of view. The momentum of this divergent discussion provides the evaluator with elements for new analyses and insights relating to the causes of these divergences.

Specific cases

The composition of the groups may sometimes alter the content of the information and the analyses under discussion.

Example

• In hierarchical societies, categories of people may have difficulties in expressing themselves in front of their superiors, not to mention challenging the general opinion. This aspect accounts for the potential artificiality of village sessions about impact analysis. Yet, the evaluators tend to under-estimate this social context.

The evaluators can also set up groups of agents and officials working for the same institution. In this case, they should be careful about the impact of a point of view expressed in public.

Example

• In poorly democratised societies, participants expressing themselves publicly at the request of the evaluator may take risks.

The participants can intentionally redirect the focus of the discussion for different reasons. They can speak solely amongst themselves, or they can defend the strategic interests of their group and attempt to manipulate the evaluator, or even accuse him/her of bias.
Thus, the evaluator must always be wary and keep control of the focus group's process. He/She must particularly avoid letting an individual participant dominate the debate, or allowing a coalition to emerge which dominates the group. He/She must also monitor the substance of comment and be fair in the allocation of time for expression.

How is the size of the group determined?

The size of a focus group varies greatly. However, it should not exceed 10 or 15 people, in order to let everybody talk effectively.
In the case of numerous groups, the evaluator should organise the participants into sub-groups for at least part of the evaluation's course. Consequently, he/she must have enough moderators at his disposal for each sub-group. Conversely, a focus group of less than 4 or 5 participants is difficult to manage, due to the lack of group interaction which constitutes the added value of this tool.

HOW IS GROUP INTERACTION FOSTERED?

Presenting the team in charge of the focus group to the participants

Prior to the focus group session, the context of the evaluation and its objectives should be reviewed and the team should be introduced. This will help avoid misunderstandings which otherwise may change the content of the participants' statements.

The evaluator must therefore remember that when he/she organises a focus group with beneficiaries, he/she is perceived as the representative of international donors, which makes him/her an agent for dysfunctions highlighted by participants. The evaluator should be careful about participants' natural tendency to consider the evaluator as a means of communicating with other groups (of civil society or the political arena). This is not a handicap for the group interaction, however, because the exchange between participants is livelier than a collection of personal experience statements.

Which methodologies should be selected to conduct a session?

Several techniques for conducting focus group sessions are at the disposal of the moderator, to avoid boredom and retain the attention of the participants, and to develop the progress of group interaction.

Focus groups are not just the sum of individual interviews. Thus, the moderator must always initiate and maintain a dynamic interaction between the participants. The quality of the interaction depends on the moderator's skills to conduct a focus group.

• Focus groups organised in a reactive way: the evaluator provides the participants with information, analyses and suggestions, which support the collective process of thinking and debating. The purpose of this method is to collect information and validate the conclusions and recommendations of the evaluator.
• Focus groups organised in a pro-active way: the information and testimonies of the participants support the development of collective analyses and suggestions.

Recommendations about the moderator's behaviour

What the moderator should encourage and avoid during a focus group
 

Focus groups do not yield quantitative data and are not representative of the whole target group (beneficiaries, for example). Thus, the evaluator should remember that the expression of personal opinion is susceptible to a group bias (need for belonging and recognition, consequences of leadership, exaggerated assertions, artificial conflict originating from the debate on various opinions, etc.).

How to manage the tempo of the focus group?

Several elements must be taken into account:

• Schedule the focus group's duration: it should last half a day as a minimum, because a group needs time before yielding collective results
• Organise the session into stages, including a mid-term debriefing to the group. Regular debriefings are necessary to record and confirm the various assertions and points of view from the participants
• Quickly create group interaction in order to go beyond the logic of "sum of individuals". The evaluator must therefore develop a confident and relaxed mood for the session, a balanced expression of opinions among the participants, etc.
• Maintain the momentum of the group's interaction by frequently changing the dynamic of the focus group

The same session can be sometimes reactive to the outline of an analysis, information, etc. and sometimes pro-active, i.e. the group is able to yield rational information and analyses by itself.

Which type of interaction should be encouraged?

Proactive participation needs to be very structured. To this end, the moderator can use visual or audio displays, and introduce practical exercises, simulations, and time dedicated to syntheses and summing up, etc.
The moderator can plan the progress of the focus group in advance. However, anticipating the quality of the interaction and the effective tempo of the group is very difficult.

HOW TO KEEP TRACK OF INFORMATION?

Taking notes on a board, or using other collection devices during a session can help the group to keep track of the information debated and the subsequent synthesis.

The evaluator must conclude the session with a debriefing of what has been said, in order to check for any disagreements or misunderstandings. This stage should be kept informal, in order to avoid time-consuming formal validation.

The moderator or the observer should categorise participants: people who often express themselves and people who seldom do; sub-groups conversing with other sub-groups and sub-groups who seldom do, etc. These notes are important to put the verbatim comments in its proper context.

After the session, the evaluation team compares notes, determines and fills any missing points. The evaluators should also study the group interaction: if it has been impeded, they will have to restructure future focus groups.

A good way to keep track of the comments concerning the organisation of the session and the content of the discussion itself is to record the session. The observer can collect all the verbal expressions, while focusing on the most important aspects (which will be developed in the report of the session) and intervening in the session when the moderator omits questions or when the questions are not properly answered.

This is particularly recommended for focus group conducted in a local language, and where notes are generally taken in another language (French, English, Spanish, etc.). Indeed, the session's verbatim record is often lost during this instant translation, and the recording fills the missing parts of the notes. The evaluator should plan an additional working day in the focus group schedule, dedicated to the transcription of the notes.

HOW ARE THE FINDINGS ANALYSED?

This stage is highly dependent on the quality of the focus group's organisation.

In structured focus groups

The best way to analyse findings from structured focus groups is to gather all the answers and identify the people in the group who have defended certain opinions on a particular question, and the people who have opposed to these points of view.

Thereafter, the findings can be easily organised into sub-groups and the evaluator can use them directly.

In less structured focus groups

These focus groups tend to be a conversation between several people. The participants express many views but can agree on only a few conclusions. Thus, the analysis of the findings is similar to the one carried out for interviews.
The findings of the focus groups represent the sum of individual opinions and can be used as such by the evaluator. But focus groups have more advantages than interviews: they provide individual opinions expressed publicly, and consequently, they are more substantial because other participants can immediately challenge them.

WHAT ARE THE FINANCIAL REQUIREMENTS FOR ITS USE?

The full cost of a focus group can be fairly small. However, cost varies with the number of sessions and the difficulties of organisation (distances, etc.).
As a whole, a focus group is less expensive than a survey.

Elements of cost to be taken into account

In addition to the remuneration of the evaluators' working days and their travel expenses, the budget must also include:

• The remuneration of the moderator, if the evaluator is not able to conduct focus groups himself/herself
• The local moderator's training time and its cost
• The remuneration of the mediators, if long-distance sessions are scheduled
• The remuneration of the interpreters
• The participants transportation expenses
• The catering and logistical expenses (room rental, etc.)

Should the evaluator pay for participant's time?

Some evaluators provide the participants with a per diem, in addition to the transportation and catering expenses. The award of a per diem may lead to bias in the responses of participants, which can undermine the validity of the findings.

However, depending on the participant's occupation (for example, a shopkeeper or a farmer), the corresponding loss of income of a session lasting half-a-day or a full day can be financially damaging.

Thus, the evaluator should be pragmatic and provide only a limited number of participants with financial compensation. The compensation should be exceptional and the amount low, because the respondents should not participate in focus groups for money, or get the impression that their opinion is being "bought".

WHY USE A FOCUS GROUP IN THIS CONTEXT?

In country/region evaluation, the main challenge is not how to collect information, but to determine what information to look for, from whom, about a wide range of questions, and during a limited period of time.

What are the specific difficulties in country/region evaluation?

The selection of beneficiaries and the understanding of their strategic goals

One of the country/region evaluation's purposes is to determine precisely the categories of stakeholders who have benefited from the implemented policies in some way. Thereafter, the evaluation must provide an analysis of the various stakeholders' strategies, and rationales considered, within the context of the evaluated policies.

Numerous stakeholders

In country/region evaluation, setting up meetings with numerous and varied stakeholders (such as institutional officials, implementation operators and agents, beneficiaries, etc.) can be challenging.

Sample preparation

The sampling of individuals and institutions from whom the evaluator collects information and points of view can yield insufficient results. Indeed, for reasons of timetable management, this selection is often carried out at the beginning of the evaluation process, and may not be adapted to the evaluation's purpose and strategic problems.

Lack of time

Usually, the evaluators spend most of their time interviewing officials, to the detriment of impact studies with beneficiaries which are particularly difficult to carry out in country/region evaluation.
In addition, time and budgets allocated to the evaluator are usually insufficient to conduct a series of focus groups with all categories of stakeholders potentially or actually involved in the evaluated policies.

To what extent is the focus group adapted to these difficulties?

Historically, focus groups have seldom been used in country/region evaluation.
However, evaluators should consider using focus groups more often at each stage of the evaluation, and particularly in the definition of strategic topics and issues, and for the selection of target groups.
In a country/region evaluation context, focus groups can be used to:

• Identify and analyse major topics and issues
• Provide a wealth of information, quickly collected
• Conduct an impact study

Dependant on the nature of the evaluation (ex-ante, mid-term or ex-post), the evaluators deal with these three tasks in a different way. For example, in an ex-ante evaluation, a focus group is conducted to highlight the needs and priorities of the stakeholders who may be affected by a programme or a policy. In this case, impact studies are not a priority. They may be useful, though, to assess past policies or the impact of funding institution's activities, with a view to improving the drafting of new policies.
In a mid-term evaluation, a focus group is conducted to collect opinions and suggestions from operators and beneficiaries. In this context, its purpose is to detail any positive elements or dysfunctions, with a view to supporting recommendations for the policies in process.
As focus groups are flexible and multifaceted, they can be adapted to the various types and stages of the evaluation. However, their use should be clearly defined, in order to meet the evaluation's requirements.

HOW TO TAKE ADVANTAGE OF THE MAIN FEATURES OF THE FOCUS GROUP?

In response to the particular difficulties of country/region evaluation, focus groups concentrate on three objectives.

Atool for defining and analysing major topics and issues

In this context, focus groups can be combined with interviews.

First stage: interviews

The evaluator carries out highly focused interviews, in order to better comprehend the purpose of the evaluation. They are organised with people:

• Who have an overall perspective of the policies, objectives and impacts to be evaluated
• Whose interest and activities are diverse, in order to avoid bias in the evaluation process arising from over-reliance on officials who are responsible for policy implementation and monitoring

Second stage: focus groups

On the basis of the findings of these preliminary interviews, the evaluator conducts one or more focus groups with:

• Stakeholders affected by and officials in charge of policy definition and implementation
• Stakeholders sharing an overall view of the policy impacts

As a consequence, the categories of people who have been interviewed individually at the first stage are increased.
The objective of this second stage is to investigate collectively the interview findings and to explore a first draft of the strategic analyses which will support the evaluative questioning.
This type of focus group can be conducted at different geographical locations and on a variety of sectors. It provides the evaluator with a speedy means of interacting with a large number of people.

A broad and rapid information collection tool

This tool obtains information:

• From different key stakeholders about the policy context
• From programme operators about programme implementation

A tool for the analysis of impact

Its major objective is to determine which categories of stakeholders have been affected by the policy (positively or negatively).
These categories of stakeholders should not be limited to the programmes' final beneficiaries. The evaluator should also consider stakeholders who have held various responsibilities for the policy, those who have benefited from it, and those who are concerned with it in other ways.

Beneficiaries not identified at the policy definition stage may also be included in the evaluation process. Consequently, in country/region evaluation, impact analysis also deals with a series of intermediary stakeholders, from decision-makers to final beneficiaries.
Designed as a tool to analyse the impact of implemented policies and to complement series of interviews, the focus group can be conducted in three ways:

Three types of focus groups
 

• Examples