Evaluation methodological approach

What is an expert panel?

Definition

An expert panel usually comprises independent specialists, recognised in at least one of the fields addressed by the programme under evaluation. The panel specialists arrive at conclusions and recommendations through consensus. Depending on the project proposal, these recommendations deal with the implementation or the impact of a programme, or part of it. When consensus is not reached for particular questions, the panel must report on the various perspectives of the experts.

The expert panel is specifically appointed for the evaluation, and in conformity with standard procedures. The panel holds meetings and provides conclusions and recommendations in accordance with a precise and replicable working plan, which accounts for its reliability.

This tool is therefore designed to take advantage of the experts' knowledge in assessing policies, programmes and projects implemented in the field of their expertise.

Various types of expert panels in evaluation

International funding institutions such as the World Bank, the European Commission, Scandinavian countries, Canada and the United States regularly appoint expert panels to evaluate programmes.

In environmental evaluations, funding institutions such as the World Bank use expert panels to assess the quality of the service providers' work and suggest improvements. These panels examine the evolution of the project studies, from their preliminary stages to completion, including the first years of implementation.

The Delphi Method is another type of expert panel used for evaluation, based on an anonymous and repeated postal survey with experts (see Means Documents).

Where does the expert panel come from and how has it evolved?

The panel's origin

The concept of the expert panel originates in the research field. It derives from the peer review of scientific work processes or programmes, where expert panels have developed as an evaluation tool.

Its evolution

Two main developments over the past few years are noteworthy.

• The panel's composition continues to diversify. Whereas the expert panel's missions were originally focused on limited areas of inquiry, their content and the panel's composition have been progressively diversified. Thus, in addition to programme specialists, economists, evaluation specialists and representatives of programme users are now appointed to panels, with a view to assessing all aspects of the evaluation.
• The design of the panel's tasks has become more professional. The emergence of rigorous methodologies (and additional studies conducted by independent consultants verifying the panel's conclusions) has increased expert panel's credibility and changed expert panels into a reliable evaluation tool. These improvements have progressively given the expert panel the characteristics of a professional evaluative tool.

What are the pre-conditions for its use?

Expert panels are appropriate for many evaluation situations, and particularly for:

• Subjects which are well defined and require the advice of highly specialised experts. This is common in the research field, where the European Commission often uses this tool during evaluations.
• Highly focused subjects (such as assistance to a small country evaluation) which do not require expensive allocations of resources.
• Highly complex subjects (such as budget support), which are difficult to assess at a reasonable cost using other tools. In these cases, the point of view of experts specialised in the subject and/or in the country can constitute credible information and evidence for the evaluation.

Various tasks of an expert panel

With which tools can the expert panel be combined?

The expert panel can be combined with almost all the usual evaluation tools.

As well as combined with other tools, the expert panel's work is sometimes complemented by external studies. This usage has become widespread in fields such as research evaluations because it eases the task of the experts and provides them with information about the programme under evaluation. External studies include preliminary studies, surveys, database analyses.

The advantages

The experts' knowledge of the subjects under evaluation is the principal advantage of this tool. It fosters:

• Significant reductions in time allocations, when compared with other evaluation tools
• Cost effectiveness compared with other tools, due to reduced time allocation
• Credibility of the conclusions, due to the experts' reputations
• Adaptability to a variety of situations encountered in evaluation

The limitations

The tool's limitations which should be minimised essentially derive from a series of risks:

• Because the panel must come up with consensus-based conclusions, its organisation tends to eliminate minority points of view and tone down conclusions.
• The point of view of a 'dominant' expert can be over-influential within the panel, to the detriment of other perspectives and the report.
• When the panel includes members who are the only specialists in the fields to be studied, an inappropriate empathic bias may occur. This could be exacerbated when the experts are acquainted with the actors of the field under evaluation. To mitigate this situation, appointments member of the panel must be reminded of the need for independence.
• Experts have a tendency to go beyond their field of competence and the credibility of the conclusions can be adversely affected. Therefore, the panel's work should be strictly focused.

Even when these risks are controlled, social science detractors remain sceptical about the reliability of expert panel's conclusions.

The time span

One of the advantages of an expert panel is in its speedy assembly process. For an evaluation, only 3 to 6 months work needs to be scheduled, and even less time for panel advice on a technical field within an evaluation.

Human resources

The core aspect of an expert panel is the issue of human resources. Experts must have recognised expertise in the field under evaluation, be independent of the programme being assessed, be able to work in a group and be available for a continuous work throughout the evaluation.

Financial resources

The expert panel is known for its cost-effectiveness, but in the case of travel into remote countries and field visits, the expenditure allocated for experts' salaries and expenses will need to be increased.

Budget line items normally taken into account while preparing estimates are as follows:

• Salaries for the experts and the technical writer. If necessary, estimates may include salaries for subcontractors responsible for the panel's external studies.
• Communication and travel costs, publication, and dissemination costs related to the reports
• Translation costs, if required

Reasons for establishing an expert panel

The use of an expert panel in country/region evaluations can be helpful in several situations, such as:

• Studying very specific fields requiring a high level of competence (such as assistance for the research field and high technologies)
• Studying subjects for which other tools are difficult to implement (for example, budget support)
• Carrying out limited-scope evaluations (such as assistance to small countries)
• Assisting the evaluators in their conclusions on a subject in complex evaluations
• Providing assistance in the drafting of final conclusions relating to the possible impacts of a programme in ex ante evaluations

Conditions for its use in country/region evaluations

No specific conditions are required for the use of expert panels in country/region evaluations. However, experts must be familiar with the context of the country assistance under study.

Examples of its contribution to country/region evaluations

No example of its contribution to country assistance has been found yet, but this tool is frequently used in programme evaluations in Western countries. A series of examples highlight the use of expert panels in the development of terms of reference documentation, recruitment profile processes, samples of reports.

Core criteria of the expert panel's composition

Professional experience

The pre-requisite for the expert's selection is his/her professional experience. He/She should have specialised in the field under evaluation, and be recognised and respected by his/her peers. The credibility of his/her conclusions is highly dependent on these elements.

Independence

Experts must be independent of the programme under evaluation, because they should not be judge and judged. Thus, experts having a direct conflict of interest (such as experts participating in the programme or belonging to a body which benefits from it) should not be appointed.

However, securing sufficient independence is difficult in fields where there are few acknowledged experts. In this case, the panel, and not the experts, must be independent, and differing points of view must be dealt with in an even-handed way.

In addition, the experts' appointment to the panel is personal, which means that they do not represent their home institution and cannot be substituted on the panel.

Ability to work in a group

The ability to work in a group, listen to other experts and be open-minded is an essential criterion. Otherwise, working conditions may quickly turn out to be unmanageable, which would impede the panel process.

Other criteria

In addition to these core criteria and specific requirements relating to the expert's profile, some commissioning agencies may have their own views on the panel's composition (such as the expert's nationality, a balanced representation of differing points of view, the participation of specific categories of panellists such as beneficiaries, consumers).

Ideally, experts should also speak the language used in the area subjected to evaluation.

Selection process for the panel members

Procedures for the recruitment of experts

Varying with the importance of the task and the complexity of the themes, the recruitment of panel members can be relatively straightforward and speedy, or require a time-consuming selection process.

Straightforward selection

In a straightforward selection, the evaluation managers have access to a list of acknowledged experts in specific fields, and limit their selection process to ensuring the expert's independence regarding the programme under evaluation.

Gradual selection

Gradual selections have become a common procedure. Preferred profiles of experts are developed with respect to the topics under scrutiny in the evaluation. These profiles are critical to satisfactory recruitment to meet the evaluation's needs.

If the panel's intervention only focuses on technical issues, the expert's lack of independence within the fields under consideration will usually have minor impact on the evaluation, because the evaluator is responsible for the main tasks.

However, if the panel has to carry out a significant part of the evaluation, it should include an expert experienced in evaluation processes and a socio-economist, in addition to experts specialised in the topics under review. Depending on the nature of the mission assigned to the experts, additional fields of competence may need to be proposed.

Development of the panel profile

The Royal Society of Canada details elements to be taken into account in developing the panel profile, as follows:

• Project scope: will the study be limited to technical problems, or will it address broad issues of public policy?
• Degree of controversy: do the problems to be addressed have alternative resolutions which are controversial? Do they affect parties who have strong emotional, political or financial stakes in the outcomes?
• Technical support: will the panel's conclusions and recommendations be based on data analysis or on the panel's expert judgement?
• Uncertainties: will the panel's conclusions discuss the uncertainties?
• Number of required disciplines: do the issues to be evaluated involve a single discipline or are they interdisciplinary?

The selection process

Once these profiles have been developed, the institution managing the evaluation should establish a "long list" of experts and remove experts with possible conflicts of interest. Thereafter, the institution contacts the selected experts.

Depending on the number of experts required (5 to 10 in most cases), the institution should pre-select a larger number of potential experts (2 to 3 times more than the final list), in order to ensure the availability of experts. Within this "long list", some funding institutions may determine nominees and alternates. This procedure is also recommended for the nomination of the chairman and key experts.

Once the selection process is over and the experts are recruited, the commissioning agency (or the experts) proceeds with the appointment of the chairman. As the role of the chairman is essential for the efficient working environment of the panel, the importance of this appointment is paramount. He/She will be responsible for setting the tone and rhythm of the panel's work.

Secretarial work should be the responsibility of a technical writer, whose ongoing availability should compensate the pressurised timetables of the experts. The tasks of the technical writer include the production of working reports, the incorporation of suggestions, and, if required, the monitoring of external studies.

A precise definition of the panel's field of work

Evaluators must take time to explain to the experts the context of their work and provide them with information about the programme, the procedures, and evaluation methodologies. In doing so, evaluators help the experts formulate their conclusions with a full knowledge of the environment of area assessed.

Usually, the content of the panel's working sessions is confidential.

Availability of documentation for the experts

Terms of reference

Terms of reference are often provided to ease the panel's working arrangements. They should specify:

• The nature of the evaluation
• The function of the panel within the evaluation and how this work is to be organised
• The questions submitted to the experts or the nature of their investigations (such as points of view about the relevance, the coherence, and the efficiency of the programme)
• The available data (for example, relating to the implementation of the programme) and the means at the disposal of the panel (including the possibility to go on-site)
• The content and timetable for the reports

Reference interview guides

In evaluations where calls for expert panels are systematic, developing procedures for the management of the panel process, as well as interview guides may ease the panel's work (for example, during interviews with officials, field study, interviews with beneficiaries). These procedures can also benefit repetitive evaluations, through the standardisation of reports, which allow for comparisons.

Interview guides may be provided to evaluation teams unfamiliar with evaluation process, with a view to assisting them in their tasks.

Fundamentals for the expert panel's work

There is no unique working process, and the expert panel should be encouraged to plan and implement its own workplan. Experts can focus their work on documentation and sessions, or broaden it to include meetings with project managers, field visits, implementation of surveys, etc. The choice of working process is highly dependent on the area studied, the experts mission, and the information and resources at the disposal of the experts.

Experts are expected to investigate and analyse the assigned topics and present their conclusions in a written report. The quality of the drafting and writing of the report is crucial and must be given careful attention.

Expert panels are usually expected:

• To provide studies incorporating the scientific and technical standards of the relevant fields
• To show complete independence from the programmes under review, which avoids any conflict of interest
• To reach a consensus in their conclusions and recommendations

Some commissioning agencies believe that the search for consensus usually results in the experts producing an anodyne and unrealistic report. They require a report highlighting the experts' different points of view and the reasons for these differences.

Confidentiality of the panel's debates and intermediate findings is another rule impacting on the panel's working arrangements.

Guidelines for the panel sessions

The first panel session

The content of the first panel session derives from the terms of reference. This session must result in the experts having a full understanding of their role in the evaluation.

During this session, the applicable methodology for the management of the panel's work must be discussed and validated. The discussion usually focuses on:

• The panel's organisation and the role of each member
• The type of investigation, the data collection methodology, and details of each panellist's task (such as field visits)
• The intervention work programme, the organisation of future sessions and their contents

During this session, panellists should be reminded of the general rules (such as independence and consensus) because although experts are generally familiar with the topics under study, they are often less well informed about evaluation principles.

The possibilities of conflict of interest with the programme to be studied should be closely examined, discussed and resolved during this session.

The next sessions

The following sessions (ranging from 3 to 5) will be directly linked to the panel's work. They will systematically deal with:

1. The work carried out since the previous session
2. Findings from investigations which are completed or in process
3. Problems encountered (such as difficulty of collecting data, problems concerning the intervention timetable, the budget)
4. Progress in editing the various documents, the review process, and quality control over these documents
5. The tasks to be achieved before the next session and its envisaged content

With a view to ensuring the confidentiality of the panel's work, certain commissioning agencies recommend that records, summaries and intermediary reports of the sessions are destroyed. Only the final report is kept as the formal output required from the panel.

The organisation of the mission

The organisation of the mission depends on the panel's tasks. Most of the experts will be unfamiliar with evaluation techniques and may live far from the session's location. Thus, the production of the expert panel work programme should be scheduled well in advance, preferably as soon as the mission starts. This work programme should be adhered to whenever possible.

Experts responsible for tasks between two sessions, such as field visits, will be expected to work at least in pairs, in order to avoid bias of interpretation or empathy (the limitations). The formation of small groups should reflect the various points of view represented on the panel.

The panel chairman plays a crucial role. He/She guides the study panel, proposes the working arrangements, records findings, encourages contributions, facilitates debates and is the chief spokesperson for the panel. The quality of the working arrangements often depends on the chairman's leadership.

The various roles of the panel chairman

The chairman as Panel Facilitator

• At the first session, the chairman guides the panel to an agreement on a workplan and report architecture (as a working outline)
• Thereafter, the chairman schedules the work of the panel, the production of the documentation and its revision
• Throughout the study, the chairman ensures that each expert takes part in the working process and understands the evaluation's content.
• Given the essentially diverse composition of a panel, and the often considerable differences in initial views, the chairman must steer the panel's progress toward consensus on the range of issues involved. Fairness and flexibility should be employed.

The chairman as Report Architect and Integrator

The chairman guides the study, defines methodologies, reviews outputs, ensures that timetables are respected, and records the findings of the panellists, which includes:

• The provision of a critical overview to the panellists' outputs, designed to improve the debate rather than to control it
• The drafting of the successive versions of the report with the assistance of the technical writer and the commissioning agency

The chairman as Project Manager

The chairman ensures that the available resources are sufficient and properly employed throughout the study. He/She is in permanent contact with the commissioning agency on financial and technical issues. If sub-contractors work for the panel, the chairman is responsible for the management of their studies, the supervision of their progress and their successful completion of their work.

He/She ensures that the panel's sessions have been properly prepared by the technical writer, and that all documentation and means required for their participation in the sessions are provided to the experts in a timely fashion.

If the mission requires a readjustment of its budget, its time allocation or its objective, the chairman presents this to the commissioning agency in order to reach an agreement.

The chairman as Spokesperson

The panel will need to be represented in various bodies (such as monitoring committees) and possibly in meetings with the commissioning agency and the press. As it is impracticable to gather all the experts for these meetings, the chairman serves as the spokesperson for the panel. He/She may delegate certain tasks to other panel members, but he/she should conduct the most important meetings.

Synthesis of the panel's study

At the end of their mission, the experts report on their work. The report, which supports the experts' contribution to the evaluation, is the only output from the panel which is made available to the commissioning agency. Consequently, the report should be carefully prepared.

Guidelines for the final report

The report's structure depends on the nature of the mission. In technical or scientific missions, the report should at least include:

• An executive summary, which is not too technical if possible
• The mission's terms of reference
• The composition of the panel describing relevant information about the background of the experts (in an appendix if required)
• The evidence gathered and reviewed. The assumptions made and their impact on any conclusions should also be stated.
• The analyses carried out
• The conclusion of the experts in the context of the report's consensus findings. If consensus has not been reached, the report should state the consensus points, the elements of disagreement, and then explain them.

Reaching a consensus is the most challenging task for managers of expert panels, because a consensus strengthens the value of the panel's conclusions. In this context, the role of the chairman is crucial in seeking consensus, or formulating the final position of the panel, even if it include some dissenting views.

• Examples

General references

• Expert panels: manual of procedural guidelines. The Royal Society of Canada. 1998.
• "Strategic management of the research and scientific field", Chap. 1 Editor Economia. 1995.
• Means Documents.

Specific references for the use of expert panels in country/region evaluations

• Expert panel report, dealing with the Megascience forum evaluation, conducted by the OECD, 21 October, 1998.
• Expert panels report on pesticides, New Zealand Radio, 11 September, 2000.