| Each step of the preparatory phase is described according to the respective role of : |
 |
The evaluation manager |
 |
The external evaluation team |
| |
|
Initial approach |
|
As far as necessary, the commissioning service clarifies in writing precisely what is to be evaluated, and who the main intended users of the evaluation will be.
An evaluation manager is appointed within the commissioning service He/she works under the responsibility of the hierarchy.
|
|
Preliminary data collection |
The evaluation manager reads the basic documents (fiche, logical framework, review, monitoring report, etc.), and has informal talks with a few key informants.
If no logical framework is available, then the logic of the project has to be reconstructed by the project/programme manager. |
|
Constituting the reference group |
|
The evaluation manager identifies the services and other interested bodies to be involved in the evaluation through a reference group.
Composition and role of the reference group
The reference group involves the project/programme management and the relevant EC services. If the evaluation is conducted in the partner country, then the group may involve development partners, experts, non state actors, and other qualified participants. Membership should remain manageable (no more than 10). The group is chaired by the evaluation manager.
The reference group discusses and comments on all intermediary documents, generally at their draft stage: terms of reference, evaluation team's proposal, evaluation questions, work plan and debriefing of the field phase, and final report. It generally has an advisory role but may be required to approve the evaluation questions.
A reference group has substantial advantages in terms of access to information, accuracy of interpretations, and ownership of conclusions.
Meetings are moderated with the aim to prevent a waste of time and to deal with conflicting views in a constructive manner. Timely circulation of working documents and minutes is essential.
A note is sent to the services and bodies invited to join, explaining the role played by the reference group. |
|
|
Preparing the terms of reference |
|
The main issues to be studied are identified by the evaluation manager. If a good logical framework is available and still valid, the evaluation manager may refine the issues to be studied into evaluation questions.
Evaluation questions
At some stage in the evaluation process, a series of precise questions (no more than ten) is selected with a view to satisfying the needs of the evaluation users and to ensuring the feasibility of the evaluation. This site presents a set of typical questions.
By focusing the evaluation on key issues, the questions allow the evaluation team to collect accurate data, to deepen its analyses, to make its assessments in a fully transparent manner, and finally to produce a useful report.
Questions are written in a simple and precise way. As far as possible, they do not cover areas where other studies are available or in progress.
The set of questions is composed in such a way that the synthesis of all answers will enable the evaluation team to formulate an overall assessment of the project/programme. For this purpose, the set of questions covers the various levels of the logical framework and the seven evaluation criteria in a balanced way. |
The profile of the external evaluation team to be engaged is specified as regards professional competence, sector expertise, and field work capacity.
A ceiling is set for the overall evaluation budget and the availability of resources is secured.
The timetable is specified in line with institutional requirements if necessary. Alternatively, the deadline for delivering the report is fixed with a view to the needs of the intended users.
The evaluation manager writes a first version of the terms of reference (ToRs), possibly building upon the template attached to these guidelines.
The reference group members are consulted on the draft version. The evaluation manager finalises the document and proceeds to the engagement of the external evaluation team, via the applicable tendering/contracting procedure.
The external evaluation team is engaged via the applicable tendering/contracting procedure.
The candidate contractor prepares a proposal in response to the Terms of Reference (ToR) issued by the commissioning service.
|
|
Basic assumptions |
|
On the basis of the ToR and his/her own expertise, the author of the proposal formulates basic assumptions on:
- Areas requiring specific expertise.
- Possibility to mobilise consultants with the right profile in the country or countries involved.
- Number, nature and probable difficulty of the evaluation questions.
- Existence, quality and accessibility of management and monitoring data.
- Existence of previous evaluations which may be reused.
|
|
Tasks, expertise and budget |
|
Assumptions are made about the evaluation design, including analysis strategy and tools to be applied. The tasks are provisionally divided among:
- Consultants from partner country or countries and international consultants.
- Senior, medium, junior consultants.
- Experts in the sector(s) of the project/programme and professional evaluation consultants.
The core evaluation team members are identified and the absence of conflict of interest is verified.
Both the budget and the time schedule are specified within the framework of constraints set by the ToR.
|
Hiring local consultants
Local consultants may be entrusted with all or part of the evaluation tasks.
Benefits:
- Possibility to involve a local perspective in data collection but also in data analysis.
- Mastery of local language(s).
- Easy use of participatory approaches involving beneficiaries and targeted people.
- Flexibility of work plan and reduction of travel costs.
- Contribution to building an appropriate evaluation capacity in the partner country.
Risks:
- Conflict of interest.
- Difficulty of being independent from the Government in some countries
|
|
|
Proposal |
|
The contents of the technical and financial proposal are as follows:
- Understanding of the context, purpose, and intended users of the evaluation.
- Understanding of the themes or questions to be covered.
- Indicative methodological design.
- Core evaluation team members, their field of expertise and their role.
- Time schedule.
- Detailed price.
- CVs in the standard format and declarations of absence of conflict of interest.
- CV of an expert from outside the evaluation team who will be in charge of quality control.
|
|
Launch |
|
The evaluation manager receives the technical and financial proposal(s) prepared by the candidates. He/she checks that the proposal(s) covers:
- Understanding of terms of reference.
- Indicative methodological design.
- Detailed price.
- Planned schedule.
- Team members' responsibilities, CVs, and signed statements of absence of conflict of interest.
The evaluation manager takes part in the analysis of proposal(s). Reference group members are consulted, especially with a view to identifying and preventing conflicts of interest and risks related to independence.
Externality and independence
External evaluations are carried out by entities and persons free of the control of those responsible for the design and implementation of the project/programme.
Independence implies freedom from political influence and organisational pressure, full access to information and full autonomy in carrying out investigations and writing conclusions.
Externality and independence are meant to achieve credibility in the eyes of outside audiences, something which is particularly relevant if the evaluation is undertaken for purposes of accountability, for learning transferable lessons or for reallocating budgetary resources. Such evaluations are called "summative", as opposed to "formative" evaluations which are conducted for the benefit of those managing the project/programme, with the focus on improving their work and preferably with their full participation. Externality and independence may be of lesser importance in formative evaluations. |
The evaluation manager assesses the quality of the proposal(s) and verifies that the human and financial resources offered are suitable for the particular difficulties identified while preparing the terms of reference.
|
Check list for assessing the quality of a proposal
Capacity:
- knowledge and working experience in the field of evaluation
- (if relevant) demonstrated ability to carry out participatory approach
- technical and sectoral knowledge and expertise
- capacity to address essential cross-cutting thematic issues (e.g. gender equality, environment)
- experience in development cooperation, and EC cooperation in particular
- experience in the partner region, similar countries and/or the partner country
- adequate language skills
Understanding:
- understanding of the ToR
- understanding of the context
Management:
- proposed individuals have the time to successfully complete their task as planned in the schedule
- clear sharing of responsibilities and adequate leadership skills for effective team management and successful relations with partners and stakeholders
- commitment to strengthen evaluation capacity in the partner country
|
The evaluation manager engages the external evaluation team in the framework of the applicable tendering/contracting procedure. This is formalised in a contractual document, the date of which marks the beginning of the evaluation team's work.
|
