Evaluation methodological approach

What is this?

Establishing quality check-points at the main phases of the process, defining the responsibilities of quality assurance, and establishing the rules of the game in case of quality defects. 

Quality assurance is organised on a step-by-step basis: successive aspects of quality are secured at each step, thus creating a sound base for the subsequent step.

What is the purpose?

• To enhance the conclusions of the evaluation in users' eyes by showing that this is impartial and rigorous work that meets professional standards.
• To gradually construct quality and thus avoid discovering a quality defect in the final report stage.
• To clarify relations between the evaluation manager, the reference group and the external evaluation team.

Rules of the game

The rules of quality assurance are specified in the terms of reference and pointed out when the external evaluation team is engaged. 

These rules concern:

• Approval of documents
• Quality criteria
• Dissemination of the quality assessment of the final report

The quality assurance process benefits from the contribution of all actors whilst limiting the potential conflicts that might arise between them, e.g.

• The actors who hold responsibilities in the evaluated intervention (the evaluees) are in a good position for assessing the relevance of data collected and the fairness of interpretations. Their assessment may, however, be distorted by a confirmation bias. They should therefore be given only an advisory role, e.g. through their participation in the reference group.
• The evaluation team leader has a major role in assuring quality, especially in designing an adequate method, and in securing the accuracy and fairness of the report. He/she may, however, overweight the views of powerful stakeholders, especially those who are likely to commission other evaluations in the future. The evaluation manager's quality assessment helps to prevent such a risk.

Key steps

Inception report

In the inception report stage, quality assessment is used to ensure that the evaluation team has clearly understood the scope and aims of the evaluation, as well as the logic of the evaluated intervention and the questions to answer.

First phase report (desk)

In the first phase report stage (desk), quality assessment is used to check the quality of the documentary analysis as well as the appropriateness of the proposed method for the field phase, including the indicators, data sources and analytical approach.

Final report

In the final report stage, quality assessment checks whether the evaluation provides valid and impartial answers to the questions asked, and whether the form of the report is suited to the targeted users' needs. 

At this stage the quality assessment concerns first the draft version of the report and then the final version.

Key players

It is expected that all key players involved in the quality assurance process have a sufficient evaluation capacity, except reference group members. If a key player is not qualified enough, he/she ensures that appropriate support is provided.

Evaluation team leader

The evaluation team leader is primarily concerned with preventing major risks that threaten quality, e.g. overlooking a major question, not consulting an important stakeholder, elaborating upon unreliable data, judging in a partial way. 

He/she also ensures that each report is submitted to a detailed quality check before it is released to the evaluation manager.

Quality assessor(s)

The company or consortium in charge of the evaluation contract appoints one or more quality assessors who carefully check each report for quality before it is released to the evaluation manager. The quality assessor should be experienced in evaluation and should not belong to the evaluation team.

Evaluation manager

The evaluation manager has the ultimate responsibility for assessing whether or not the methodological quality of a report is sufficient for allowing the next step to be taken. 

In this process he/she takes advice from the reference group members, and relies upon technical support as far as necessary.

Reference group

Reference group members receive all draft reports for comments. They may comment on the factual bases substantiating the evaluation, which is a contribution to quality assurance. When their comments apply to the content of conclusions, they are passed on to the evaluation team which freely decides on whether or not to take them into account.

What is this?

Formal recognition by the evaluation manager that a delivered document has an adequate content and meets the applicable quality criteria in methodological terms.

What are the approval steps?

Generally, a document is approved in the following way:

• The manager checks that the document has the required form and content, and has no major quality defect. If it does, he/she immediately requests a new version.
• The manager asks for the opinion of the individuals or services that have to be consulted, either at a meeting (reference group) or by email. He/she gives a deadline after which there is no guarantee that comments will be taken on board.
• The manager lists his/her own requirements regarding quality improvements, attaches other requests received, distinguishes requests applying to the methodological quality from requests applying to the substance of the text, and sends all the comments to the external evaluation team.
• The evaluation team takes into account all the requests for quality improvements, either by amending its document accordingly or by mentioning in an annex how the request will be treated at a later stage, or else by explaining in an annex why the request has not been treated.
• A new version of the document is soon presented to the manager who accepts or refuses it.
• When approval is required, it is also a prerequisite for moving on to the next stage.

Which documents are approved and by whom?

The following documents require approval:

• Inception report
• First phase report (desk)
• Pilot mission report (when the evaluation includes missions in several countries)
• Draft final report
• Final report

Approval is generally performed by the evaluation manager and may be confirmed by his/her superior. 

The reference group members validate the set of evaluation questions, i.e. group members formally accept, or at least do not reject, the set of evaluation questions. If necessary, divergences are resolved by the evaluation manager. In that case dissenting views are noted in subsequent reports. 

Rules of approval and independence

The approval process does not contradict the external evaluation team's independence. In this respect the requests for amendments made during the approval exercise need to be differentiated:

• Comments concerning methodological quality have to be taken into account. If the evaluation team considers that it cannot take a demand into consideration, this should be fully justified. An assessment of the methodological quality is attached to the final report.
• Comments concerning the substance of the document, statements and conclusions are freely taken into account or rejected by the evaluation team. The team must nevertheless, in a note or in the annex, mention the requests not taken into account and comment upon its decisions in this respect.

What is this?

Assessing the quality of an evaluation is a multifaceted task. The commissioning body and the stakeholders are in the best position to assess whether their needs have been treated impartially. The experts and those who implement the intervention are in a good position for assessing the relevance of data collected and the fairness of interpretations. It is the evaluation manager's role to assure that the evaluation has sound methodological and factual bases. The manager receives a quality assurance support if he/she lacks training or experience.

Who can play this role?

Within the EC headquarter Evaluation Unit, managers have the required training and skills to assess the methodological quality of reports. All evaluations are monitored by two members of the Unit, both of whom participate in quality assessment. 

In the case of devolved evaluations, the manager uses the quality criteria proposed by this site as well as the indications given to fill out the quality assessment grid, if necessary with the help of the Evaluation Unit. 

The Evaluation Unit can perform cross-cutting quality assessments autonomously.