Synthesis phase (phase 3)

Findings

The evaluation team formalises its findings, which all derive from facts, data, interpretations and analyses. Findings may include cause-and-effect statements (e.g. "partnerships, as they were managed, generated lasting effects"). Unlike conclusions, findings do not involve value judgements. 

The evaluation team proceeds with a systematic review of its findings with a view to confirming them. At this stage, its attitude is one of systematic self criticism, e.g.:

• If statistical analyses are used, do they pass validity tests?
• If findings arise from a case study, do other case studies contradict them?
• If findings arise from a survey, could they be affected by a bias in the survey?
• If findings arise from an information source, does cross-checking show contradictions with other sources?
• Could findings be explained by external factors independent from the project / programme under evaluation?
• Do findings contradict lessons learnt elsewhere and if so, is there a plausible explanation for that?

Conclusions

The evaluation team answers the evaluation asked through a series of conclusions which derive from facts and findings. In addition, some conclusions may relate to other issues which have emerged during the evaluation process. 

Conclusions involve value judgements, also called reasoned assessments (e.g. "partnerships were managed in a way that improved sustainability in comparison to the previous approach"). Conclusions are justified in a transparent manner by making the following points explicit:

• Which aspect of the project/programme is assessed?
• Which evaluation criterion is used?
• How is the evaluation criterion actually applied in this precise instance?

The evaluation team strives to formulate conclusions in a limited number so as to secure their quality. It either clarifies or deletes any value judgement which is not fully grounded in facts and entirely transparent. 

The evaluation team manages to use evaluation criteria in a balanced way, and pays special attention to efficiency and sustainability, two evaluation criteria which tend to be overlooked in many instances. 

The evaluation team synthesises its conclusions into an overall assessment of the project/programme, and writes a summary of all conclusions, which are prioritised and referred to findings and evidence. Methodological limitations are mentioned, as well as dissenting views if there are any. 

The evaluation team leader verifies that the conclusions are not systematically biased towards positive or negative views. He/she also checks that criticisms may lead to constructive recommendations.

Recommendations and lessons

The evaluation team maintains a clear distinction between conclusions which do not entail action (e.g. "partnerships were managed in a way that improved sustainability in comparison to the previous approach") and other statements which derive from conclusions and which are action-oriented, i.e.

• Lessons learnt (e.g. "the successful way of managing partnerships could be usefully considered in other countries with similar contextual conditions")
• Recommendations (e.g. "the successful way of managing partnerships should be reinforced in the next programming cycle").

Recommendations may be presented in the form of alternative options with pros and cons. 

As far as possible, recommendations are :

• Tested in terms of utility, feasibility and conditions of success
• Detailed in terms of time frame and audience
• Clustered and prioritised.

The evaluation team acknowledges clearly where changes in the desired direction are already taking place, in order to avoid misleading readers and causing unnecessary offence.

Draft report

The evaluation team writes the first version of the report which has the same size, format and contents as the final version. Depending on the intended audience, the report is written:

• With or without technical terminology.
• With either a summarised or a detailed presentation of the project/programme and its context.

In general, the report includes a 2 to 5-page executive summary, a 40 to 60-page main text, plus annexes.

The evaluation team leader checks that the report meets the quality criteria. The report is submitted to the person in charge of the quality control before it is handed over to the evaluation manager.

The evaluation team leader and the evaluation manager discuss the quality of the report. Improvements are made if requested. 

In the case of a multi-country programme, the country notes are published as part of the overall evaluation exercise in annexes to the synthesis report (editing is therefore required).

Quality control

The evaluation manager receives the first version of the final report. The document should have the same format, contents and quality as the final version.
The evaluation manager assesses the quality of the report on the basis of an eight-criteria assessment grid. The assessment is double-checked by a second person.

The quality assessment should enhance the credibility of the evaluation without undermining its independence. It therefore focuses on the way conclusions are substantiated and explained and not on their content. The quality assessment must not be handled by those who are involved in the evaluated project/programme.
The evaluation manager and the evaluation team leader discuss the quality assessment. Improvements are requested if necessary.

Discussion meeting(s)

Discussion of draft report

The evaluation team presents the report in a reference group meeting. The presentation is supported by a by a series of slides which cover:

• Answered questions and methodological limitations
• Overall assessment, conclusions and lessons learnt
• Recommendations.

Comments are collected in order to:

• Further check the factual basis of findings and conclusions
• Check the transparency and impartiality
• Check the utility and feasibility of the recommendations

Discussion seminar

The evaluation manager submits the draft report to the reference group members for consultation. If appropriate, he/she convenes and chairs a meeting where the report is presented and discussed. Special attention is paid to the utility of conclusions and the feasibility of recommendations. At this stage, the evaluation manager may decide to convene a discussion seminar with a wide range of stakeholders. The purpose would be to discuss the content of the conclusions and the utility of the recommendations in the presence of the evaluation team. Attendance may include the delegation staff, national authorities, civil society, project management, other donors and/or experts. Participants are provided with an updated draft report.

Finalising the report

The evaluation team finalises the report by taking into account all comments received. Annexes are also finalised in one or the following forms:

• Printed attachments to the report.
• Annexes on CDROM.

The report is printed out according to the instructions stated in the terms of reference. 

The evaluation team leader receives a final quality assessment from the manager. If necessary, he/she writes a note setting forth the reasons why certain requests for quality improvement have not been accepted. This response will remain attached to both the quality assessment and the report.

Final report

Comments are taken into account by the evaluation team in a new version of the report. The evaluation manager also receives an electronic version of the slides presented by the evaluation team.
He/she checks that the comments received have been taken into account in an appropriate way, and that the report is ready for dissemination, including the full set of annexes.

He/she runs a final quality assessment against the eight criteria of the assessment grid, writes qualitative comments for all criteria, and decides upon the overall quality score.

The evaluation manager sends the final version of the report and quality assessment to the reference group members, and thanks them for their contribution.

Check lists

See for inspiration the Check lists for Geographic, thematic and other complex evaluation.