Field phase (2)

.

This section is structured as follows:

                                                                                                                                              .
Each step of the field phase is described according to the respective role of :
evaluation manager The evaluation manager
external evaluation team The external evaluation team
                                                                                                                                             . 
Preparation as from the previous phases
Technical and financial proposal
  • Assumptions about the methodological design and evaluation tools to be implemented.
  • Field phase budget.

Inception report

  • Draft methodological design, including evaluation tools to be applied in the field phase.
  • Countries to be visited (if the evaluation covers several countries and the terms of reference do not specify which countries are to be visited) and projects / support to be studied in more depth in the next steps.

First phase report (desk)

  • Final methodological design, and evaluation tools to be applied in the field phase

National / regional consultants' availability and skills permitting, the tools are thoroughly developed and tested at the end of the first phase (desk). Failing that, the finalisation and testing of tools become part of the field phase.

The field phase takes place in one or several countries. Unless stated otherwise, the following points are applicable to each country involved.

.

1420995755_mgteam.png

Preparation

.

The evaluation team leader organises the works:

  • Distribution of data collection tasks and responsibilities.
  • Definition of documents to be delivered and required level of quality.
  • Schedule of various tasks and agenda (or agendas per country) sent to the evaluation manager for information.
  • Conclusion of contracts with the national / regional consultants, if that still remains to be done.

The evaluation team leader briefs those who are responsible for the tasks:

  • Main predictable risks during field data collection works and how to behave in case a problem arises.
  • Need to strictly focus data collection on what is needed to answering the questions and to look for information relating to the selected judgement criteria (also called reasoned assessment criteria) and indicators as a priority.

The manager receives a detailed agenda prepared by the external evaluation team before the beginning of the field data collection mission(s).

Specific situation(s)

Evaluation covering several countries

In this situation, the first mission may be considered as a pilot mission and be used to test the data collection plan and the evaluation tools. 

The data collection plan may include a questionnaire sent to all of the delegations in the region. In this case, the manager writes an introduction letter signed by the head of his or her service.

.

1421077476_1418988054_teams.png

Initial meeting

.

The evaluation team proposes to the Delegation to hold an information meeting within the first days of the field data collection process in the concerned country. The following points are covered in the meeting:

  • Presentation and discussion of the work plan.
  • How to access data and key informants within the national government and within the other donors' missions.
  • How to deal with and solve potential problems.

In certain cases, such points have already been covered on the occasion of a previous visit of the evaluation team leader to the Delegation, for instance, during a pilot mission at the inception stage.

.

1421077476_1418988054_teams.png

Data collection and analysis

.

The evaluation team implements its field data collection plan. This guideline offers detailed advice for the implementation of major data collection tools: Interviews, Focus Groups, Questionnaire and Case Studies. 

The evaluation team leader ensures that the tools have been appropriately tested and coordinates their implementation. 

He/she is informed in case a problem arises and makes sure that the problems encountered will not weaken the answer to the evaluation questions. If appropriate, he/she appeals to the Delegation to help to solve the problem. If the solution to a problem requires the work plan to be adjusted, he/she informs the evaluation manager accordingly. 

One or several meetings are held among the evaluation team members to :

  • Monitor and coordinate the progress of the works.
  • Identify unintended effects or external factors.
  • Derive provisional answers to the questions.
  • Start cross-checking data.

The evaluation team gathers the documents containing the collected data and their first analysis. These documents may be:

  • Minutes and syntheses of interviews.
  • Minutes and syntheses of focus groups.
  • Responses to a questionnaire and analysis tables.
  • Case study minutes and monographs.

At this point, the analysis done is not only exploratory (e.g., searching for unintended effects or external factors) but also confirmatory (e.g., testing the assumptions made at the end of the previous phase).

.

evaluation manager

Follow-up

.

The manager is prepared to interact swiftly at the evaluation team's request if it encounters a problem during its mission (or one of its missions), provided that the problem cannot be solved with the help of the concerned delegation.

Specific situation(s)

Global sector or thematic evaluation

For each mission, the manager receives a country note. It is by no means an evaluation of the intervention in the country. The country note includes the following items:

  • Data collection method used and assessed reliability.
  • Brief description of the context in the visited country (relevant elements in connection with the evaluation questions).
  • Relevant facts in connection with the evaluation questions (without any analysis or judgement).
  • Synthesis of the main aspects of data collected in the country (for the delegation).
  • Elements likely to confirm/invalidate the assumptions to be tested with a view to feed overall analyses as well as the synthesis report.

.

1421077476_1418988054_teams.png

Quality control

.

The team leader checks the quality of the work, following in particular the quality criteria below:

  • Detailed presentation of the method actually implemented.
  • Compliance with work plan and/or justification for adjustments.
  • Compliance with confidentiality rules.
  • Self-assessment of biases and of the reliability of data.

The quality of the work is verified by an expert who is not part of the evaluation team. 

This site offers detailed instructions for quality control of major data collection tools: interviews, focus groups, questionnaire and case studies.

.

1421077476_1418988054_teams.png

Final meeting

.

The evaluation team offers the Delegation to hold a final meeting within the last days of the field data collection process in the concerned country. If held, this meeting deals with the following points:

  • Progress of the works, problems encountered and solutions adopted.
  • Most significant factual elements in the context of the partner country.
  • Preparation of a discussion seminar at the stage of the draft final report, if appropriate.

.

1421077476_1418988054_teams.png

Country note

.

In the case of a global thematic or sector evaluation, the team draws up a country note and forwards it to the evaluation manager. The first country may be used to test the evaluation method.

.

1420995755_mgteam.png

Debriefing

.

This stage concerns the visited country (or countries altogether). 

At the end of the field phase, the manager organizes a debriefing meeting with the reference group and the evaluation team.

The manager asks the evaluation team to report on the data collection work, to present both the problems encountered and the solutions found, to examine the reliability of the collected data and to present its main findings. This presentation is carried out with the support of (PowerPoint) slides and there are no minutes or approval. The presentation deals with the following points:

  • Reporting on field data collection works.
  • Problems encountered and solutions found.
  • Self-assessment of reliability of the collected data.
  • Particularly significant facts and findings in connection with the questions asked.
  • Next analysis and judgement steps.

.

Check lists

.

Content of a country note (themaic evaluation)

Introduction 

  • Context of the evaluation 
  • Purpose of the note 
  • Reasons for selecting the country as a case study country 

Main text 

  • Detail of the method applied for data collection and analysis.
  • Comments on limits and possible constraints. 
  • Short description of the sector in the country. 
  • Findings on the sector (focused on facts and not going into analysis). 

Conclusions at two levels:

  • covering the main issues on this sector in the context of the country (for the Delegation) and;
  • covering the elements confirming or contradicting the first phase (desk) assumptions (to be fed into the synthesis report).

Annexes (indicative)

  • List of people interviewed
  • List of documents consulted
  • List of projects and programmes specifically considered
  • All project assessment fiches
  • Detail of tools developed and applied, e.g. questionnaires, structure of interviews, minutes of focus group discussions, etc.)
  • Acronyms and abbreviations
  • etc..

Recommendations 

Length: maximum 20 pages (excluding annexes). The country note should include the following text:

  • This country note cannot be considered as a stand alone evaluation of the sector in the country
  • This evaluation is partly based on a number of country case studies. These case studies allow the evaluation team to gather information on the EC support (to the sector/theme/region) at the country level, which together with the desk phase findings should feed the global assessment reported in the synthesis report. This reporting is needed for transparency reasons, i.e. to clearly account for the basis of the evaluation, and also to be able to have a factual check with the concerned EC Delegations and other stakeholders.
  • This reporting should be seen as building blocks for the evaluation and as documents to be circulated in the reference group and the Delegations involved. In the end of the evaluation the country notes will be published as part of the overall evaluation exercise in annexes to the synthesis report (so editing is required). These notes should be prepared after the country missions according to the above structure. They should go further than the oral presentations conducted at the end of the missions.
  • Furthermore, the evaluation questions are to be answered on the global level on the basis of a synthesis of information collected and analysed through the desk study and the country case studies. Evaluation questions should hence not be answered at the country level in the country note.

Debriefing meeting

This is usually the third reference group meeting.

Agenda

  • Reminder of data collection and analysis plan
  • Reporting on data actually collected by the evaluation team using slides:

       - Problems encountered and solutions adopted.
       - Assessment of the coverage and reliability of collected data.

    • Discussion with a view to verifying that collected information will be sufficient to answer the questions.
    • Presentation of the most significant findings by the evaluation team.
    • Main verification and analysis work to be carried out at the next stage.

Author

FC
Former Capacity4dev Member
last update
7 December 2022

More actions