|
.
1ST STAGE: PREPARATORY STEP
1/ Documentary research
The preparation of a cultural and social analysis mission requires a preliminary documentary research which should be broad and thorough. It helps the evaluators focus on the real goals of the evaluation, ask relevant questions and check their tentative hypotheses on-site with the organisation of expert panel sessions and interviews. In other words, information should be found prior to the mission through a documentary research.
The objectives of the documentary research are:
- To obtain an accurate knowledge of the society
- To focus on the real goals of the evaluation within the boundaries of the three themes constituting the reference framework and ask relevant questions (the hypotheses will be tested on-site afterwards)
- To be able to moderate substantive discussions
- To give the evaluator credibility with the experts through to a thorough knowledge of the topics
As the questions of cultural and social analysis are very large, the documentary research must be carried out efficiently. Below is a methodology for undertaking this documentary analysis.
Bibliographies per country or theme
A bibliographical index or publication report edited on a regular basis and dealing with all the publications dedicated to a country or a region, would be an information asset. Indeed, the reading of bibliographical indexes accelerates the identification of relevant documents for the study of a country strategy and focuses the work on its core issues. However, the evaluator must be careful with bibliographies such as books which are not systematically updated because of the rapid pace of contributions and the delays required to publish an index.
Books and directories such as a handbook with country and/or theme entries
In most cases, the evaluator will not find any bibliographical index. The documentary research should start with articles taken from a directory or a handbook which summarise the history of a country and update its political, economic and social current events regularly. These types of reference books can be found in French, English and in other languages. Although the interest of these articles may seem limited, they enable the evaluator to get familiar with the history of the country quickly and obtain a first set of reference bibliographies.
| Example |
| The State of the World (Etat du monde), published each year by the Editions La Découverte, provides the reader with articles dedicated to each country and written by well-known specialists, and bibliographical references which are usually relevant and updated. There are also useful links to websites. |
.
Country-focused books
Such books usually deal with ancient history, languages, social structures, religions, etc., without a specific focus and their quality can be uneven. However, they can be invaluable for their short yet thorough presentation of the cultural and social realities and be used to focus the documentary research on specialised bibliographies which investigate further each of the themes selected.
Reference or controversial books
These books or articles, specialised in a theme or subject, can be consulted for the sociology, anthropology, history, human geography, political sciences or economic fields. Reading one or more of these books dealing with each of the three themes of the cultural and social analysis is recommended.
The evaluator should be acquainted with (and if possible have read) the authors who have been working on the country under study and their main contributions. This will give the evaluator a solid foundation and arguments to focus the expert panel debates on topics which are familiar and debated in the country.
| Example (taken from the tool's testing mission in Tanzania) |
| Goran Hyden's well-known book (Beyond Ujamaa in Tanzania. Underdevelopment and an uncaptured peasantry) has fostered intense scholar debates among anthropologists of development at its publication and still does 25 years afterwards in Tanzania, where the typically Hydenian notion of "economy of affection" is supported by some and contested by others. This notion was an interesting opening for the first expert panel session to debate over the peasantry attitude towards liberalisation reforms. |
.
Articles on large themes and subjects taken from specialised reviews
Reading such articles can be time effective. Many specialised reviews can now be easily found on the Internet.
Local publications, working papers and other documents produced by research institutions or observatories in the country under study
During the mission, local publications which are not always available in Europe should be consulted. The evaluator should:
- Systematically obtain local publications which deal with the themes of the cultural and social analysis
- Contact research institutes or observatories (such as the Economic and Social Observatory) where indexes, working papers and studies based on local surveys can be found
- Ask the respondents of the interviews (such as researchers and professors) to bring the working papers and other studies that they have written but which are not yet edited Local press
Local press
The evaluator should regularly consult the local press prior to the mission, in order to get familiar with the topics debated. During the mission, a short press review is recommended to:
- "take the feeling" of the country
- identify the major topics and issues of public debate
- be able to foster the debate during the expert panel sessions and the interviews through the presentation of an article
The reading of the local press is not compulsory but can usefully complement other documentation research.
Official documentation of co-operation agencies
The documentary research should not be limited to the bibliographical research of books and published working papers:
- Official documentation of co-operation agencies must be consulted.
- All the reports and studies relating to the country under evaluation should also be used.
2/ Adaptation of the question grid to the specificities of the country
The background between one country and another varies too much to be able to present a list of pre-determined questions without risking the omission of specific aspects of the context which could be crucial. The suggested series of topics and issues should be adapted to each country following the documentary analysis.
3/ Carrying-out preliminary interviews
These interviews have two objectives: the provision by respondents of interesting contacts and bibliographical references, and complementary information to the documentary research which supports the construction of a precise question grid.
Three categories of respondents seem important to favour:
- specialists of the country under consideration
- specialists of the sector under consideration
- managers of international institutions, such as those working for a desk in Brussels
4/ Determination of the list of experts involved in the panels
Selection of the experts and appointment-making
The preparation of a list of experts can be facilitated by the identification and use of 2 or 3 key informants. The expert list should exceed the number of the experts who will take part in the panel and be constituted of people specialised in the fields under consideration.
The role of key informants is crucial because it supports the triangulation of the sources and eases the participation of target people in the expert panels.
If 3 expert panel sessions of 6 participants each are planned, the list of experts should include at least 18 people. As cancellations may happen, it is advisable to identify more experts than needed.
Appointment-making strategy
The experts should be first contacted by e-mail. The message should include a short description of the mission and the sessions, and the dates. Only the experts responding to this mail and expressing an interest and availability should get more details. The experts who would not have replied to the mail should be contacted by telephone if their number is available.
- specialists of the country under consideration
- specialists of the sector under consideration
- managers of international institutions, such as those working for a desk in Brussels
2ND STAGE: DURING THE MISSION
The expert panels and the interviews organised for the cultural and social analysis should not be used to collect the basic information about the socio-anthropological, institutional or historical characteristics of the country. They are implemented to validate or invalidate primarily hypotheses, to foster the debate on unresolved issues, and to highlight issues which have not been considered before.

The expert panel sessions
Meeting with the members of the panels
Each expert should be interviewed before his/her participation to the panel sessions. During this interview, the evaluator should introduce the participant to the mission and to the issues covered by the analysis. The evaluator can also take this occasion to discuss openly with the expert's experience and approach of his/her field of specialisation, and to collect his/her general opinion about external assistance in his/her country. Questions selected from the grid can be developed during this interview, so as to compare the views of the expert during the panel sessions and the interview.
Organisation of the expert panel sessions
A session is organised for each of the three themes of the cultural and social analysis:
- history, beliefs and legacies of the past
- the social, legal and political organisation of society
- empowerment through technology: cultural issues and social innovations
The session should last about 4 hours, with a break of 15 minutes.
The objectives are:
- to debate the analyses derived from the preliminary readings dealing with the main socio-anthropological, historical, political, economic and cultural characteristics of the society under study, and to analyse recent changes;
- to underline the major topics and issues of interest for the European Commission's country strategy implementation (for example, elements highlighting socio-political reorganisations due to proposed reforms)
Content of the sessions
One of the expert panel's difficulties is to differentiate within the discourse of the experts the personal opinions, interpretations and analyses about a topic they have studied. Indeed, as the expert panel generates general debates which can exceed the specialisation of each expert, the participants can be encouraged to express themselves on topics they are not specialised in.
Several elements can reduce the quality of the participant's expression:
- A lack of familiarity with free expression due to a social, political or religious control, or to a recent past marked by the alienation of fundamental freedoms
- The participation of experts who co-operate with international agencies very often and are used to the standardised speeches of the funding agencies
Moderation of the session
The moderator should be fully aware of the context, in order:
- to be credible towards the participants and be able to moderate the discussion in an informed way
- to avoid descriptive testimonies and focus the debate on the shared knowledge of the experts
- to ask questions which will stimulate the interest and participation of the experts
The interviews
Interviews should not only be considered as a complement to the expert panel methodology. Interviews are as effective as expert panels in the collection of information.
The objective of the interviews is to explore particular points debated during the panel session, or to meet other experts who could not participate in the session because of their unavailability or because the sessions were full. Three types of interviews can be differentiated:
- Interviews scheduled from the home base and organised to complete the expert panel's contribution
- Interviews organised in the country, consequent upon issues arising during the panel's session
- Interviews organised in the country, following suggestion from experts already interviewed or key informants
The work programme should therefore plan enough time for these interviews, which should normally be organised before and after the expert panel sessions.
The focus group
Through the organisation of a focus group, the evaluator could meet the stakeholders and/or the final beneficiaries of the strategies implemented, and validate the data collected during the expert panel sessions and the interviews.
The organisation of a focus group with the stakeholders or with the target population of a development policy differs from the organisation of an expert panel. Whereas experts are contacted by e-mail or telephone (which can be done from the base country), the participants of a focus group can only be contacted in-country, which requires a time-consuming investigation. As the mission's work programme is usually tight, a local informant should be put in charge of the organisation of the focus group. An interpreter should also assist the evaluation team, because local participants will not usually speak the language of the evaluators, or English. Moreover, the richness and subtlety of the participants' expression in their own language can be the object of a specific analysis, all reasons calling for a good interpreter.
3RD STAGE: AFTER THE MISSION
Data analysis
The data derived from the documentary analysis, the panel sessions, the interviews and the focus group are gathered and analysed with a view to contributing to the evaluation questions identified during the preparatory stage. As in other stages, the evaluators should make the distinction between personal opinions, interpretations and analyses, which does not mean that interpretations and opinions should be put aside (but that they should be taken for what they are).
Drafting of the report
The report will normally be divided in three parts, each corresponding to one of the 3 themes constituting the reference framework. In each of these parts, the data analysed will be distinguished from recommendations.
What are the preconditions for cultural and social analysis?
Human resources
The evaluation team should include a sociologist or a senior anthropologist.
The time span
The person responsible for cultural and social analysis in the evaluation team should allow approximately 40 working days for the task.
Details of these 40 days are provided in the summary table. It forecasts the number of working days in an average situation. In cases where difficulties are likely to occur which will slow down the evaluation progress, 40 to 50 days should be scheduled.
Financial resources
Each expert participating in an expert panel is remunerated for a day's work (corresponding to the panel session and the review of the note prepared by the evaluator). This remuneration should match national rates when the expert is a citizen of the country under study, and international rates when he/she is neither a citizen nor a resident of the country.
The remuneration of an expert participating in several panels will be proportional to the number of sessions.
SUMMARY TABLE OF THE PROCEDURES TO FOLLOW IN COUNTRY/EVALUATIONS

|