Gathering community feedback in times of remote evaluation

MAKING VOICES COUNT:

Gathering community feedback in times of remote evaluation

Johanna Pennarz, PhD, Lead Evaluation Officer, Independent Office of Evaluation of IFAD Prashanth Kotturi. Evaluation Officer, Independent Office of Evaluation of IFAD [First published on EvalForward.org]

In 2020, IOE undertook a project evaluation of the National Programme Community Empowerment in Indonesia. This was a Community Driven Development (CDD) project in Papua and West Papua provinces in the eastern-most part of Indonesia. These provinces are characterized by geographic remoteness, lack of infrastructure, scant presence of government structures, civil unrest in some parts and large indigenous populations.

Due to the Covid-19 outbreak, international travel was not possible. As a result, the evaluation had to adopt a methodology for remote data collection. Whilst the methodology provided some useful findings, it affected the scope and inclusiveness of the approach.

The approach: a vertical sampling and data collection

The evaluation adopted a bottom-up approach. It did this by examining the experiences and perceived benefits of community groups, first through interviews and then triangulating these with the perspectives and views of project staff such as village facilitators, district facilitators and regency facilitators. The interviewing and data collection strategy followed the vertical facilitation structure, composed by community groups, village, district, regency and provincial facilitators.

@Unsplash/Tyler Morgan

The villages were first selected using a simple random sampling procedure within each province using the village database available; then the evaluation selected a random sample of groups in each selected village. Once a community group was identified for interviewing, the respective village, district and regency facilitators supporting that group, directly and indirectly, were also identified and interviewed. The questions for each level of the facilitation structure were formulated only after interviews for the level below were finished. The vertical interviewing strategy enabled the evaluation to review the issues emerging from the interviews at a lower level in the facilitation structure and validate them during the interviews at the higher level of the facilitation structure and vice versa.

Issues encountered in the remote evaluation process

Inability to reach selected community groups due to lack of cell phone coverage. Linguistic diversity in Papua and West Papua and lack of fluency in lingua franca – Given the lack of reliable mobile coverage, the local consultants made an introductory call to set up an appointment with the groups so that they could be present in an area with better mobile coverage. Given the linguistic diversity of the indigenous populations, the evaluation team also sent the interview questions in Bahasa Indonesia before the appointment. This ensured that the evaluation team made the most of the limited time and network coverage. In cases where calls were not sufficiently audible due to network coverage, video recordings of answers on selected questions were provided through WhatsApp.

Saturation of development interventions and inability to distinguish between programmes – Papua and West Papua have large public programmes and numerous donor-funded projects. The evaluation team found that the target groups were unable to distinguish between various programmes. To address this, the evaluation team used the introductory call to introduce the evaluation and to clarify the project it would focus on.

Speaking to female community members – The evaluation tried to reach female community members by phone. In some cases, however, the evaluation found that men took over the phone while women were speaking to the national consultants, and insisted that the evaluation team should speak to them. The evaluation team had to be careful to avoid any serious consequences for the women given the high rate of domestic violence prevalent in Papua and West Papua. In those instances, the evaluation team would continue the interview with the man by asking a question or two before closing the call. The community member was then replaced randomly with another female community member to ensure integrity of the sample.

Final reflections on the evaluation methodology

CDD gives control of decisions and resources to communities. They are expected to make informed decisions about how they want to use local resources, who will benefit and how they will benefit. Therefore, they should participate in the evaluation from the outset. Participatory M&E would have been a key element of a mixed methods suite of evaluation tools. Ideally, people’s indicators should become the most important indicators of change.