BREAKING THE MOULD: Discussing pandemic-shaped evaluations

Airplanes grounded. Cruise ships at bay. Passenger trains silenced. Post-apocalyptic science fiction? No, COVID-19. The year 2020 caught the world by storm, but there were those who found a way to weather it. The ability to react to the pandemic proved the prerogative of those able to adapt to the unforeseeable, to react to the unimaginable. While perfect coping strategies may have been difficult to come by, creative planning, imaginative approaches and blue-sky thinking paved the way for some exciting results. The pandemic called for a resourceful mind-set; and a resourceful mind-set was what IOE put on display.

As the first evaluations carried out by IOE during the pandemic are becoming publicly available, Independent Magazine caught-up with Fabrizio Felloni, IOE Deputy Director, for an insightful one-on-one chat.

Good morning, Fabrizio.

Good morning, Alexander.

What were the main challenges IOE was confronted with in the face of the pandemic?

Arguably, the main challenge was the inability to conduct international missions, as well as extensive field data collection, which had a number of implications for IOE’s work. For instance, we had to consider how to maintain methodological rigour and credibility vis-à-vis corporate governing bodies, how to command sufficient attention from IFAD management and government counterparts, how to maintain IOE office functionality and working relationships, and how to ensure the well-being of all staff members and consultants, to name but a few.

To address these challenging circumstances, what measures did IOE take to ensure robustness of the analysis, while maintaining the functional and behavioural independence of the evaluation team?

Evaluations adopted an innovative combination of approaches. These included remote interviews and mini-surveys, where feasible, by phone, Skype and Zoom, in addition to questionnaires sent via email to stakeholders. We also tested alternative methods for primary and secondary evidence collection and validation. For instance, in some cases we used geospatial data and analysis, especially for projects investing in physical infrastructure (for example on market infrastructure in Bangladesh). A colleague also tried Rapid Evidence Assessment for his work on climate change adaptation for smallholder farmers. This technique synthesizes secondary evidence related to an evaluation from published literature and websites. We also used external reviewers to crosscheck contextual information and enhance the quality of the analysis. Obviously, given that IOE staff and international consultants could not travel, more responsibilities were devolved to national consultants.

Did this increased reliance on national consultants pose challenges in terms of impartiality and behavioural independence? If so, how did IOE go about it?

Inevitably, there can always be risks. National consultants may receive pressure from national stakeholders, or might not be entirely impartial if they have spent a significant part of their professional careers working with the Government or the local IFAD Country Office. To minimize these risks, we spent additional time briefing them, preparing questionnaires and checklists, and agreeing on the scope of their work, so that, for example, they would focus on collecting factual data. In addition, we ensured that the whole team, including IOE staff and international consultants was fully engaged in discussing and triangulating the data and information collected.

What about your office, how was it affected by this rapid turnover of consultants?

First, I must say that, in addition to consultants, interns also proved critical for delivery. As for team building, team meetings and staff meetings were held during the lockdown and partial lockdown period. However, engaging the whole team remained a challenge in the absence of more informal and unplanned opportunities for interactions and gatherings.

These challenges notwithstanding, did the cancellation of international travels offer a silver lining in terms of cost savings?

In part this was true, but not for all evaluations. The reality is that cost savings were often offset by the need to increase the workload for international consultants – for example to conduct more thorough desk review or studies, and to collect alternatives to primary evidence, such as geospatial data and analysis, and secondary evidence – or by the increased number of national consultants.

In light of all this, would you say that the alternative approaches and methods used by IOE provided a valid substitute for traditional evaluation approaches?

Valid, yes. Perfect, no. To be honest, I would say that there is no perfect substitute for face-to-face meetings, household interviews and primary data collection through field visits. No available technology can replace in-person interactions with stakeholders, visits to project sites and interaction with the beneficiaries.

Are you suggesting that, ultimately, the results achieved by the evaluations were below expectations?

I would not say that, no. What I would say, instead, is that limitations of the evaluations during COVID-19 have to be clearly stated up front, where possible indicating the level of confidence of findings, as well as those where more field visits would have helped. It is important to highlight the substitute methods undertaken to warrant the level of confidence claimed. The level of confidence brought about by substitute methods can be adequate when evaluating completed operations, especially for those with good, monitored evidence. This may not be the case for new and ongoing projects.

That being said, do you think that the experience in dealing with COVID-19 offers opportunities to revisit some of the evaluation practices and processes, which could be considered not only for future crises but also for ‘normal’ times, to add evidence and simplify processes?

Absolutely, yes. There are a number of very valuable lessons learned and best practices that we could look at mainstreaming. For a start, rationalizing travels and reducing the ‘carbon footprint’ of evaluations is important, regardless of crises or pandemics. The COVID-19 experiences has in part proven that this is possible. We can enhance efficiency of country missions through a combination of remote and face-toface, in-country interviews and greater emphasis on project site visits. We could spend less time driving in the traffic of capital cities while moving from an interview place to the next one. Some of the interviews in the capital could in fact be replaced by remote interviews. We can develop mechanisms and practices to ensure more in-depth document review and preparation for missions and interviews. We can make greater use of information technology to augment evidence and analyse secondary evidence.

Any final thoughts?

I would simply add that in case of (partial) lockdown, it is important to ensure that teamwork is not interrupted, while recognizing the social-psychological burden this entails. In particular, we must always pay special attention to colleagues who are away from their family and network of friends. This may even call for advocating for institutional support for families to balance professional and family needs. At the same time, it is also important to avoid overwhelming staff and teams with formal gatherings and very lengthy video interactions.

Thank you very much.

You are welcome.