Independent Magazine - Issue n.5, 2023

IMPACT EVALUATIONS: A DISCUSSION ON PAST, PRESENT AND FUTURE (WITH A SINCERE PLEA)

n 31 October 2022, Fabrizio Felloni, IOE Deputy Director, was among the speakers of the event titled ‘Impact Evaluations: Lessons learnt from IEU’s Learning-Oriented Real-Time Impact Assessment (LORTA) programme and other international organizations’, organized by the Independent Evaluation Unit (IEU) of the Green Climate Fund (GCF). Since 2018, the IEU’s LORTA programme has worked to strengthen the capacity of GCF-accredited entities and implementing partners to assess the impact of GCF-financed projects, through the use of impact evaluations (IE).

During the event, Mr Felloni shared the experiences matured by IOE in the conduct of impact evaluations. Between 2012 and 2019, the Office carried out IEs in Sri Lanka, India, Mozambique, Georgia, Kenya, Niger and Ethiopia. With the onset of the COVID-19 pandemic, these types of missions were brought to a halt to avoid the risk of exposure for people in rural area, for government representatives and IOE staff. For the moment, IOE is keeping these evaluations on hold, as the COVID-19 crisis is not over. Moreover, the Office currently faces strong requests from IFAD governing bodies and Management for evaluations at a more strategic level.

IEs were heavy exercises, as IOE was assessing a project against the full set of evaluation criteria, impact being only one of them. The Office would run large surveys – with anywhere between 1,500 up to almost 9,000 households –, in addition to assessing other criteria in a more qualitative manner, including through expert field reviews. IOE may come back to impact evaluations in the future, but more selectively.

After the IEU event, Independent Magazine caught-up with IOE’s Deputy Director for a quick chat on the past, present and possible future of IEs.

Good afternoon, Fabrizio.

Good afternoon, Alexander.

What purpose(s) would you say that an impact evaluation can or should serve?

I would argue that an impact evaluation needs to serve both accountability and learning, and would advocate towards de-emphasizing this dichotomy, which does not serve well the cause of evaluations. An IE should tell us what did and did not work, for whom and why. In order to do this, in addition to having statistical and data collection skills, we need to have the ability to understand the local context and the project history.

What projects are a better fit for impact evaluations?

Ideally, projects that have few or only one main intervention that is implemented in a homogenous manner. A typical example is in the health sector, as in the case of medical treatment or a vaccine: it is provided or inoculated always in the same dosage, always following the same practice (upper arm, for example) to everyone. Conversely, rural development projects à la IFAD are a challenging case. We have many components and sub-components, such as irrigation, training, rural finance, rural roads and rural enterprises, all of which are not provided ‘with the same dosage’ and in the same way everywhere and to everyone. Some people may receive only one component, while others may receive three and so on. There are ways, statistically, to deal with this but respondents may not indicate correctly which components they have received and we often do not have detailed records. So, are we evaluating a whole project or only a part of it, maybe one intervention package? There are advantages and disadvantages to both approaches but, either way, you can see that the level of analysis is not 100% clear from the outset.

Another problem we find in rural development projects is the spillover effect to non-targeted populations. This is especially true when interventions can be reproduced at low cost. And last but not least, we must keep in mind that our projects do not have reliable baseline data. We can work around this limitation, for instance, by controlling for sampling bias and using propensity score matching. These are reasonable palliatives, but they do not fully solve the problem of lack of baseline data.

What is the future of impact evaluation within IOE?

IOE has moved away from systematically carrying out one IE per year. Going forward, we will conduct IEs when we need to enhance the evidence for a future strategic evaluation, and when we see opportunities for methodological innovation.

What do you see as the future of impact evaluation within the broader evaluation community?

My advice to the evaluation community is simple. Impact evaluations are important. However, impact, alone, will not give you a complete indication of the worth and value of a development intervention. We need to avoid

the risk of concentrating on short-term effects. If an organization works with direct transfer of resources to households – such as productive assets, access to training, technical advice, subsidized productive inputs and infrastructure – it is not that difficult to observe ‘impact’, especially if by that you mean some short-term change. In reality, however, you may be observing an output rather than an impact. Think of a scheme based on ‘passing the gift’ of small livestock: what the survey will tell you is that people now own more small ruminants, which is simply what the projects have given to them. It may seem trivial but there are findings of this type, even in peer-reviewed journals. Our development goals are not short-term small gains. If changes are short-lived, we may not attain the SDGs. By the way, the “S” in the SDG acronym stands for the adjective Sustainable…. We need household- or community-level changes, married with stronger institutions at the community, local and central levels. In this context, impact is fundamental but impact assessment in isolation may not provide the full picture. Just by looking at the impact ratings from our project level evaluations and completion report validations, 81% of projects have positive scores over the past ten years. However, 60% of projects have a positive rating for both impact and sustainability. While most projects seem to have convincing evidence on impact, some have sustainability issues, which is something that IFAD Management recognizes and has committed to act upon.

Any final thoughts?

Perhaps some thoughts for the evaluation community: if you or your organization are conducting impact evaluations, please continue doing so, do not stop. However, do not lose sight of other criteria, such as efficiency and sustainability. We need comprehensive evaluation frameworks. We owe this to the hundreds of millions of people who live in poverty and whom we want to move out of poverty in the next years and decades – not just for a few months.

Thank you, Fabrizio.

You are welcome, Alexander