Independent Magazine - Issue n.5, 2023

“If

you can’t improve it.”

Indran A. Naidoo, IOE Director

Thank you for your attention to the 5th edition of Independent Magazine. The magazine seeks to stimulate debate on the critical role that independent evaluation can play in improving society given its emphasis on transparency and accountability, democratic ideals which are highly valued by citizens and partners.

IOE tests performance claims. Setting a culture where objectivity is privileged over interests is challenging. Independent evaluation, should present facts for decision-making in a dispassionate manner, and enjoys credibility by not being interest-based.

IOE is part of the Evaluation Cooperation Group (ECG), the professional network of the independent offices of the international financial institutions. Members of the ECG work according to policies which guarantee structural, behavioural and managerial independence, ensuring the credibility of their outputs. The highest standards and practices for evaluation are codified in norms and standards of international networks and have been the basis for conversations over the past decade at events such as the UNDP National Evaluation Capacity (NEC) conference, which has created a formidable community of practice across over 180 countries. IOE supported the 7th edition of the NEC conference in Turin, Italy, last year.

We understand capacity building as something that goes beyond training, as IOE Deputy Director Fabrizio Felloni makes clear in this edition of the magazine. In his interview, Fabrizio highlights how we can, taking up a facilitating role, use our evaluations to identify countries and national agencies that have a particular need to develop M&E systems. In consultation with IFAD Management, we help them interact with specialized networks, such as the Global Evaluation Initiative.

IOE is also building evaluation capacity by sharing its in-house expertise globally through practice-based methodological lessons, resulting in publications and presentations at world-renowned academic institutions. This endorsement attests to an experienced office that is able to draw upon a formidable body of core outputs, over 70 evaluation reports have been tabled at the IFAD Board and governments around the world. Our emphasis on building coalitions can be seen in the diverse content of over 30 international evaluation events.

In my own words

While it is true that independent evaluation can create institutional and other tensions, these should be viewed as creative tensions that challenge organizations to become more self-critical and reflective and act upon performance feedback. Calling for more evaluation affirms this culture and demonstrates organizational maturity. In this regard, in his statements at the learning event organized by the Independent Office of Evaluation of the New Development Bank, IFAD President Alvaro Lario affirmed that evaluation brings positive messaging into the Fund, builds evaluative dialogues to improve organizational culture, and is important for organizational learning, course correction, transparency and accountability. He also stressed that evaluations have their full potential when they are viewed as necessary insights. Recent statements by members of the IFAD Executive Board during the replenishment discussions, also emphasized the need for evaluation to be used and respected.

This sets the right tone to build a more reflective and critical organizational culture. Feeding off this level of interest, and building on a revised evaluation policy and multi-year evaluation strategy, IOE’s range of products is now more diverse, more engaging and collaborative, as evaluation is everyone’s business.

This edition explores capacity as a complex element, which has moving and interacting parts. Its full potential emerges when multiple elements work in synergy, towards the same outcome. In this context, the IFAD Evaluation Manual serves as a central resource, and is now available through a suite of on-line training modules.

Seeking to always remain at the cutting edge of our field of work, we have moved into exploring, researching and discussing the psychological dimension of our discipline, by drawing expertise from neuroscience, a growing field and applying it to the field of evaluation. In the coming year we will advance the area through new event, publications and training modules.

In essence, this is a very exciting time for all of us, as we embrace the virtues of evaluation to become ever more reflective, and as we build a stronger IFAD with its partners and Member States.

I trust you will find this edition inspiring.

Dr Indran A. Naidoo Director

Independent Office of Evaluation of IFAD

Editorial Board

Proofreading

Revisioning

Profile

Independent Magazine brings to the forefront of the global development dialogue the major efforts undertaken by the Independent Office of Evaluation of IFAD, while seeking to advance the organization’s vision of vibrant, inclusive and sustainable rural economies, where people live free from poverty and hunger. To present the richness of rural life, and detail facets of local community lifestyle, Independent Magazine also zooms in on cultural activities and landmark occurrences in countries featured by IOE’s evaluations.

Dr Alexander Voccia

Laure Vidaud

Profile

Sarah Pasetto

Profile

Gresia Bernandini Marino

THE UN HELPS GOVERNMENTS BUILD

MAJOR CHANGES TO ECONOMIES AND “

Major changes in our economies and societies need to be effectuated in record time. In order to succeed, these changes will need to be systemic, comprehensive and include voices, values and world views from a variety of cultural and ethnic backgrounds. The focus must be on climate justice, just transition, and the full range of the Sustainable Development Goals”. Indran A. Naidoo, IOE Director, delivered this unequivocal call to action on the global stage of the 7th National Evaluation Capacity (NEC) conference, in Turin, Italy.

Organized by the UNDP Independent Evaluation Office (IEO) and the Global Evaluation Initiative (GEI), from 25-28 October 2022, NEC 7 reflected upon ‘Resilient National Evaluation Systems for Sustainable Development’ following the upheaval brought by the pandemic.

“Evaluation is needed to improve the performance of public policy, to improve people’s lives. The ultimate goal of our presence here is to obtain better development results, and to achieve sustainable development results that we want not only for us, but for our children, and the children of our children. If we don’t act now, we won’t have that future”, said Oscar Garcia, IEO Director.

The event built on the success of past conferences, and brought together over 300 influential evaluation stakeholders and actors from over 100 countries, spanning oversight and accountability sectors across Government, civil society, academia, the United Nations, as well as bilateral and multilateral partners. The event gave renewed emphasis on sharing progress and lessons learned in strengthening national evaluation systems.

NATIONAL EVALUATION CAPACITY

SOCIETIES NEEDED IN RECORD TIME

“NEC 2022 comes at a time when resilient government systems are all the more important to mitigate crisis, like the one the world is still emerging from. The conference draws on the formidable capacity of the GEI global network, offering more support to governments in their important work of delivering better results to citizens”, noted Dr Naidoo in his remarks during the opening session of the event.

IOE supported the event with other donors, and played an active role in the conference by participating in various sessions. Dr Naidoo was joined by Fabrizio Felloni, IOE Deputy Director, and Dr Mónica Lomeña-Gelis, IOE Senior Evaluation Officer, who each moderated a session stream discussion. In addition, the IOE Director also acted as a panellist during the plenary session on day two.

Moving from the premise that evaluation must consider development impacts across all programmes, Dr Naidoo joined Andrea Cook, Director of Evaluation at WFP; Olivier Cossée, senior evaluation officer at the FAO Office of Evaluation; Dr Anastasia Aladysheva, Impact Evaluation Specialist, Green Climate Fund; and Bala Yusuf-Yunusa, Office of the Senior Special Assistant to the President of Nigeria on SDGs, to discuss the transformative role of the evaluation function, on the second day of the conference.

“From a rural-based perspective, the transformative change that we are looking for is the shift in governments’ and international agencies’ paradigm of agricultural interventions, moving from an anthropocentric approach to one which is more environmentally focused”, explained Dr Naidoo in his intervention.

Moderated by Alan Fox, UNDP IEO Deputy Director, the plenary panel discussion recognized that transformational change is required to avoid further catastrophes caused by the three environmental crises that human actions have caused, namely the climate crisis, nature crisis, and pollution and waste crisis. Participants agreed that evaluation can contribute to finding durable solutions based on sound science and experiences from the field, but that in order to do so it must broaden its vision.

“Development pathways have had a hand in shaping the Anthropocene. Evolution must consider the development impact across all programmes, and not only those addressing environment, climate and food security. Is evaluation facing up to this reality? Whether we wish it or not, a new normal is coming. COVID-19 is just the tip of the sphere”, said Mr Fox.

Whether evaluation is innovative and nimble enough for a rapidly changing world, and how evaluation can foster a systems-thinking approach to bring about transformative change, were the questions at the centre of debate during the parallel session in which the IOE Director acted as moderator and presenter.

“There has been a tendency to view development through a silo approach, not considering that there are multiple pillars and multiple elements that interact. Today, we need to understand all elements of development. For this to happen, we need to look at evaluation, not just

@UNDP_Evaluation

in terms of describing what happens, but also in terms of how it can be transformative”, Dr Naidoo underscored.

Titled ‘Evaluating beyond the 2030 Agenda’, the session drew from the experiences of Dugan Fraser, Coordinator GEI, and Dr. Mita Marra, Associate Professor and Editor-in-Chief, Evaluation and Program Planning, University of Naples and the George Washington University. Mr. Fraser presented the experiences of gender-based and feminist evaluation, while Professor Marra highlighted the importance and value added of adopting multiple evaluation methods.

“When we consider the state of evaluations in our public sector evaluations, what is so striking is that monitoring is so much stronger than evaluation. I think there is a reason for that. This relates to a culture of accountability and control. The institutional arrangements around evaluation are such that evaluation gets side lined and marginalized. It’s not mainstreamed into the way that organizations do business”, said Mr Fraser.

On the afternoon of the second day of the conference, Fabrizio Felloni, IOE Deputy Director, acted as moderator during the parallel ses-

from these approaches. The session drew on the experiences of Orifjan Namozov, Deputy Director for Strategic Planning, Programming and Analysis, Ministry of Agriculture of Uzbekistan; Sanjeev Sridharan, Country Lead, Learning Systems and Systems Evaluation at the India Country Office of the Bill & Melinda Gates Foundation; Dr. Anastasia Aladysheva, Impact Evaluation Specialist, Green Climate Fund; and Alok Mishra, Director General, Ministry of Commerce & Industry, Government of India.

The panel included cases where a governmental M&E system at the sectoral level (agricultural sector in Uzbekistan) still needs to be established and other cases where this has been set up, such as in India (trade) and in Pakistan (health). There where important discussion as to the levels of the analysis of the system (project, sub-sector, country) and what type of data is needed and how to aggregate the analysis at different levels in order to gen-

sion titled ‘What can we learn from Sectoral Evaluation systems?’. Recognizing that certain sectors – notably health, education and agriculture – have advanced further in building evaluation systems than cross-sector systems, the session discussed what could be learned

erate findings that are relevant to the national policies. The case of Pakistan was of particular interest, showing examples of disaggregation at the provincial and divisional levels, with the use of GIS-based mapping as a tool to provide feedback to policy makers.

On the third day of the conference, Dr Mónica Lomeña-Gelis, IOE Senior Evaluation Officer co-moderated the parallel session titled ‘Innovative M&E systems’, alongside Renata Mirulla, EvalForward Facilitator. The session discussed opportunities for strengthening the national M&E system and its use for decision-making in the agriculture and rural development sectors. The session was enriched by contributions from Rodrigue Siangoye Owoumbou, Planning, monitoring and evaluation officer, Ministry of Agriculture, Gabon; Mor Seck, Permanent Secretary, Commission for the Evaluation and Monitoring of Public Policies and Programs, Presidency of the Republic of Senegal; Sekou Tidiani Konaté, Director of Coordination, Cooperation and Planning of the National Institute of Statistics, Djibouti; and N’Dia Youssouf, Director of control, M&E at the Ministry of Planning and Development of Côte d’Ivoire.

All panellists coincided with the challenges faced to implement evaluations at the national level due to budget and human resources constraints. The case of the upcoming government-led evaluation of the national agricultural policy in Senegal can be an interesting case where international organizations, such as the UN Rome-based agencies, can contribute as external peer reviewers and strengthen national evaluation capacities.

The UNDP IEO organized the first NEC conference over a decade ago, in 2009 in Casablanca, Morocco, with 55 participants. The conference was a success as it was able to draw unprecedented country and government support. During his eight-year tenure as Director of IEO, Dr. Naidoo, developed and expanded the NEC series to become the largest constellation of thought leadership and training session globally by government participation, reaching 180 countries. The sixth and latest conference in the series was held in Hurghada, Egypt, in 2019, four years after the adoption of the SDGs, with more than 500 participants [here].

The NEC conferences bring together influential players from oversight and accountability sectors to connect, engage and share about issues that fall within the ethos of good governance, evidence-based decision making and better performance management [here].

Documents Photos Videos

Previous conferences [2009 - 2020]

FIRST ON-LINE TRAINING NOW AVAILABLE

For the first time in its over forty-year history, IOE has launched a fully interactive on-line training course, available also to users outside the Fund. Through an audio-visual immersive experience, users will learn about the principal contents of Part 1 of the 2022 IFAD Evaluation Manual, prepared by IOE in collaboration with IFAD Management and launched earlier this year.

Complete with a successful course completion certificate, the training sensitizes users on the principal contents of the Evaluation Manual, while making their reading more engaging. Through roleplay exercises, case studies, quizzes and tests, users will gain a solid understanding of the institutional set-up of evaluation at IFAD; the key principles of conducting an evaluation; the implications of the key evaluation principles in connection with Agenda 2030; IFAD’s criteria and rating system; and an overview of methods available.

After completing the course, participants will be able to describe the key features of the institutional evaluation set-up at IFAD; recognize the roles and responsibilities of Management and IOE; apply the key evaluation steps indi-

cated in the Manual to a concrete evaluation exercise; identify key contributions from international debates and locate key sources; and use the evaluation criteria and apply them to a given context in order to elaborate specific sub-questions and indicators.

The on-line training accompanies the Evaluation Manual by presenting its contents in a more dynamic fashion. The primary purpose of the Evaluation Manual is to ensure the quality, consistency, rigour and transparency of the evaluation function at IFAD, in order to increase the effectiveness of IFAD’s efforts. Along these lines, it provides guidance on adapting international standards, practices and evaluation criteria to the context of rural development, particularly when the end-clients of development interventions are smallholder farmers and small rural producers. The Manual also offers methodological guidance and standards for evaluations across the Fund.

Environment and natural resource management ranked top IFAD-supported interventions

Environment and natural resource management (ENRM) is the criterion that exhibits the highest percentage of positive ratings among IFAD-supported activities between 2011 and 2020, closely followed by innovation and relevance. The Independent Office of Evaluation of IFAD (IOE) presented this finding in the first edition of its new flagship publication, titled ‘2022 Annual Report on the Independent Evaluation of IFAD’, published on 22 November 2022.

Since 2003, IOE has produced an Annual Report on Results and Impact of IFAD Operations. On the occasion of its twentieth anniversary, the report has been re-pitched and renamed as the Annual Report on the Independent Evaluation of IFAD (ARIE), reflecting upgraded contents and a broader scope. The ARIE is in line with the 2021 Evaluation Policy, which emphasizes the creation of an overall evaluation culture and reinforces the importance of learning and collaboration. The focus of the ARIE is on substantive findings and on adding value to the existing evaluations.

Within the 2022 ARIE, ENRM and climate change adaptation are the only two criteria that exhibited a constant increase over the past decade.

The positive performance of ENRM and climate change may be linked, at least in part, to IFAD institutional efforts, including the preparation of relevant strategies, guidance tools, mobilization of climate financing and establishment of a dedicated unit to mainstream climate responses.

On the opposite end of the spectrum, the ARIE notes that performance continued to be lowest against efficiency, government performance and sustainability. With regard to government performance, while there is no single cause for the decline of government performance in the projects

that reached completion point between 2013 and 2018, a recent evaluation synthesis observed that IFAD-funded projects were increasingly implemented by ministries of agriculture and project design had become more complex, with value chain development objectives being added to primary production. Overall, the ministries did not have sufficient capacity, resources and expertise to manage these projects. IFAD country presence was important to support country programme performance. However, on its own, it was not sufficient to improve the performance of operations, which depended on other factors as well. IFAD country presence was strained when operations were reaching into remote locations and in contexts with weak local government capacity.

Among the report’s other salient findings, recent evaluations noted that community-driven development operations performed even better than others in fragile, remote and marginal contexts. In addition, the joint evaluation conducted by IOE and the evaluation offices of FAO and WFP on the collaboration between Rome-based agencies (RBAs) determined that such combined efforts have enhanced the sharing of knowledge, lessons and good practices in the areas of gender, nutrition and emergency response. However, in other development areas, RBAs have made limited progress in reducing overlap, competition and duplication.

Looking ahead, the ARIE highlights areas for further improvement of IFAD-supported activities. With regard

to climate change, the report notes that there are still gaps related to climate financing. In particular, there is a need for a shared vision between IFAD management and staff to integrate climate change adaptation in IFAD interventions; improvements in the design of climate change adaptation interventions and non-lending activities; and a results framework to track and allow analysis of the performance of interventions.

The 2022 ARIE draws on evaluations of 284 projects completed between 2011 and 2020, as well as 49 country strategy and programme evaluations conducted between 2011 and 2021. The report compares findings across evaluations, analyses rating trends according to the established evaluation criteria, and formulates a number of key messages.

INDEPENDENT EVALUATION SPARKS TRANSFORMATIONAL CHANGE THROUGH ORGANIZATIONAL LEARNING

A conversation with Alvaro, Marcos and Indran

T“he work of the Independent Office of Evaluation of IFAD (IOE) is key for us. From a management perspective, one of the critical elements that independent evaluation provides is learning. For example, a 2018 corporate-level evaluation led by IOE spurred a big revamp of IFAD’s financial architecture, which led to the Fund’s current AA+ rating as well as a new way of allocating resources for borrowed funds”, affirmed IFAD President, Alvaro Lario, during the second lecture of the series hosted by the Independent Evaluation Office (IEO) of the New Development Bank (NDB), on 12 January 2023.

The participation of the IFAD President and of the NDB President, Marcos Prado Troyjo, illustrated the deep commitment of both IFAD and NDB to cooperation across multilateral organizations. The Director of IOE, Indran A. Naidoo, also delivered remarks during the seminar, which was organized and moderated by the IEO Director General, Ashwani

K. Muthoo.

Mem -

bers of NDB and IFAD senior management and staff attended the meeting, alongside members of the NDB Board of Directors.

In his keynote address, Mr Lario shared insights and experiences on the role of independent evaluation in promoting organizational transformation and better results on the ground. Furthermore, the IFAD President also highlighted good practices and areas for consideration in the interactions between management and independent evaluation functions. The NDB President, Mr Troyjo, further emphasized the importance and added value of evaluative work for multilateral organizational development.

“The multilateral development world has had a new member, the NDB, for the past seven and a half years. We have grown a lot since our establishment. As we continue to grow and mature as an organization, it is key to have quality evaluation to help us improve day after day. In this regard, I congratulate IFAD’s IOE for conducting transformational independent evaluation that has contributed to institutional changes and reforms over the years”, stated Mr Troyjo.

Marcos Prado Troyjo, President, NDB

“I congratulate IFAD’s IOE for conducting transformational independent evaluation that has contributed to institutional changes and reforms”.

Marcos Prado Troyjo, President, NDB

Drawing from the successful experience of IOE in generating lessons that have sparked transformational change through organizational learning, Dr Naidoo presented the pillars of IFAD’s Evaluation Policy. These include usefulness, impartiality and credibility, transparency, evaluability, value for money and cost effectiveness, and partnership, consultation and collaboration. In the same context, the IOE Director listed the key facets of the Office’s first multi-year evaluation strategy.

“The spirit of the times we are now in stresses how crucial the work of IFAD is. That is why it is so important to have this opportunity to exchange. If we were to juxtapose our two institutions and look at the strategic objectives, we would see many overlaps. It is the case for environmental sustainability, climate resilience and market participation. IFAD’s emphasis on support to poor rural people is a shared development objective by many of our projects. I hope that his dialogue serves as a starting point to strengthen collaboration between our organizations”, noted Mr Troyjo.

“Based on our experience, elements that I would recommend looking at are how evaluation topics are selected, ensure that evaluation products are balanced, strengthen engagement with management, and build evaluation capacity to the extent possible in order to raise the bar”, explained Dr Naidoo.

Through the evaluative lenses, the seminar offered an opportunity for cross-fertilization of views, and provided a stimulus for both NDB and IFAD to explore opportunities for knowledge exchange, co-financing and other forms of partnerships. Event participants welcomed these discussions, especially given that IFAD portfolios in most NDB member countries –such as Bangladesh, Brazil, China and India –are amongst the largest of the Fund.

The IEO Lecture Series was introduced in mid2022, with the broad aim of fostering an evaluation, results and learning culture across the Bank and amongst its main stakeholders. It also serves as a key platform to share thoughts and ideas on topics of high rele vance for accelerating the SDGs.

Alvaro Lario, President, IFAD

Presentation by Indran A. Naidoo

Full event video recording

@pixabay/Kuloser

“

The work of IOE is key for us. From a management perspective, one of the critical elements that independent evaluation provides is learning”.

Alvario Lario, President, IFAD

The daily life of women in Guinea*

The daily life of women in Guinea was the focus of a recent exhibition at the Portuguese Cultural Center in Bissau.

Substantivo Feminino featured 30 paintings by Guinean plastic artist Nuno Tambá, commonly known as Young Nuno Drake Mars. The 30-year-old Guinean explained that the exhibition aims at “countering the wrong way” in which women are portrayed in the country and are labelled “as lazy or dependent on men”.

Nuno said he has witnessed the courage of the Guinean women based on the example of his own mother. “I am the son of a single mother, that is, a widow. My father died very young. I was raised by my mother and stepmother. Our house is almost glued to the market of Plubá, Feira da Cabaceira.”

In the paintings, you can see portraits of women’s daily lives, from a policewoman to a woman carrying food on her head, or with a child on her lap or on her back, and even girls carrying out domestic chores. Accompanying the exhibition at the Centro Cultural Português was musician Eric Daró, who performed songs related to love and women.

Sociologist Cadija Mané, said that Nuno had “managed to capture the difficulties of the Guinean woman”. The director of the Centro Cultural de Bissau, António Nunes, said “it is important” that the young Guinean artist chose women as the theme of his first exhibition, in which “he sought to reach, in a transversal way, the whole of society. The issue of women is something that came from him. He decided to do it this way. It is a very strong tribute to his mother, to his mother’s effort to educate him and his family”.

@Young Nuno Drake Mars

Rice-based production systems improved in Guinea-Bissau

IFAD-supported interventions in the southern regions of Guinea-Bissau have led to the improvement of rice-based production systems, through the rehabilitation and development of lowlands’ rice fields. This is one of the key findings of the very first Country Strategy and Programme Evaluation (CSPE) of Guinea-Bissau, carried out by the Independent Office of Evaluation of IFAD (IOE). Stakeholders of IFAD-funded operations in the country discussed this and other findings, conclusions and recommendations of the soon-to-be published report during the course of an online workshop, on 12 December 2022.

“The partnership with IFAD contributed to improved exploitation of agricultural resources, smallholder food self-sufficiency, and enabling market surpluses. This was achieved through the rehabilitation of lowlands for the intensification of rice cropping systems, based on endogenous knowledge and the introduction of improved farming practices and techniques”, stated Dr Indran A. Naidoo, IOE Director.

Co-organized by the Government of Guinea-Bissau and IOE, in collaboration with IFAD’s West and

Central Africa Division, the workshop brought together high-level attendants, including Kaoussou Diombera, Advisor to the Minister of State in Charge of Agriculture, and Donal Brown, Associate Vice-President, Programme Management Department, IFAD. Over 60 participants joined the event. Government institutions in attendance included the Ministry of Agriculture and Rural Development, the Ministry of Environment and Biodiversity, and the Ministry of Economy and Finances, in addition to the regional government of Buba and technical implementing partner institutions. The workshop also benefitted from inputs of bilateral partners, such as WFP and the European Union Delegation in Bissau, and the participation of non-governmental organizations, civil so -

ciety organizations, and IFAD management and staff. In addition to improving ricebased production systems, the projects financed by IFAD also promoted market gardening and innovative social mechanisms. The latter included the establishment of integrated development committees and their financing mechanism, consisting of local development initiative funds. Agricultural research was supported through participatory trials of new rice varieties aimed at facilitating the establishment of a seed sector, and a catalogue of rice varieties was developed, which included local and selected varieties.

“The evaluation found that there is increased rice production in the areas supported, a reduction of difficulties in periods of hunger, possible improvements in household incomes, strengthening of social capital, and emergence of local rural institutions”, explained Dr Kouessi Maximin Kodjo, IOE Lead Evaluation Officer.

The sustainability of these achievements, however, is called into question. While producer organizations delivered tangible results in terms of the management of collective resources, these groups are still in their infancy, have

Dr. Kouessi Maximin Kodjo Lead author, Guinea-Bissau CSPE

Profile

not been legally formalized, and have a long way to go to act as critical institutions for decentralized development.

“In terms of rice production systems, support was insufficient for the post production segments, namely processing, marketing and access to markets. To this effect, the farmers and grassroots organizations are not at this moment able to provide supporting services to fill the gap linked to weak public institutions”, noted Dr Naidoo.

The performance of the organizations created is not yet ensured and sustained, given their weak technical and managerial capacities, and the fact that they rely on financial resources mobilized from farmers, which are unfortunately insignificant and unstable. Similarly, the sustainability in the rice-seed supply sub sector still appears fragile, since the purchase of seeds produced by trained farmers is solely ensured by IFAD-funded projects.

“The evaluation found very low capacities on the part of organizations, lack of water all year round in the garden fields, and lack of governmental budget programmes for anchorage of projects, which are issues that weaken the sustainability of the benefits”, highlighted Dr Kodjo.

The diversifying of farmers’ income sources and the protection of ecosystems’ neighbouring lowland fields are other issues of concern. Indeed, activities promoted in the wetland rice-growing

fields did not consider the protection of the vegetation cover of adjacent mangroves and watersheds.

ducing poverty of rural populations and enable us to combat unemployment”, affirmed Kaoussou Diombera, Advisor to the Minister of State in Charge of Agriculture, speaking on behalf of H.E. Aladje Botche Candé.

“There remain considerable challenges in a number of areas, which need to be improved, such as sustainability, impact on rural poverty, natural resource management, climate change, innovation and scaling up of achieved results”, summarized Donal Brown, IFAD Associate Vice-President, Programme Management Depatment.

Workshop participants recognized the recommendations put forth by the CSPE, including the need to prioritize ecosystem and natural resource management alongside climate change adaptation in the next COSOP. Equally important will be the development of agricultural systems in the wetlands of the regions supported, and the reduction of gender inequalities through the promotion of economic opportunities for women, functional literacy, and management and skills training.

“We continue to count on the technical and financial assistance of our partners, such as IFAD, to combat poverty in rural areas. The financing of IFAD’s projects will make a significant contribution to re -

With a gross domestic product per capita of US$ 494, Guinea-Bissau is among the poorest countries in the world. Agriculture is central to the economy. Two crops dominate: rice for domestic consumption and cashew nuts, which generate 95% of the country’s exports, as a cash crop. Overreliance on growing cashew nuts exposes twothird of the population to economic shocks. Diversification, either by moving up the value chain, or improving agricultural technology and access to markets, will be essential to achieve sustained growth and combat poverty.

Donal Brown, AVP PMD, IFAD

Kaoussou Diombera, Advisor to the Minister of State in Charge of Agriculture, Guinea-Bissau

IFAD AND THE GOVERNMENT OF SIERRA LEONE STRENGTHEN STRATEGIC PARTNERSHIPS

Adelegation of IFAD Executive Board members from eight countries visited Sierra Leone, from 19 to 26 November 2022. The visit promoted strategic partnerships between the Government and IFAD on agriculture and rural development. Discussions built on the efforts that the Government and IFAD have undertaken to address the findings and recommendations that emerged from the 2020 Country Strategy and Programme Evaluation (CSPE), carried out by the Independent Office of Evaluation (IOE).

The IFAD Executive Board (EB) members who attended the visit represented Canada, China, Cote d’Ivoire, Germany, Indonesia, Mexico, Nigeria and Switzerland, and included the new Chair of the IFAD Evaluation Committee, H.E. Miguel Jorge García Winder, Ambassador and

Permanent Representative of Mexico to the Rome-based UN agencies. Dr Indran Naidoo, IOE Director, joined the delegation which also featured IFAD Management representatives, including Donal Brown, Associate Vice President, Programme Management Department, Sana Jatta, Regional Director for West and Central Africa, and Pascaline Barankeba, Country Director for Sierra Leone and Liberia.

“IFAD will never let down Sierra Leone. It is committed to accompanying Sierra Leone in its process of recovery and rural development”, affirmed Ms Barankeba.

The 2020 CSPE report highlighted that the Government of Sierra Leone has been a close partner of IFAD, providing active support in the design and implementation of projects. Over

the years, the Government has played a strong role in the conception and implementation of the lending operations. Its decision to have a dedicated National Project Coordination Unit to coordinate all IFAD-supported projects has led to good implementation of projects under its charge. In the same vein, its decision to decentralize part of its project management staff to the district level, to be closer to project activities, has been a further step in the right direction.

During the course of the visit, the EB delegation gained deeper knowledge and understanding of these undertakings, and of how they relate to IFAD’s work in Sierra Leone, including the challenges and constraints faced by IFAD-supported operations in the country. On-site visits took place at IFAD-supported projects in Bo, in the south of the country, Kenema in the east, and Makeni and Lungi in Port Loko, north of the country. During the visits, Board representatives saw first-hand how programme participants, in particular women and young people, have improved their lives, and discussed with them what further challenges they face. In addition, the EB members also held discussions with farmers’ organizations, cooperatives, small- and medium-enterprise representatives, and local government authorities.

of the finance service providers in rural areas, in order to increase their outreach and provide demand-driven services to rural communities.

The delegation’s comprehensive perspective of the country situation was further enhanced through a series of high-level meetings with Sierra Leone’s senior policymakers. In Freetown, the Members of the EB met with the President of the Republic of Sierra Leone, H.E. Julius Maada Bio; the Minister of Agriculture and Forestry; the Minister of Finance, Planning and Economic Development; the Minister of Foreign Affairs and International Cooperation; the Minister of Local Government; the Minister of Trade; the Minister of Youth Affairs; the Minister of Gender Affairs; the Governor of the Central Bank, and other Government officials at the central and local level. They also met the United Nations Country Team and the donors in the country. These meetings provided EB members with an opportunity to discuss the Government’s vision and framework for IFAD activities for the coming years.

The two projects currently ongoing in Sierra Leone are the Agriculture Value Chain Development Project (AVDP) and the Rural Finance and Community Improvement Programme Phase II (RFCIP2). The overall goal of the AVDP is to improve the livelihoods, food security and climate change resilience of rural farming households in Sierra Leone. With regard to the RFCIP2, the programme seeks to strengthen the capacity

IFAD is the largest donor in Sierra Leone’s agricultural sector, and it has supported the government in its fight against the Ebola outbreak from 2014 to 2016. When many institutions closed or downsized their operations, IFAD-supported community banks and financial services associations known as village banks remained the sole providers of banking and financial services in most areas of the country. More recently, the Rural Poor Stimulus Facility, implemented through the Agricultural Value Chain Development Project, has helped mitigate the negative impacts of COVID-19 on the livelihoods of the small-scale farmers and other rural people across the country.

Kalima farmers association

Miguel Garcia-Winder, Mexico Perm. Rep. to UN in Rome Indran A. Naidoo, IOE Director

Freetown Fashion Design and Creative Arts Network*

On Friday 16th December 2022, after months of working together to identify opportunities for collaboration between the fashion design industries and creative communities, the Freetown Fashion, Design and Creative Arts Network was launched.

With support from the Overseas Development Institute (ODI) and Labrum (a menswear fashion brand fusing British tailoring and West African design) the platform will be used to develop and channel opportunities for skills exchanges between designers, broker networks and alliances amongst textile manufacturers and more broadly support business development and investment in Freetown’s fashion industry. The ultimate aim is to open the Milan market to Freetown fashion, thereby growing the industry and creating jobs.

Marta Foresti, ODI project lead and Foday Dumbuya, Labrum founder and creative director, came from London to participate in the launch event and to engage with Freetown-based creatives who have been working on the concept of the Freetown Fashion, Design and Creative Arts Network for several months.

“We were excited to listen to three Freetown creative makers share reflections about their experiences in London and Milan in July and September this year. It was also a pleasure to listen to a recorded message from Mayor Giuseppe Sala of Milan” they said.

DEVELOPING

Global professionalization FEATURE

STORY

DEVELOPING EVALUATIVE CAPACITY professionalization and contextual adaptation

“If you give a man a fish, you feed him for a day. If you teach a man to fish, you feed him for a lifetime.” So goes the old adage. The UN took this lesson to heart a long time ago, and for the past decades has been endeavouring to develop local capacities. The international evaluation community has followed suit, with efforts focusing on building local evaluative capacities at national and sectoral levels. Emblematic is the work of the National Evaluation Capacities series.

The growth of the NEC series responded to the unmet need of national governments for an evaluation-related training platform to exchange ideas and plans with development counterparts. The NEC set out to provide tailored training opportunities for governments, customized to the key development issues being faced by countries on the ground. Over the years, the series has afforded a safe space for candid discussions on the challenges in conducting oversight in political contexts – where transparency and democracy were not always supported –, gaining legitimacy through strategic partnerships with host governments, networks and associations.

The involvement of the funding partners of the Global Evaluation Initiative (GEI) in NEC 7 has further bolstered the series by helping to explore important topics around building resilient national evaluation systems. IOE is one of these partners.

The 2022 IFAD Evaluation Manual – the first jointly prepared by IOE and IFAD management – is a clear example of the importance that IOE affords to capacity development. Conceived as a living document, the 2022 Manual draws on contemporary evaluation literature and advances made since the launch of the 2030 Agenda for Sustainable Development, such as the notion of transformative change and addressing sustainability and climate resilience.

The recent launch of the Manual’s on-line training course (presented a few pages ago), and the growing attention towards using information and communication technology for evaluation, offer further examples of IOE’s efforts to produce capacity development tools

Against this backdrop, Independent Magazine sat down with Fabrizio Felloni, IOE Deputy Director, to discuss perspectives, thoughts and insights related to the world of evaluation capacity development.

Good afternoon, Fabrizio.

Good afternoon, Alexander.

The traditional approach to Evaluation Capacity Development focuses on training, attending courses and getting certificates. Based on your experience, what do you think are the strengths and weaknesses of this approach?

There is no doubt that participating in training programmes will be useful and this is true for more or less any subject. Knowledge and ‘good practices’ evolve and at some stage we all need to brush up on our skills or upgrade them. What we learnt and was state of the art twenty years ago may not be entirely so now.

Having said this, we also know that practice and application of what we have learnt from regular studies or training courses is fundamental. We can come back later to this item.

Fabrizio Felloni, IOE Deputy Director

I would suggest that capacity development in evaluation is not limited to building capacity of individuals, as important as it may be. We also need to strengthen the capacity of institutions. From an IFAD perspective, for example, we would like to see capacity for evaluation strengthened in the Ministry of Agriculture or any agency that has a prominent mandate for rural development. What are the areas of capacity? I can think of things such as issuing bylaws or perhaps a policy for evaluation, which defines those who will be responsible for conducting an evaluation, and how independent they will be from those who manage the programmes that are evaluated. Equally important is understanding who will be responsible for following up on the evaluation recommendations, and who will have the oversight and scrutiny of the whole process. This is fundamental not only to bolster the legitimacy and the credibility of the evaluation process, but also to make sure that evaluation supports changes and improvements in development interventions and policies, and in the organizations that are responsible for the same.

Institutions also need reference to codified good practices and standards. They need principles and methodologies for evaluations to be rigorous, transparent and consistent in their conduct.

We at IFAD have a mature and quite sophisticated evaluation function. We have an Evaluation Policy, a multi-year strategy for evaluation and a manual. As of 2022, the Evaluation Manual covers independent evaluations carried out by IOE as well as self-evaluations undertaken by management. This does not mean that national agencies in developing countries should try to mirror exactly our approach. After all, they are not a multilateral organization. Yet, they will need to define who is responsible for evaluation and how the evaluation process works. This means putting in place something like a policy or statute for the evaluation function. They may need some prioritization of what they want to evaluate and how, as well as some codified collection of good practices (i.e., a methodology). This is where IFAD’s experience can provide insights to national organizations, although what works in an international agency does not necessarily work in a national agency, and thus needs to be adapted.

Lastly, some development practitioners, when discussing evaluation capacity, use the expression of enhancing the capacity of an ‘evaluation ecosystem’. This came out clearly from the National Evaluation Capacity Conference of Turin, in October 2022. The ecological metaphor relates to the diversity of actors involved in the evaluation system, such as: (i) public sector agencies in charge of public programmes; (ii) policy research outfits; (iii) private entities; (iv) civil society organizations; (v) citizens; and (vi) mass media. These entities have different roles and interests in evaluation and may need to be given capacity and space. As an example, public sector agencies may need to enhance capacity for commissioning and oversight of evaluation. Civil society organizations need capacity (and space) to articulate demand for evaluation of public programmes and capacity to draw from findings to prepare their campaigns. Policy research outfits and private entities (including private foundations) may need support on how to conduct an evaluation. The media need to be aware of reliable evaluation sources to engage the audience in a debate that is meaningful and correctly informed.

How do you see on-the-job training as a means to build evaluative capacity?

As I was alluding to before, it is essential. Unless we apply what we have learnt, we are bound to forget most of it very quickly. It is for this reason that many professional training programmes, even when delivered as a part of an undergraduate or graduate programme, include some period of practice or internship. While this applies generally, it is even more important in the realm of evaluation, for two reasons.

First, evaluations happen in an institutional context. They are typically conducted or commissioned by a government, an international organization, or a non-governmental entity. This requires an understanding of the organizational setup, goals, roles, incentives and hierarchy. Second, an evaluation happens in a ‘political’ environment. By this, I mean that we evaluate programmes that involve many and diverse stakeholders, wielding different levels of power. Moreover, programme implementation is inevitably shaped by the interactions between partners that are unequal.

Please do not get me wrong. I am not arguing that evaluators should be involved in the politics of organizations. Evaluators are and should remain technical people. However, they should be perceptive enough to understand organizational power structures and their dynamics.

What are your views on the notion of the professionalization of evaluation? Does it help to enhance quality by creating internationally recognized standards, or does it run the risk of creating an ‘evaluation guild’, which could be exclusionary and elitist, and thus raise equity issues in contexts characterized by weaker evaluation capacity?

Professionalization and adaptation to specific contexts are not at odds. In principle, the more we professionalize evaluations, the more we should learn how to adapt. Besides, I believe in the importance of standards. While agreeing that we need to adapt to the local context, I think it is misleading to dismiss evaluation standards and good practices such as the evaluation criteria.

Some argue that standards and criteria are ethnocentric and tilted ‘to the North’. I am not entirely convinced. There are questions, such as those on relevance, effectiveness and efficiency of a development intervention, which are universal. If we cannot answer these criteria and questions, it is difficult to provide any meaningful information through an evaluation. The way in which we answer these criteria and questions can change, of course, and evaluators need to understand, appreciate and respect the perspectives, the values and the paradigms of the people they interact with

and the context in which they operate. They need to be open to adapting the panoply of tools that they use.

In recent decades, we have seen the rise of approaches such as empowerment evaluation, indigenous evaluation and feminist evaluation, all of which seek to challenge established approaches. Some may disagree with me, but I believe that some standards, such as the evaluation criteria, or the principles of independence and impartiality can be adapted to fit these approaches as well.

Stemming from the previous question, is there a need to balance international evaluation standards for capacity development, on the one hand, with tailored capacity development efforts focused on local contexts and local needs? If so, how might we achieve this?

I say that we need to adopt and adapt international standards to the local context, to the local stakeholders and to the ultimate end-clients of development programmes. At the end of the day, standards should not be seen as conceptual straitjackets, but as practices that have been successful for a number of times in a given setting. We should not dismiss the standards, we need to understand what they are meant for. They may help us strength-

en the quality of evidence; they may help us strengthen the quality of reports. Importantly, they may help us protect the integrity of an evaluation and build an evaluation process that is impartial, on the one hand, whilst also ensuring inclusiveness and ability to integrate the perspective and experience of under-represented groups, on the other. At the same time, we do not want to be rigid and build over-formality when it is unnecessary and hampers dialogue.

I would also add that standards need to be adapted to the institutional and decision-making framework where we operate. It is one thing is to evaluate a programme that is managed by voluntary groups, grassroots organizations and civil society representatives that interact with local government. It is a very different matter to evaluate a programme run by a ministry and by public agencies under the tight supervision of the central government. Evaluation in multilateral organizations is different further still.

At the international level, evaluation networks and initiatives have sought and continue to seek to advance evaluation capacity development through different approaches, leveraging different entry points. These include the National Evaluation Conference series, the Global Evaluation Initiative (GEI), the Evaluation Cooperation Group (ECG) and the UN Evaluation Group (UNEG). Given the on-going global challenges and transformations, what areas of intervention do you think could be prioritized in order to fast-track evaluation capacity development?

These initiatives have sometimes different constituencies but need to work in synergy. The UNEG and ECG are communities of practices of representatives of evaluation offices in multilateral organizations. They exchange knowledge on practices, methodology, major evaluation findings, trends in evaluation

products, topics for evaluation, and interactions with evaluation stakeholders.

NEC is a forum for discussion and exchange of experiences for evaluation practitioners, particularly from developing countries. In the recent editions of NEC, we have noticed more and more presentations on M&E systems in the public sector, focused on the type of data they collect and the use of big data. Interestingly, we have also started to see presentations by civil society organizations, either about their experience in evaluating their own programme, or about their role as participants in steering groups for the evaluation of public programmes.

GEI is a network of international organizations, aiming at supporting capacity at the individual and institutional level. In the network, there are also institutions that specialize in post-graduate and professional education and run training programmes. Some of the organizations that are members of GEI are also members of UNEG and ECG, and contribute to the NEC (in fact, IOE is engaged in all four of these networks!). Synergy is necessary. As to what to prioritize, here I hold my IFAD cap and would prioritize the institutional technical backstopping to agencies in developing countries that request capacity support. For example, a ministry of agriculture that wants to set up an M&E system for rural development interventions.

By the way, if we want to engage in individual capacity development, bilateral and multilateral agencies can help hone the skills of young evaluators, for example via internships and temporary assignments or consultancies. I understand GEI plans to come up with a database of young evaluators that have gone through some form of accreditation via renowned international training centres. Evaluation offices are always looking for interns or young consultants, so there is a good match in principle.

At country level, certain sectors – notably health, education and agriculture – have advanced further in building evaluation systems than cross-sector systems. Why do you think this is the case, and what can be learned from these sectoral approaches?

This is in part historical heritage. Attention to evaluation was borne in the USA out of the studies that tried to assess the performance of the so-called Great Society initiatives, launched by President Lyndon B. Johnson, in the 1960s. Many of these were programmes in the education and health sectors. Perhaps this is also a reason why some evaluation practitioners resent the Western orientation of evaluation standards. The reality is that, after six decades, international evaluation practices are no longer following a single country approach. Nonetheless, those early studies were seminal contributions.

The second reason is that educational and health programmes tend to be standardized, and are formulated with explicit goals and results to be achieved that can be expressed in numbers, thus lending themselves to evaluability. On the positive side, this shows the importance of designing a programme with evaluability in mind, and of making programme goals explicit and measurable in others sectors too. Having said this, I would caution against trying to have all goals expressed in numbers, in all programmes. I would also caution against the rhetoric of taking randomized control trials and quasi-experimental methods asthe ‘gold standard’, or as the only legitimate way to conduct evaluations. Anyway, consensus on the gold standards has waned in the past decade.

Looking in-house, how does IOE advance the capacity of its professionals? Are changes in said capacity measured and, if so, how?

Our workforce is well qualified and committed. That said, we need to keep pace with the evolution of the profession and with the approaches and tools that emerge. We need to keep learning. When we prepared the 2022 Evaluation Manual, we had to take stock of emerging concepts and perspectives that were linked to Agenda 2030. As an example, the notion of ‘nobody left behind’ brings to the forefront the issue of social justice and equity, which is at the core of IFAD’s mandate, and thus requires attention when we design evaluation instruments. The emphasis on transformative change calls upon evaluators to have the capacity to identify transformation in the way in which systems perform. That calls for attention when we prepare the evaluation approach, and some brush-up in our knowledge of system analysis.

I would also like to highlight the importance of using information and communication technology for evaluation. Geo-based tools are particularly important for evaluations at IFAD. We need to stay abreast with new tools, with the type of data they can generate, with the type of questions they can help us answer, and how they can help us save resources. IFAD is moving forward on this, fortunately. We in IOE have launched an internal initiative on geo-based tools.

We care about the progress and professional growth of our staff. We calibrate the work programme of staff members considering expertise, skills, track record and past performance. Ultimately, we aspire to ‘graduating’ our staff to more and more complex tasks. However, in many cases, the perfect entry point is a project-level evaluation or a project completion report validation. The latter is based on desk review. While new staff may be impatient to go to the field, it is very important to familiarize with the structure of a typical IFAD project, with the application of IOE’s methodology and criteria, and to hone writing skills (yes, writing cogently and concisely is a challenge for anyone!). I would say that patience and humility are key skills for evaluators, and are those that will allow them to move forward to evaluations that are more complex.

We do help colleagues by giving them exposure to training, in any form, but we want them to put learning in practice, and share their new skills with other colleagues in our office.

Any final thoughts or insights?

To conclude this interview, if I may, I would like to come back to a specific dimension of institutional capacity for evaluation, which I mentioned before. When we issued our IOE Multi-year evaluation strategy for 2022-2027, we committed to engage more in evaluation capacity development. We want to be realistic and prudent in the use of our resources, and do not want to create a new unit for capacity development in IOE. Instead, we take the opportunity of our evaluations to identify countries and national agencies that have special needs to develop M&E systems. In consultation with IFAD Management, we help them interact with specialized networks, such as GEI. We see ourselves in a facilitating role to start-up initiatives. Governments need to be in the driver’s seat, and IFAD can provide further support.

Thank you, Fabrizio.

Your’re welcome, Alexander.

TIME IS RIPE FOR M&E TRAINING IN THE UZBEKISTAN AGRICULTURAL SECTOR

Strengthening national monitoring and evaluation (M&E) systems is at the core of IFAD’s new country strategy in Uzbekistan. The priority afforded by the Government to the agriculture sector, combined with the dynamic institutional set-up of the country, present a fertile ground for partnership and collaboration around national evaluation capacity development. Senior representatives and technical experts from the United Nations Rome-based agencies (RBAs), and the Global Evaluation Initiative (GEI) came together to discuss these and other prospects on 6 December 2022.

“Opportunities exist to create country-specific M&E training for the agricultural sector in Uzbekistan”, affirmed Heather Bryant, GEI Evaluation Advisor.

The Independent Office of Evaluation of IFAD (IOE) hosted the meeting, which featured the participation of the evaluation offices of the Food and Agriculture Organization of the United Nations (FAO) and World Food Programme (WFP) , as well as of members of IFAD’s Near East, North Africa and Europe Division. During the meeting, Dugan Fraser, GEI Programme Manager, and Heather Bryant presented many facets of the work carried out by GEI, including at the strategic and country level. The meeting opened the door for discussions on present and future areas of collaboration, partnership and joint endeavours. These include the possibili-

ty of leveraging the GEI platform, institutional network and technical expertise to strengthen M&E capacities in Uzbekistan, in the agricultural sector.

“The GEI is a possible partner to help train trainers in sector-wide M&E in the rural space, in Uzbekistan”, stated Fidy Rajanson, IFAD Country Director for Uzbekistan and Georgia.

Indran A. Naidoo and Fabrizio Felloni, IOE Director and Deputy Director respectively, highlighted the support of IOE in helping to broker connections among country needs that the Office uncovers in its evaluations, such as those in Uzbekistan, and existing initiatives that have the ability to address those needs, such as the GEI.

“In the spirit of the National Evaluation Capacities (NEC) conference series, we want to expand

Left to right: Mònica Lomeña-Gelis,

existing M&E initiatives, and support synergies by identifying programmatic linkages”, stated Dr Naidoo.

This approach resonated with representatives of WFP and FAO, who noted the importance of leveraging existing platforms and using existing resources to complement the GEI, such as the Evalforward.org community of practice. These efforts should be geared towards broadening the knowledge about GEI across country offices, in order to provide clear and practical support to work being implemented on the ground.

the current status of M&E systems at country level, including in terms of the extent to which what exists is used to inform policymaking and programme design. Having come to a common understanding with the Government, what follows is the development of a tailored strategy for capacity development at systemic and individual levels.

“We are establishing a database on higher education offerings and a repository of training courses on evaluation that are available through GEI partners. This will be available later this year on GEI’s Better Evaluation platform”, Mr Fraser explained, in this regard.

The GEI is a global network of organizations and experts supporting the governments of developing countries with strengthening monitoring, evaluation, and the use of evidence in their countries. IOE is an active member and funder of GEI, to which it has provided substantial contributions. The GEI was signed into existence in 2020, when a memorandum of understanding was drawn up between the IEO of UNDP and the IEG of the World Bank.

Mr Fraser agreed with this approach, and underscored the rationale for a two-pronged approach to the relationship between RBAs and GEI, focused specifically on building capacities of governments vis-à-vis the agriculture space and smallholder farmers.

“First, there is a need for practical work at the country level, where tangible impact can be shown. Second, knowledge products should be distilled and shared across platforms, also tapping into the resources that Evalforward.org provides”, Mr Fraser said.

The proposed way forward is in line with the modus operandi of the GEI, which currently operates in 24 countries through regional Centers for Learning on Evaluation and Results (CLEAR). The GEI advocates for carrying out analyses of

RESOURCES

Lomeña-Gelis, Fabrizio Felloni, Indran A. Naidoo, Dugan Fraser, Christiane Kuhn, Heather Bryant

Inclusive and tailor-made access to financial services achieved in India

with project stakeholders and beneficiaries. Building on the evaluation evidence, IOE also conducted an analysis on the effects of the project’s financial services at the household level, making use of a grounded theory approach.

IOE’s evaluation report found that the project’s approach to provide loans through community-based organizations (CBOs) and financial services arrangements with local banks proved successful.

their current incomes by providing quick access to small loans. In addition, the local bank arrangements contributed to ensure fiscal discipline among the benefited communities.

An IFAD co-funded project has enabled inclusive access to financial services for all vulnerable targeted populations in India. By working with insurance companies and local banks, the Post-Tsunami Sustainable Livelihoods Programme for the Coastal Communities of Tamil Nadu (PTSLP) was able to develop loans and insurance schemes tailored to local needs. This, according to the Project Performance Evaluation report (PPE) published by IOE.

The PTSLP was designed to support long-term recovery from the effects of the Indian Ocean tsunami on 26 December 2004. The project came into force in July 2007 and was completed in June 2020. The PPE carried out a desk review of project-related documents, interviews and discussions

Download

The project demonstrated that CBOs can be a suitable vehicle for financial inclusion when capacity-building covers various topics and is provided over a significant period. Services such as the vulnerability reduction funds (VRF) were highly competitive and widely popular among the population, particularly women. During COVID-19 lockdown periods, the VRFs also enabled people to maintain

The project also showed that when women’s CBOs are the channel for financial services, women’s skills, confidence and status in their households and communities are enhanced. Supporting women through CBOs increased their access to loans and expanded opportunities to establish microenterprises, thus contributing to their economic empowerment. The project also contributed to women’s political empowerment, leveraging their voices in the communities and organizations.

At the same time, the PPE report also noted that the project’s focus on rebuilding and strengthening livelihood activities that communities were already familiar with, reinforced occupational segregation, which tends to place women in low-paid work. Moreover, while the project’s approach helped to ensure the viability of women’s microenterprises and income-generating activities, the enterprises promoted were of low returns, due to gender biases and lack of scale and value addition.

Massiel Jiménez Lead author, India PPE report

Other areas of concern include the declining fish stocks caused by over-fishing and pollution, which strain the fisheries-based livelihoods. While artificial reefs helped to regenerate stocks and biodiversity in some inshore areas, only a limited number of fishers can operate within these reefs. Climate change adds to these effects as higher sea temperatures affect fish stocks, more extreme weather causes rough seas and reduces the number of fishing days, and recurrent cyclones damage productive infrastructure.

Looking ahead, IOE’s evaluation report highlights the importance of developing a sound and systematic sustainability strategy focusing on managing key risks that may hinder the overall sustainability of benefits in the long term, including those of small-scale fishers in Tamil Nadu. In addition, the PPE recommends developing and implementing a multisectoral strategy for coastal community resource management and livelihood development.

India is the world’s second largest country by population, and eighth largest by area. The economy grew by an average of 7% from 1997 to 2017, and India moved from low-income to lower-middle-income status in 2009. This, along with structural reforms and government investment in social protection, contributed to a reduction in the poverty headcount from 37% in 2004 to 22% in 2011. IFAD has been working in India for more than 30 years.

Celebrating the golden era of the Hindi film industry*

In the late 1940s, when a young independent India was still finding direction, Bollywood was already a well-established industry with busy film studios and numerous movie stars to boast of. When JH Thakker arrived in the metropolis in 1947 as a migrant from Karachi, little did he anticipate that some of the city’s most renowned names will become frequent visitors to his photo studio that stood in Dadar in Central Mumbai.

The golden period of the film industry photographed by him is now being celebrated through an exhibition titled “Sitaare Zameen Par” at the Kiran Nadar Museum of Modern Art in Noida. “Thakker quickly understood the exceptional role assigned to him — that of shining the glittering light of these stars on the earth, while masquerading as any other crew member on the movie set. Thakker not only utilised his technical acumen and astute sense of the “chiaroscuro” (the light and dark gradation of tones) but deployed imagination to formalise and posture glamorous subjects as the icons of popular romance. Within these “temples of desire” exquisitely perfumed by timeless music, song, and dance compositions, Thakker’s practice can be contextualised as that of an idol-maker,” notes Roobina Karode, director and chief curator, Kiran Nadar Museum of Art.

@Nalini Jaywant Photograph by JH Thakker, silver gelatin print collection and image of the Kiran Nadar Museum of Art

EVALUATIONS HAVE

TO SIGNIFICANT INSTITUTIONAL CHANGE IN IFAD

T“he evaluations of the Independent Office of Evaluation of IFAD (IOE) have contributed to remarkable institutional changes in IFAD. They have served to promote accountability through measuring and reporting on results. They have also generated lessons and made key recommendations for improving IFAD operations”, stated Indran A. Naidoo, IOE Director, describing the impact that IOE has had in shaping reforms within IFAD. This, during the 2022 Fall meeting of the Evaluation Cooperation Group (ECG).

The World Bank Independent Evaluation Group (IEG) hosted the ECG meeting, which took place in Washington D.C. on 14 – 15 November 2022. The in-person meeting brought together the heads of the evaluation offices of the multilateral development banks and other institutions members of the ECG. IOE was represented by Dr Naidoo who attended in person, and Fabrizio Felloni, IOE Deputy Director, who attended virtually.

Organizations that participated in the event included the African Development Bank, the Asian Development Bank, the Black Sea Trade and Development Bank, the Central American Bank for Economic Integration, the European Bank for Reconstruction and Development, the European Investment Bank, the Inter-American Development Bank, the International Monitory Fund, the Islamic Development Bank, and the

Organisation for Economic Co-operation and Development. In addition, the UN Evaluation Group and the Global Environmental Facility attended as observers.

During the course of a moderated roundtable conversation, on the second day of the gathering, Dr Naidoo presented examples of transformative and impactful corporate-level evaluations (CLEs) that have made a significant contribution to the work of IFAD in recent years. These included the 2016 CLE on IFAD’s Decentralization Experience, which informed the IFAD 2016 Action Plan on Decentralization; the 2018 CLE on IFAD’s financial architecture, which supported reforms of the Debt Sustainability Framework at IFAD and of the Fund’s first credit rating; the 2019 CLE on IFAD’s Engagement in Pro-poor Value Chain Development, which supported the preparation of new Guidelines for Value Chain Development at IFAD; and the 2021 Joint Evaluation on the Collaboration among the United Nations Rome-based Agencies, which supported the preparation of a new Memorandum of Understanding for collaboration between the agencies.

“The purpose of an evaluation office should be to charter processes that ensure that every evaluation leads to multiple levels of conversation and consideration by the stakeholders”, highlighted Dr Naidoo in his intervention.

The roundtable conversation also zoomed in on the challenges of institutionalizing an independent evaluation function, by drawing on the experiences of various institutions. Discussants who joined Dr Naidoo included Alison Evans, IEG Director General; Ivory Yong-Prötzel, Director of the Inter-American Bank Office of Evaluation and Oversight; José Efrain Deras, Chief of the Central American Bank for Economic Integration Office of Independent Evaluation; and Jan Willem van der Kaaij and Sabine Bernabè, European Investment Bank, Inspector General and Head of Evaluation, respectively.

“Evaluation is often met with resistance, pushback and sometimes excitement, depending on who the evaluand is and what the subject of evaluation is. We need to be quite clear: evaluative work is political. There is always pressure. The evaluands may not always appreciate the perspectives coming in. Therefore, the reason for independence is a critical one. You want a unit, within an organization, that is able to conduct its work without fear, favour or prejudice, that is not under any political or administrative pressure”, underscored Dr Naidoo in his intervention, elaborating on the critical importance of the independence construct within an evaluation office.

The two-day meeting was organized under three overarching themes, namely ‘building a work programme: striking the right balance’; ‘challenges and opportunities in self-evaluations of private sector operations’; and ‘assessing country programmes’. The event also afforded opportunities for participants to share key trends affecting their institutions, and elaborate on what these means for their evaluation functions.

In this context, Dr Naidoo explained that IOE has accommodated demands from IFAD’s governing bodies and senior management, during the past two years. This has meant diversifying products, and making high investments in unpacking and repackaging evaluations via the Office’s new website. Recent big ticket items include evaluations linked to IFAD’s leadership, such as those on the Fund’s decentralization and knowledge management. The IOE Director also addressed the perennial question of oversight being either learning or accountability, arguing that learning is a by-product of a robust and independent evaluation.

The ECG was established in 1996. The African Development Bank is the current Chair of the Group. IFAD served as Chair in 2017, and is scheduled to serve as Chair again in 2025.

Transformative solutions needed to avoid catastrophic consequences

“Climate change is a very tangible and concerning reality. Life on earth faces catastrophic consequences unless drastic and immediate action is taken. Transformative solutions are needed. Evaluation can play an important role in this regard, by capturing and presenting evidence of said transformation”, affirmed Indran A. Naidoo, Director of the Independent Office of Evaluation of IFAD (IOE), in his keynote speech during the opening of the event entitled ‘Transformational change towards a sustainable future’. The three-day meeting took place on 7-9 September 2022.

Set against the prestigious backdrop of Wilton Park House, the ‘by invitation only’ event was the third dialogue in the series on transformational change, which focuses on climate change and the environment/development nexus. A host of high-level evaluation experts flew- n to bring concrete, evidence-based guidance on designing, supporting and implementing transformational change for a sustainable future.

The climate crisis is a defining issue of our time. It continues to threaten societies and its economic, social and environmental impacts are a challenge for us all. The time for incremental changes has passed and governments and organizations need to design and implement dramatic policy and infrastructural changes to ensure a sustainable future. Dr Naidoo underscored the importance of ensuring an evidence base for holistic climate change adaptation solutions that build climate, environmental and developmental resilience together in order to ramp up climate finance and environmentally sustainable policies and practices.

“Many governments face significant challenges to incentivize sustainable climate adaptation responses. Evaluations could and should be looked upon as having the potential to play a critical role in this regard. The transformative change that we are looking for is the shift in governments’ and international agencies’ paradigm of agricultural interventions, moving from an anthropocentric approach to one which is more environmen-

tally focused”, the IOE Director highlighted.

Event participants echoed this perspective, calling for major changes in our economies and societies to be effectuated in re-

cord time. In order to succeed, these changes will need to be systemic, comprehensive and include voices and world views from a variety of cultural backgrounds.

The conference brought to the forefront examples of transformative pathways and actions that could truly work to mitigate climate change risks and foster systemic resilience, as societies innovate, experiment, and scale up effective climate responses as the world transforms in the aftermath of the COVID-19 pandemic, and the global shocks caused by the responses to the Ukraine crisis. The evidence presented helped to better understand how systems are connected and interdependent; what changes to systems need

to happen for transformational change to take place; and what processes might generate commonly held beliefs about how to transform cultures, values and societies.

In closing, participants agreed that options exist for creating stronger, more coherent research, science, monitoring,

gy and Industrial Strategy.

Following closure of the conference, IOE endorsed the high-level note titled ‘Promoting transformational change for a sustainable future: 5 calls for action’. In order to stop actions and policies that hurt the planet, the note calls upon governments to start taxing both industries and consumers who must pay the costs for rectifying and remediating practices that are destroying our future. The document also underscores the need to remove barriers for introducing new energy technologies. Barriers include disinformation and targeted campaigns by companies and groups that invest in and are supportive of unsustainable and extractive use of natural resources.

evaluation and learning systems that can provide in a timely and appropriate fashion the evidence and knowledge needed to scale up transformations.

The event was organized in partnership with the Climate Investment Funds, Global Environment Facility, UNDP Independent Evaluation Office, and IOE, with support from the Green Climate Fund, Institute of Development Studies, International Evaluation Academy, the German Institute for Development Evaluation and the UK Department for Business, Ener-

The IOE-supported note includes a total of five calls to action. In addition to the aforementioned first call, the second call pinpoints the need for societal partners to create a shared and inclusive vision of a transformed world. The third call notes the need for increased support for transformational change initiatives, with rich countries bearing a larger share of the costs of responding to the crisis. The fourth call underscores the need to guarantee open access and transfer technologies that help humanity to survive and adapt to climate change. The fifth call highlights the need to create a global knowledge centre for policies, programmes and interventions aimed at transformational change, including practical guidance on how to transform production and consumption and achieve sustainability.

Support to land and water management in The Gambia: efforts and challenges to innovation

Raymond S. Mubayiwa

The National Agricultural Land and Water Management Development Project (Nema) introduced a matching grants scheme under the Capital Investment Stimulation Fund. For The Gambia, this was a new and innovative concept. The opportunity was not seized, however, as the scheme was not appropriately designed and targeted to achieve the expected outreach and impact. This, accordingly to the Nema Project Performance Evaluation (PPE) report published by IOE.

Nema was implemented in The Gambia between 2012 and 2020. The project’s development objective was to increase income through improved productivity based on sustainable land and water-management practices. This was to be achieved

through watershed development; agricultural commercialization; and project facilitation. To evaluate the project, IOE adopted a mixed modality approach, which included a desk review of project documents, remote interviews with key project stakeholders, and in-person interviews in the project communities, given the context of the COVID-19 pandemic.

failure by applicants to fulfil all conditions such as down payments, lack of collateral and bankable business plans.

These shortcomings aside, arguably Nema’s greatest challenge can be found in the project’s rice infrastructure development approaches. These were constrained by gaps in design, quality of construction, and operations and maintenance. Furthermore, the social, environmental, economic and gender aspects of rice production in the different ecologies in The Gambia were not always sufficiently considered, especially for tidal irrigation. In final analysis, constructed infrastructure often did not have the technical quality to make rice production profitable and sustainable, or even permit farmers to use it.

Nema’s matching grants aimed to promote investment and access to productive farm machinery, equipment and infrastructure to enhance productivity. The PPE found that while these were innovative and new to IFAD in The Gambia, they were not sufficiently oriented towards market businesses, IFAD target groupswomen and youth -, and only partly linked with Nema’s infrastructure development. In the end, the total number of grants delivered was relatively small. This was a result of

Other shortcomings included the limited attention given to climate-smart infrastructure and production technologies at project design, as integration of climate change adaptation and natural resource management in mini-watershed development and infrastructure planning was not sufficiently considered. In addition, the projects also established fewer linkages to private sector market demand than envisaged at design. This was in part because many of the activities came at a late

Lead author, The Gambia PPE report

stage.

On the upside, the PPE found that vegetable gardens had strong achievements in terms of productivity and sustainability, including beyond the Nema project. This was largely because the gardens followed a standard design with clear specifications for materials. Evidence of success was mostly observed in the women- and youth-managed vegetable gardens, which were found to be profitable and economically empowering. Nonetheless, investments in women and youth vegetable gardens will only be sustainable in the long run if they are firmly embedded in strong market and demand linkages.

In response to the challenges observed, IOE’s report advances a series of recommendations, including the need to support the development of a new strategy and national master plan for rice development in, ensuring that they are informed by watershed analyses. Furthermore, the PPE also suggests moving ongoing and future vegetable schemes consistently towards market, demand and private sector orientation, and advises to address the root causes of gender inequality and discrimination, using contextually appropriate upstream and downstream strategies.

The Nema project followed a long line of IFAD projects in The Gambia that were dedicated to lowland rice production and targeting women farmers.

Reviving fading storytelling*

Some two decades and half ago, storytelling was a significant activity in Gambian communities. The stories are commonly fictitious but they can help children and youth to draw lessons of moral consciousness from them that can stimulate their maturity and build cautiousness.

The Rural Child, which is a rural youth organisation, is now trying to revive this vibrant culture of storytelling in rural communities alongside the International Organisation for Migration Migration Information Center (MIC). The revival process, named Moonlight Storytelling, helps young people to draw moral lessons from the stories narrated by migrant returnees that can remind them about the dangers associated with irregular migration. The initiative also promotes excellence in informal education and supports young people with learning aids.

Since its conceptualization, the Moonlight Storytelling has already reached four communities in Kombo East District.

Looking at the environmental consequences of IFAD’s climate change adaptation related projects

The forthcoming report by the Independent Office of Evaluation of IFAD (IOE), titled ‘Thematic Evaluation of IFAD’s Support for Smallholder Farmers’ Adaptation to Climate Change’, “found that 70% of IFAD’s climate-adaptation-related projects could end up harming the environment, to different degrees”, explained Suppiramaniam Nanthikesan, Lead Evaluation Officer at IOE, and lead author of the evaluation, during the 2022 Asian Evaluation Week (AEW).

Dr Nanthikesan served as a panellist in an online session of the AEW, during which participants learned about factors and strategies that make evaluations influential and enhance

utilization and uptake. Juha Uitto, Director, IEO, Global Environment Facility (GEF), facilitated the session, which saw panel presentations by Oscar Garcia, Director, Independent Evaluation Office, UNDP; Analiza Rebuelta-Teh, Undersecretary, Department of Environment and Natural Resources, Philippines; and Ashwani K. Muthoo, Director General, Independent Evaluation Office, New Development Bank

“Our actions have consequences to ecosystems whether we intend them or not. Therefore, it is important that we are conscious and aware of what the environmental impacts of our actions are. For instance, although it is among the sectors most impacted by climate change,

agriculture contributes 17% to greenhouse gas emissions”, noted Dr Nanthikesan in his intervention, contextualizing the percentage of IFAD projects that cause some degree of harm to the environment.

Systematically assessing and analysing these consequences and their impacts is essential in order to produce a credible body of knowledge, to help ensure its utilization and to hold organizations accountable. In this regard, it much easier for evaluations if the organization itself recognizes that the environment is an issue that they have to worry about in their interventions, and makes it a corporate priority, Dr Nanthikesan highlighted.

IFAD was one of the pioneers in the UN system to recognize this need, and to evaluate the intended and unintended environmental consequences of development interventions. Since 2011, IOE has institutionalized environmental and social considerations in all its evaluations. Today, its reports are required to assess the performance of projects along a prescribed set of evaluation criteria. These include how well the interventions promoted

environmental and natural resources management and strengthened adaptation to climate change.

“Other organizations could benefit from IFAD’s experience in institutionalizing evaluating environmental impacts in all its evaluations, including through evaluation policies. There should be clear guidance on how to mainstream environmental consequences in all evaluations. You also need the necessary resources and capacity in the evaluation team to carry out the assessments”, Dr Nanthikesan stated.

The AEW, now on its 7th year, is a leading evaluation knowledge sharing platform in the Asia and Pacific region. Held on 13-15 September 2022, in hybrid modality, the subthemes of this year’s event were ‘innovating and retooling evaluation towards resilient recovery’, ‘driving collective actions on country-based evaluation’, and ‘influential evidence-based evaluation for green, healthy and inclusive societies’.

Dr. S. Nanthikesan Lead Evaluation Officer, IOE

Profile

Diagnostic assessment of existing institutional capacity necessary to implement complex interventions

The IFAD co-financed Market Infrastructure, Rural Finance and Value Addition Support Programme (MIVARF) in the United Republic of Tanzania undertook capacity-building of a wide range of stakeholders across a wide geographic area, which spread financial and management resources thin. The ambitious scope of the project diluted the capacity-building effort, with required support for institutions proving to be beyond what can be rendered in the lifetime of one project. This, according to the MIVARF Project Performance Evaluation, published by IOE.

The MIVARF supported a total of 14 value chains across the project location. The goal of the programme was to reduce rural poverty and enhance rural economic growth in the participating districts on a sustainable basis. IOE’s evaluation focused on the IFAD-financed components of the project, namely producer empowerment and market linkages, support

to institutions and systems development for the rural/ microfinance industry, establishment of a risk-sharing facility and setting up of an innovation fund.

sign lacked a diagnostic assessment of capacities and willingness of partner agencies to accept or be equipped to implement its multi-faceted and complex design in an integrated manner. Thus, during implementation, changes had to be made to suit partners’ existing capacities, and new partners had to be selected leading to substantial delays in implementation of value chains and rural finance interventions, and redesign of the implementation arrangements.

The MIVARF spanned a wide geographical area, with broad thematic and institutional focus. The project was able to reach marginalized smallholders via its value chain interventions through selection of smallholder appropriate value chains such as staples. Unfortunately, the project’s ambitious de -

The scattered and discrete nature of capacity-building for marketing groups and financial institutions, and the wide thematic, institutional and geographic focus of the project, meant that most of the groups and institutions had lingering capacity challenges. This prevented the MIVARF from systematically providing backstopping to address residual capacity gaps, and building an exit strategy into its capacity-building efforts during implementation.

The experience of the MIVARF highlights that future programmes should undertake a thorough diagnostic assessment of existing institutional capacity and willingness to implement complex interventions. Furthermore, value chain development activities should be targeted at those value chains and target groups which are characterized by suitable quality and quantity of production and capacity to participate in value chain development activities.

IOE’s evaluation report also underscores the importance of having a more concentrated geographic and thematic focus to ensure focused capacity-building, mentoring, backstopping and integrated delivery of interventions.

In the United Republic of Tanzania, despite sustained economic growth and a persistent decline in poverty, the absolute number of poor people grew from 13 million in 2007 to 14 million in 2020. Vulnerability is also reported to remain high; for every four Tanzanians who moved out of poverty, three fell into it. Access to markets is limited, particularly in the northwest and southeast, areas typically characterized by severe poverty. Lack of market access thus traps rural farmers in poverty and exacerbates inequalities between rural and urban areas.

Coexistence between people and wildlife is a national priority for sustainable development and wildlife conservation in Tanzania. As the government moves on to see the country’s tourist attractions thrive to ensure biodiversity conservation, experts on the field have alerted on the need for all stakeholders to have a wide understanding of drivers of Human-Wildlife Conflicts (HWC), human dimensions of HWC and diversity of approaches to mitigate HWC.

The government of Tanzania sees addressing human-wildlife conflict as a key goal for sustainable development and wildlife conservation in the country, hence stakeholders must work towards human-wildlife coexistence for the benefit of the nation’s people and wildlife.

Speaking at a refresher and master training classes for journalists in Bagamoyo, Mr. John Noronha (Monitoring and Evaluation Manager implementing the ‘USAID Tuhifadhi Maliasili’ Project) affirmed that HWC is an important topic due to its impacts in the country. The training addressed dynamics that threaten habitat connectivity and the longterm persistence of biodiversity in Tanzania.

“Conflict occurs when the needs and behaviour of wildlife impact negatively on the goals of humans or when the goals of humans negatively impact the needs of wildlife,” said the expert.

However, he added that it is important to recognise that human-wildlife conflicts do not result solely from the direct impacts of wildlife on people or vice versa but may often involve disagreements between stakeholders over conservation objectives.

Peaceful coexistence between people and wildlife is vital*

Use of artificial intelligence boosting relevance of evaluations

Can methodological innovations – such as the use of artificial intelligence –boost the impact of evaluations in ways that are both practical and useful? The Artificial Intelligence for Development Analytics (AIDA) platform is doing just that: improving the speed, accuracy and relevance of evaluative results. Oscar A. Garcia, Director of the Independent Evaluation Office of UNDP (IEO), shared insights on the use and impact of AIDA during a one-day briefing session hosted by IOE, on 20 October 2022.

Representatives from the evaluation offices of the three Rome-based UN agencies joined the hybrid session, in-person and virtually. Mr Garcia, formerly IOE Director, explained that a powerful set of machine-learning and artificial intelligence (AI) algorithms have created a tool at the cutting edge of technology for development. What is truly ground-breaking about AIDA is its capacity to intelligently search and make sense of unstructured data from over 6,000 UNDP evaluation reports. AIDA can analyse this wealth of information right down to the paragraph level, pulling out precisely what is needed from vast swathes of information in seconds.

Today, AIDA is helping evaluators expand their evidence base by searching existing data in a smarter, more efficient and innovative way. By virtue of its granular search capacity, AIDA finds pieces of evidence that would not be found using conventional desk research. Thus, the platform enables evaluators to manage and analyse data more accurately, faster and smartly.

AIDA has already proven successful in facilitating both inductive and deductive analysis approaches. Through the inductive approach, the system has facilitated the identification of emerging insights from an evidence base.

Through the deductive approach, the system has facilitated the identification of evidence supporting previous statements.

In addition to discussing the present applications and potential future ramifications of AIDA, Mr Garcia also addressed other salient areas of work in which IEO is currently engaging, including the 7th National Evaluation Capacities series conference (NEC) and the UN Evaluation Group (UNEG). The IEO Director explained that the NEC conference would focus upon ‘Resilient National Evaluation Systems for Sustainable Development’, following the upheaval and socio-economic fragility brought by the global pandemic.

With regard to UNEG, meeting participants discussed key areas of work vis-à-vis the Sustainable Development Goals, including the need to synthesize existing evaluative data on what has worked and not, why and where; the need to provide insights and analysis towards the key barriers and enablers to accelerate SDG achievements; the need to identify evidence gaps for future evaluations; and the need to provide policy recommendations to achieve the goals in the last five years before 2030.

Access the Artificial Intelligence for Development Analytics (AIDA) platform [here]

Oscar Garcia Dr Indran A. Naidoo IEO Director IOE Director

IMPACT EVALUATIONS: A DISCUSSION ON PAST, PRESENT AND FUTURE (WITH A SINCERE PLEA)

n 31 October 2022, Fabrizio Felloni, IOE Deputy Director, was among the speakers of the event titled ‘Impact Evaluations: Lessons learnt from IEU’s Learning-Oriented Real-Time Impact Assessment (LORTA) programme and other international organizations’, organized by the Independent Evaluation Unit (IEU) of the Green Climate Fund (GCF). Since 2018, the IEU’s LORTA programme has worked to strengthen the capacity of GCF-accredited entities and implementing partners to assess the impact of GCF-financed projects, through the use of impact evaluations (IE).

During the event, Mr Felloni shared the experiences matured by IOE in the conduct of impact evaluations. Between 2012 and 2019, the Office carried out IEs in Sri Lanka, India, Mozambique, Georgia, Kenya, Niger and Ethiopia. With the onset of the COVID-19 pandemic, these types of missions were brought to a halt to avoid the risk of exposure for people in rural area, for government representatives and IOE staff. For the moment, IOE is keeping these evaluations on hold, as the COVID-19 crisis is not over. Moreover, the Office currently faces strong requests from IFAD governing bodies and Management for evaluations at a more strategic level.

IEs were heavy exercises, as IOE was assessing a project against the full set of evaluation criteria, impact being only one of them. The Office would run large surveys – with anywhere between 1,500 up to almost 9,000 households –, in addition to assessing other criteria in a more qualitative manner, including through expert field reviews. IOE may come back to impact evaluations in the future, but more selectively.

After the IEU event, Independent Magazine caught-up with IOE’s Deputy Director for a quick chat on the past, present and possible future of IEs.

Good afternoon, Fabrizio.

Good afternoon, Alexander.

What purpose(s) would you say that an impact evaluation can or should serve?

I would argue that an impact evaluation needs to serve both accountability and learning, and would advocate towards de-emphasizing this dichotomy, which does not serve well the cause of evaluations. An IE should tell us what did and did not work, for whom and why. In order to do this, in addition to having statistical and data collection skills, we need to have the ability to understand the local context and the project history.

Fabrizio Felloni, Deputy Director, IOE

What projects are a better fit for impact evaluations?

Ideally, projects that have few or only one main intervention that is implemented in a homogenous manner. A typical example is in the health sector, as in the case of medical treatment or a vaccine: it is provided or inoculated always in the same dosage, always following the same practice (upper arm, for example) to everyone. Conversely, rural development projects à la IFAD are a challenging case. We have many components and sub-components, such as irrigation, training, rural finance, rural roads and rural enterprises, all of which are not provided ‘with the same dosage’ and in the same way everywhere and to everyone. Some people may receive only one component, while others may receive three and so on. There are ways, statistically, to deal with this but respondents may not indicate correctly which components they have received and we often do not have detailed records. So, are we evaluating a whole project or only a part of it, maybe one intervention package? There are advantages and disadvantages to both approaches but, either way, you can see that the level of analysis is not 100% clear from the outset.

Another problem we find in rural development projects is the spillover effect to non-targeted populations. This is especially true when interventions can be reproduced at low cost. And last but not least, we must keep in mind that our projects do not have reliable baseline data. We can work around this limitation, for instance, by controlling for sampling bias and using propensity score matching. These are reasonable palliatives, but they do not fully solve the problem of lack of baseline data.

What is the future of impact evaluation within IOE?

IOE has moved away from systematically carrying out one IE per year. Going forward, we will conduct IEs when we need to enhance the evidence for a future strategic evaluation, and when we see opportunities for methodological innovation.

What do you see as the future of impact evaluation within the broader evaluation community?

My advice to the evaluation community is simple. Impact evaluations are important. However, impact, alone, will not give you a complete indication of the worth and value of a development intervention. We need to avoid

the risk of concentrating on short-term effects. If an organization works with direct transfer of resources to households – such as productive assets, access to training, technical advice, subsidized productive inputs and infrastructure – it is not that difficult to observe ‘impact’, especially if by that you mean some short-term change. In reality, however, you may be observing an output rather than an impact. Think of a scheme based on ‘passing the gift’ of small livestock: what the survey will tell you is that people now own more small ruminants, which is simply what the projects have given to them. It may seem trivial but there are findings of this type, even in peer-reviewed journals. Our development goals are not short-term small gains. If changes are short-lived, we may not attain the SDGs. By the way, the “S” in the SDG acronym stands for the adjective Sustainable…. We need household- or community-level changes, married with stronger institutions at the community, local and central levels. In this context, impact is fundamental but impact assessment in isolation may not provide the full picture. Just by looking at the impact ratings from our project level evaluations and completion report validations, 81% of projects have positive scores over the past ten years. However, 60% of projects have a positive rating for both impact and sustainability. While most projects seem to have convincing evidence on impact, some have sustainability issues, which is something that IFAD Management recognizes and has committed to act upon.

FURTHER READING Evaluation through participatory narratives

Any final thoughts?

Perhaps some thoughts for the evaluation community: if you or your organization are conducting impact evaluations, please continue doing so, do not stop. However, do not lose sight of other criteria, such as efficiency and sustainability. We need comprehensive evaluation frameworks. We owe this to the hundreds of millions of people who live in poverty and whom we want to move out of poverty in the next years and decades – not just for a few months.

Thank you, Fabrizio.

You are welcome, Alexander

Fabrizio Felloni, IOE Deputy Director, recently co-authored an article published in the Evaluation journal. The study explores the use of Participatory Narrative Inquiry (PNI) in evaluating development interventions by contributing to the debate of using participatory narrative methods. Stories on personal experience are used to evaluate the project’s effects with similar methods such as Most Significant Change and Sensemaking. To the authors’ knowledge, this is one of the early applications of PNI to the evaluation of international development programmes. The study discusses advantages and limitations, and provides a scholarly reflection based on an application of PNI in the evaluation of gender and women’s empowerment in Niger. The study concludes that PNI is a powerful alternative to existing qualitative and participatory narrative evaluation methods.

Within mixed-method approaches, PNI allows for greater inclusion of project beneficiaries in the evaluation process, while helping to elaborate a thorough theory of change, understand the complexity of the context, identify and assess outcome pathways, and provide an evidence-based evaluation.

Zucchini, E., M.Carbon, C.Bosch and F.Felloni (2022): “Evaluation through narratives: A practical case of Participatory Narrative Inquiry in women empowerment evaluation in Niger” Evaluation 1– 20, Sage Publisher [here]

IOE-SUPPORTED AWARD

IN SUPPORT OF TRANSFORMATION

The evaluation report titled ‘Transformative Change in Tropical Forest Landscape Initiatives’ won the 2022 IDEAS Evaluation for Transformational Change Award, under the category ‘Evaluation in support of transformation’. The Award was supported by IOE, the Independent Evaluation GroupWorld Bank and the International Development Evaluation Association (IDEAS), and was assigned by an external panel of international evaluation specialists.

The IDEAS Award recognizes the growing understanding of the need to transform our societies, our economies, and our relationship with nature to be able to achieve the aspirations expressed in the 2030 Agenda for Sustainable Development and the Paris

Agreement. To this end, the competition aims to enhance awareness and understanding among different stakeholder groups of the value of evaluation in generating knowledge for learning and accountability in a domain of great importance and urgency.

The winning evaluation report explores the transformative change within a specific donor programme (P4F), providing a pragmatic framework to support learning to inform adaptive management through a series of learning loops. The report delves into how transformative change (i.e. change that is systemic in nature) is anticipated to occur due to programme interventions and draws on evaluative learning study findings (2019 to 2021) using a Transformative Change Framework. It

CONFERRED TO EVALUATION TRANSFORMATION

explores what transformative change is and how to assess it, providing a methodology and empirical insights.

Fabrizio Felloni, Deputy Director of the Independent Office of Evaluation of IFAD (IOE), served as a panellist during the special event organized to confer the Award. Prior to that, he attended the deliberations of the 2022 IDEAS Conference and Global Assembly, organized under the theme ‘Power of Evaluation for decision-making in a transforming world’. The event was held in hybrid modality with selected delegates meeting in Bonn, Germany, and other participants attending online, from 27 to 29 September 2022. Conference themes included: ‘power of evaluation in a transforming world’; ‘partnerships for sustainable devel-

opment’; ‘the future of evaluation: a perspective from young and emerging evaluators’; ‘evaluation within national ecosystem’; and ‘gender and equity responsive evaluation for better evidence and better policies’. The event was organized in partnership with the German Institute for Development Evaluation.

The International Development Evaluation Association was inaugurated in September 2002. IDEAS is the only global professional evaluation association which focuses on international sustainable development. It is deeply involved in strengthening and promoting the profession, fostering capacity development, and improving and advancing evaluation theories and practice, methods and use of evidence.

Academic outreach opportunities explored at Seoul National University

IOE recently explored opportunities for academic outreach at the Seoul National University, in South Korea. This, through a seminar held by Mikal Khan, IOE Evaluation Officer, at the Center for International Development Evaluation (CIDE) of the University’s Graduate School of Public Administration, on 16 December 2022.

The title of the seminar was ‘Introduction to the Independent Office of Evaluation of the International Fund for Agricultural Development: Governance and Evaluation methods. In his presentation, Mr Khan outlined IOE’s guiding policy and principles, the safeguards of the Office’s independence, its objective and evaluation products. The event also addressed IOE’s methodologies for evaluation, with a focus on IOE’s new Evaluation Manual, the Office’s most used investigative methods and data collection tools.

The event also allowed for discussions on interesting evaluation methods that CIDE is pursuing and applying. CIDE strives to improve the evaluation capacity of national and international organizations as

well as of recipient countries through various education and training programmes. At the same time, CIDE aims to provide practical assistance to national and international organizations by scientifically measuring the impact of the development programmes, and provide evaluation feedback that can be reflected in for the future programmes.

CIDE is a research centre specialized in international development sector evaluation launched at Seoul National University in March 2014. The centre’s mission is to make contributions in improving the development programmes of recipient countries and donor agencies by assessing aid effectiveness with rigorous evaluation methods in accord with international standards.

Mikal Khan, Evaluation Officer, IOE

Profile

FOCUSING ON PEOPLE FIRST (NOT COMMODITIES)

What is unique to IFAD’s targeting strategy compared to other international financial institutions? This was one of the questions that members of IOE’s Country Strategy and Programme Evaluation (CSPE) team addressed during their second half-day mini-retreat, on 12 October 2022. The answer is that IFAD targets the poorest and most vulnerable people first and enable them benefiting of IFAD’s support secondly, as opposed to focusing on commodities, explained Jeanette Cooke, IOE Evaluation Analyst.

Building on the success of the first mini-retreat, on 8 June 2022, the second CSPE gathering put various salient topics on the discussion table. Indran A. Naidoo, IOE Director, set the context, framing IOE’s work around the ‘independence’,

‘credibility’ and ‘utility’ pillars, while Johanna Pennarz, Lead Evaluation Officer, presented her experiences vis-à-vis the key points to check when assessing the government’s performance during CSPEs.

How to enhance logistic support to CSPEs in demanding contexts was the issue that Daniela Asprella, Evaluation Assistant, talked about head-on. During this session, participants looked at examples of (real life) worst-case scenarios, with seemingly unsurmountable last-minute security hinderances thrown into the mix. Advice on how to ship satellite phones to remote locations at the eleventh hour aside, the half-day event proved fun, dynamic and thoroughly enjoyable – once again. A best practice in the making, no doubt.