Independent Magazine - Issue n.4, 2022

Welcome to the fourth edition of Independent Magazine. This edition focuses on evaluation as a reflective activity. Evaluation is often met with resistance, pushback and sometimes excitement, depending on who the evaluand is and what the subject of evaluation is. We see in the world today an increased focus and attention on the element of oversight and independence. At the political level, we’ve seen courts more active than ever in having to come in and adjudicate decisions of politicians in the democratic process. This is an evaluative activity because it is about evidence being used, being presented and decisions being made. The key question is “whose evidence and for what purpose?”. We also see great conversation and discourse about what constitutes objectivity, what constitutes truth. To a marked extent, social media has re-defined this. With the huge amount of information that is coming in, it is very difficult to distinguish what is true and what is noise.

IFAD makes huge investments from voluntary contributions from countries against its mandate in order to have impact on the ground. The purpose of IOE is to charter process through its policy and through its procedures that ensure that every evaluation leads to multiple levels of conver sation and consideration by the stakeholders. We can say quite confidently that, from the time a country is selected for a Country Strategy Programme Evaluation, there is a great amount of consultation with the government, with IFAD and with all stakeholders. Over a period that may span up to twelve months, people get an opportunity to reflect on the performance based on the penetrating questions that we ask. That is the process of reflection. But reflection is not just reflection on its own. Reflection is a way to improve the quality of what comes out. Evaluators try to ensure that each level of feedback is adequately captured, that they have the diversity of opin ions and views. Through this iterative process you have a very good and well-constructed under standing of how something works and, if it doesn’t work, what are the reasons for it not working. This provides us the basis for preparing findings, as well as a basis for making recommendations.

Evaluation is not just a reflective process at the level of the countries, where we spend most of our time, it is also reflective at the level of the IFAD Board, which is the legal forum where mem ber states make decisions about where the funding should go. IOE provides the information for this reflection, as does Management and the evaluand, i.e., the governments.

In my own words

The reason for independence is a critical one. You want a unit, within an organization, that is able to conduct its work without fear, favour or prejudice, that is not under any political or administrative pressure. The new evaluation policy that was adopted in 2021 entrenched earlier policies, and safe guarded the independence, credibility and utility of the function. The report line of the IOE Director, directly to the Board, and the term limitation prevents the various levels of conflicts of interest.

This edition wishes to articulate the products and demonstrate where the reflection takes place. If we were to accept that, as international civil servants, we are here to serve a higher good, then we should also be able to accept that we can learn. The learning comes from receiving frank and direct feedback. Our job is to provide this feedback. We want to make sure that the product received by the beneficiaries on the ground is the best they could get.

We need to be quite clear: evaluative work is political. There is always pressure. The evaluands may not always appreciate the perspectives coming in. But we also need to be clear that the only work that IOE does is evaluation, and as evaluation professionals we spend time thinking and ensuring that the highest methodological standards are upheld. In this context, I set-up the Evaluation Advi sory Panel (EAP) last year. IOE has just had the first EAP annual meeting. The Panel has confirmed that we are on the right path, and that it is important to engage more.

IOE focuses on both learning and accountability. It’s not just about saying what is right or wrong. We are dealing with social data, where it is sometimes not easy to cast a judgement, and often times we are dealing with a motion picture, not a simple snapshot. It’s for this reason that CSPEs cover long periods, even up over ten years. What is clear is that it is possible, within the rubric of a fully independent office, to engage in learning if we consider that as a reflective activity and we also take into account that both evaluators and evaluands can learn and deepen their understanding through a very interactive process.

This edition tries to bring together these various strands, and indicate the importance of the conver sation that evaluation brings.

I trust you will find this edition inspiring.

Dr. Indran A. Naidoo Director

Independent Office of Evaluation of IFAD

PILLARS

INDEPENDENT OFFICE

GUIDING PRINCIPLES

QUALITY ASSURANCE

HISTORY

EVALUATION OF IFAD (IOE)

IOE EVALUATION PRODUCTS

[click on icons to access database]

Aggregation level Aggregation level

Sub regional Evaluations Corporate Level Evaluations

Primary objective

Assess strategy, common intervention approaches and IFAD organizational set up in a set of countries that share salient characteristics.

Main users

Regional and country director(s), technical advisors, operational staff, and government counterparts.

Evaluation Synthesis Thematic Evaluations

Primary objective Contribute to knowledge generation by consolidating findings from past evaluations.

Main users

Senior management, Directors, staff of regional and technical divisions, and members of governing bodies.

Primary objective

Provide evidence of development effectiveness, performance and results of operations in a thematic topic.

Main users

Senior management, Directors, staff of regional and technical divisions, and members of governing bodies.

Primary objective Assess the organizational performance and institutional effectiveness of IFAD.

Main users

Senior management, Directors, staff of regional and technical divisions, and members of governing bodies.

Annual Report of Independent Evaluation

Primary objective Report all of IOE’s evaluation activities in a given year, and presents a synthesis of IFAD’s performance, lessons and challenges.

Main users

Senior management, Directors, staff of regional and technical divisions, and members of governing bodies.

Aggregation level

Project Completion Report Validations

Primary objective

Validate the project completion reports prepared by IFAD Management.

Main users

IOE and IFAD Management for reporting (ARIE and RIDE) and feedback.

Project Cluster Evaluations Project Performance Evaluation

Primary objective

Assess the performance and results of project level operations funded by IFAD.

Main users

Regional and country director(s), technical advisors, operational staff, and government counterparts.

Aggregation level

Impact Evaluation Country Strategy and Programme Evaluation

Primary objective

Provide a rigorous quantitative assessment of the impact on rural poverty of selected IFAD’s operations.

Main users

Regional and country director(s), technical advisors, operational staff, and government counterparts.

Primary objective

Assess the experience of several projects that have a common theme or common major component.

Main users

Regional and country director(s), technical advisors, operational staff, and government counterparts.

Primary objective

Assess performance and results of country strategy and operations and provide lessons and recommendations to guide preparation of next country strategy

Main users

Divisional and country director, country team, and government.

INDEPENDENCE

Guarantees the avoidance of conflicts of interest by upholding the provisions that safeguard the behavioral, organizational and structural independence of the Office.

EVIDENCE BASED MANAGEMENT

Reflects IFAD’s increasing focus on embracing a culture of evidence based management to maximize development effectiveness, in which evaluation has a critical role to play

LEARNING & ACCOUNTABILITY

Clarifies that accountability and learning are objectives of evaluation, and emphasizes the effective use of evaluation products to this end

EVALUATION COMMITTEE

Presents the roles and responsibilities of the Evaluation Committee and, of its Chair, in the evaluation function on behalf of the Executive Board

Independent Magazine brings to the forefront of the global development dialogue the major efforts undertaken by the Independent Office of Evaluation of IFAD, while seeking to advance the organization’s vision of vibrant, inclusive and sustainable rural economies, where people live free from poverty and hunger. To present the richness of rural life, and detail facets of local community lifestyle, Independent Magazine also zooms in on cultural activities and landmark occurrences in countries featured by IOE’s evaluations.

Dr. Alexander Voccia

Laure Vidaud

Ryan

INDEPENDENCE REMAINS UNMOVABLE BEDROCK OF EVALUATION FUNCTION

Maintaining and safeguarding the inde pendence construct that permeates the functioning of the Independent Office of Evaluation of IFAD (IOE) is essential to ensure the continued production of credible, high-quality and useful evaluations. The newly appointed Chair of the IFAD Evaluation Com mittee (EC), H.E. Miguel Jorge García Winder, Ambassador and Permanent Representative from Mexico to the UN Agencies, emphasized this crucial node during his meeting with the IOE senior management team, on 6 October 2022.

H.E. Miguel Jorge García Winder was appointed Chair of the IFAD EC at the Committee’s 118th session, on 2 September 2022. Former Under secretary of Agriculture of Mexico, Ambassador Winder has led and coordinated hemispheric

and global initiatives aimed at the develop ment of competitive, sustainable and inclusive agriculture, with emphasis on sustainable and innovative agribusiness models for small and medium producers and for family farming, in the development of new market organizations and in strengthening entrepreneurial skills. In his work, he has paid particular attention to developing sustainable ways of linking small farmers to markets.

During the course of the meeting with IOE senior management, which took place at IFAD Headquarters in Rome, Ambassador Winder put the spotlight on key strategic issues for the EC. In addition to emphasizing the need to uphold the principle of independence enshrined in the IFAD Evaluation Policy [here], he also underscored the importance of maintaining

focused evaluative conversations, avoiding an over proliferation of issues, and highlighted the urgency to identify constructive ways in which IFAD member states can actively support the follow-up process to the recommendations contained in IOE’s evaluation reports.

The strategic innovations brought about by IOE over the past 18 months were also discussed during the meeting, alongside the Office’s work programme and key deliverables for the forthcoming year. In this context, Dr Indran A. Nai doo, IOE Director, explained the significance of the new Evaluation Policy, Multi-year Strat egy [W – a first for IOE – and revised Manual [here], whilst pinpointing the renewed and en hanced focus on quality control, and affirming the heightened collaborative relationship with IFAD senior management. IOE’s re-envisioned approach to communications and outreach –manifest in its new website [here] and maga zine [here] – complement these undertakings.

Fabrizio Felloni, IOE Deputy Director, pre sented the pillars underpinning the IOE work programme for 2023. These include forging a corporate culture vis-à-vis the need for IFAD to be an increasingly transparent, accountable and learning-oriented organization; improving evaluation coverage; building evaluation dialogues; and retaining and deepening IOE’s

position as an internationally recognized lead er in the evaluation of rural development pro grammes, strategies and policies.

The IFAD Evaluation Committee performs indepth reviews of selected evaluation issues. Meeting at least four times a year, the EC re views IOE’s strategies and methodologies, dis cusses selected evaluation reports and IOE’s annual work programme and budget. It also makes suggestions for including evaluations of particular interest to the Committee in IOE’s annual work programme.

IOE WORK PROGRAMME

Reviews the IOE work programme and budget and makes recommendations to the Executive Board

IOE INDEPENDENCY

Ensures that independent evaluations are shielded from external influences

RECOMMENDATIONS

Makes recommendations to the Executive Board and oversees implementation of evaluation recommendations and agreed follow up by IFAD Management.

IOE DIRECTOR APPRAISAL

The Chair of the Evaluation Committee is also responsible for the annual performance appraisal of the Director, IOE.

IOE DIRECTOR APPOINTMENT

Reviews the report of the search panel for the selection and appointment of the Director, IOE.

DOCUMENT REVIEW

Reviews selected evaluation documents to ensure learning and accountability from independent and self evaluations.

Role and functions of the Evaluation Committee and its Chair

IFAD to include environmental restorative solutions in climate change adaptation responses

Profile

Climate change adap tation (CCA) interven tions must do no harm to the environment. However, across the world, this is not always the case, observes the soon-to-be-published report by the Independent Office of Evaluation of IFAD (IOE), titled ‘Thematic Evaluation of IFAD’s Support for Small holder Farmers’ Adaptation to Climate Change’. High-lev el representatives from the IFAD Executive Board (EB) joined the Fund’s senior man agement to address this and other related issues head-on, during a public virtual learn ing event on 19 May 2022.

“The Thematic Evaluation found that there is a need to develop climate adaptation

solutions that simultaneously promote not only climate re silience, but also environmental and economic resilience. Where that is not feasible, we should aim to ‘do-no-harm’ or better, with the necessary offsets to compensate for the damage that is being done to the ecosystems”, explained Suppiramaniam Nanthike san, Lead Evaluation Officer at IOE, and lead author of the CCA Thematic Evaluation.

IOE’s Thematic Evaluation (TE) found that IFAD is al ready implementing CCA projects that do no net harm to the environment. These successful interventions are landscape-scale, integrated interventions providing natu ral solutions to underlying cli mate threats, and they involve strong engagement with ben eficiaries and stakeholders during design and implemen tation. These offer important lessons to help IFAD expand its CCA guidance to include restorative solutions in order to fulfil its commitment to go beyond doing no harm and restore the environment.

“Climate change is here to stay, that is why it’s such an important topic. IFAD was an early mover on CCA smallscale agriculture through the

adaptation for smallholder ag riculture programme that has had some really good results, which is why we want to scale it up and deepen in. However, although we’ve always had ambitions in the areas of cli mate adaptation, it’s fair to say that IFAD’s capabilities still need to grow in a number of areas, as the report brings out, presenting us with spe cific recommendations in this regard”, stated Donal Brown, IFAD Associate Vice-President, Programme Management Department.

Hosted jointly by IOE and IFAD management, the online virtual workshop was a fully public event that brought together a wealth of high-lev el attendants. Over 80 par

Donal Brown Associate Vice Predisdent Programme Management

Dr. S. Nanthikesan Lead author Thematic Evaluation

ticipants joined the event, in cluding representatives of the EB and government counter parts, in addition to members of civil society organizations, development partners, aca demia, private sector, IFAD senior management and staff, and the general public.

The event aimed to promote broad ownership and use of the evaluation report, and to provide a full understanding of findings, key messages and recommendations. It involved a brief presentation of the evaluation followed by a question and answer session. The sec ond part of the event featured a discussion panel around the following three themes: conceptualizing climate re silience in projects; ensuring environmental sustainability of climate solutions; and promoting scaling-up and knowledge management related to climate change adaptation response. The panel members comprised of IFAD managers, as well as technical special ists involved with successful climate projects.

Discussants recognized that IFAD’s experience in working with marginalized communi ties in the rural agricultural sector, often facing adverse climatic and environmental conditions, has positioned it well to address the accelerat ing risks from climate change and to place CCA as a strategic institutional priority. As it learns from experience, IFAD’s approach to CCA is evolving and progressing in the right direction. Over the past dec ade, IFAD has developed and updated its climate strategy and continues to improve the

institutional environment for CCA responses. Furthermore, IFAD has assessed climate risks in all its country strate gies and operations and in tegrated climate response in every all intervention.

“IFAD has really established itself as a major player in cli mate adaptation. We are com mitted to ensuring that 40% of our core resources are dedi cated to climate finance, with more than 90% of this going to adaptation. This is really a break-away from most of the other agencies that work in climate, because they spend a much larger proportion on mitigation. The key areas that we are focusing on are innova tion, scaling-up and measure ment”, affirmed Jyotsna Puri, IFAD Associate Vice-Pres ident, Strategy and Knowledge Department.

To better understand its re sults, IFAD’s climate efforts need a clear conceptual framework to assess actu al improvements to climate resilience of smallholders and corporate guidance to strengthen smallholders’ climate resilience together with environmental and socio-eco nomic resilience. In addition, IFAD’s analysis highlights significant gaps in technical capacity at headquarters and project level to mainstream and monitor CCA responses.

Against this backdrop, workshop participants recalled the recommendations of the CCA TE. These include the need to update the IFAD Strategy and Action Plan on Environ ment and Climate Change 2019–2025, to comprehen

sively address bottlenecks to CCA performance; the impor tance of systematically prior itizing scaling-up and other non-lending activities with dedicated resources, to suc cessfully strengthen smallholder climate resilience; and the need to ensure sustained organizational learning from operational experience to im prove current and future CCA performance.

“In addition to the conclu sions and recommendations, the report provides a wealth of evidence across a range of issues. This includes an over view of the historic evolution of climate responses in IFAD, a comparison of related poli cies and approaches in select IFIs and UN agencies, an anal ysis lessons from all available scientific literature on what works, to name a few. In addi tion, drawing from the experi ences in 20 countries, the an nexes provide rich information on the elements that make cli mate responses more relevant, effective and efficient”, noted Indran A. Naidoo, Director of IOE. The impetus for this evalua tion came in 2019, when the EB approved it as part of the work plan of IOE. The evalu ation assessed the extent to which IFAD efforts have pro moted climate resilient live lihoods for smallholders and improved their food security and covered the period since CCA was declared as a cor porate priority in 2010. The scope of the evaluation was comprehensive and included all related IFAD operations and country strategies.

Expanding evaluation processes to increase engagement at the heart of IOE’s work

“Afully independent office that is well-es tablished has the foundations and the security that allow it to broaden its field of engagement – doing so will not threaten its raison d’être. On the contrary, by meaning fully engaging, evaluation offices will improve quality and data, whilst also ensuring long-term ownership of their evaluations”. This, accord ing to Indran A. Naidoo, Director of the Inde pendent Office of Evaluation of IFAD (IOE), who was a session chair and presenting panellist during the Spring meeting of the Evaluation Cooperation Group (ECG), on 9-10 June 2022.

Hosted in Washington DC, the ECG meeting was the first to be held in-person since the outbreak of the COVID-19 pandemic, and tru ly brought together a parterre de rois of mul tilateral development evaluation experts. Sen ior representatives of evaluation offices from all major multilateral development banks (MDBs) and other partners joined Dr Naidoo in delivering presentations and participat ing in discussions during the Spring meeting.

The IOE Director actively contributed to the event in several ways by engaging in different sessions at multiple levels, including as a chair, presenter and discussant. Dr Naidoo chaired the session focused on reflections from evaluating the COVID-19 response and implications for other shocks and crisis. The key joint session of ECG Heads and multilateral working group on ‘Managing for Development Results’ also saw featured a strong IFAD presence, with Dr Nai doo being joined by Nigel Brett, Director of the Operational Policy and Results Division at IFAD. Mr Brett represents the Fund on the working

group, and made inputs during the joint session. The session was chaired by Lisandro Martin, head of the Results division in the World Bank.

The importance of finding ways to expand the evaluation processes to increase engagement, and what this means in practice, was at the centre of the presentation that Dr. Naidoo deliv ered in the context of the session theme titled ‘Innovations in Learning and Accountability’, in which the IOE Director was a panellist. Moving from the premise that organizations that develop initiatives to change mindsets are two times more likely to succeed, Dr. Naidoo explained the importance of customizing brain and neu ral science to the field of evaluation in order to prompt behavioural change by evaluators, in creasing levels of reciprocity with the evaluand, and ultimately taking the debate to a higher level.

“Evaluation methodology is not just the choice of which tools to use, it is also the processes that are put in place in order to engage with the evaluand. Over the years, office transform ative work was informed by insights from the research of Dr. Srini Pillay who brought to the fore ways to merge imperative and judgment imperatives with engaging more effectively with evaluands, who are influenced by their own psychological make up. Evaluators need to communicate more effectively and understand

from traditional static approaches that foresee linear learning sequencing processes, in fa vour of dynamic engagement centred around a three-way relationship between independence, credibility and utility. This approach is exempli fied by IOE’s recent efforts and successes in ex panding engagement through the design and launch of several new and revised products and initiatives. These include the new Evaluation Manual, Independent Magazine, IOE-led learning events, IOE’s first independent website, and an upscaled evaluation report product mix.

“Evaluation supports accountability and learn ing to improve transparency and accountabili ty. However, the area that can benefit from user perspectives is that of learning. In practice, the construct of all evaluation needs to accept en gagement as a part of process credibility, and seen as necessary for generating ongoing re flection as part of the learning process to build understanding”, summarized Dr. Naidoo in ex plaining IOE’s engagement-centred approach.

Contributions to the session included presenta tions by Alison Evans, Director General Evalua tion of the World Bank Independent Evaluation Group, who addressed reforms to the World Bank’s Management Actions Record for more meaningful accountability and learning; Emmanuel Jimenez, Director General of Independent Evaluation, Asian Development Bank, on rec ommendations verses ‘issues to be addressed’.

Dr. Naidoo also updated ECG members of key changes and innovations at IOE, including the new Evaluation Manual, new Multi-Year Evaluation Strategy, new Evaluation Policy, new independent website, new Independent Magazine, and enhanced engagement with the Global Evaluation Initiative, includ ing vis-à-vis joint work towards the seventh National Evaluation Capacities conference.

the psychological dimensions, which is what the field of neural-psychology brings. A pres entation made at IOE in its Coffee Talk series debunked five evaluation myths and provided insights on how to engage more constructive ly in the sensitive area of presenting evaluative findings”, Dr. Naidoo explained in this regard.

The IOE Director put the spotlight on the concept of ‘principled engagement’, which moves away

The ECG was established in 1996 to promote a more harmonized approach to evalua tion methodology. Members work together on joint and meta-evaluations and discuss such evaluation issues as the independence of evaluation offices in MDBs, evaluabili ty assessment, the results agenda of MDBs, and lessons learning and dissemination.

IOE enriches evaluation debate at gLOCAL 2022

Sensemaking’. On 31 May, Kouessi Maximin Kodjo, IOE Lead Evaluation Officer, mod erated a panel discussion during the event titled ‘Inno vate in monitoring and evalu ation to better meet the needs of decision-makers at the na tional level’.

How to adapt internation al standards, practices, and evaluation criteria in the context of rural devel opment; how to strengthen the complementarity between monitoring and evaluation for decision-making; and how to use Sensemaking and Partici patory Narrative Inquiry (PNI) in evaluation: these were the discussions that the Inde pendent Office of Evaluation of IFAD (IOE) contributed to during gLOCAL Evaluation Week 2022.

This year, IOE and its staff members were involved in three gLOCAL sessions. On 2 June, IOE hosted the event ti tled ‘A new tool for rural devel opment evaluation practition ers – the 2022 IFAD Evaluation Manual’. On 1 June, Fabrizio Felloni, IOE Deputy Director, was among the speakers during the event titled ‘Using stories in evaluation: Partici patory Narrative Inquiry and

IFAD’s Revised Evaluation Manual was presented on 2 June, as over sixty repre sentatives of international organizations, governments of developing countries and evaluation practitioners came together to share insights on translating key 2030 Agenda concepts into evaluation practice, adapting evalua tion criteria and standards to rural poverty alleviation, and addressing the needs of and opportunities for rural development and evaluation practitioners and institutional stakeholders.

“While the Manual has been conceived for IFAD staff and consultants, I am confident that it will be of interest to many rural development prac titioners and evaluators out side IFAD, who may be work ing in a government, in an international organization, in an NGO, civil society organ izations or in a private sector company”, noted Indran A. Naidoo, IOE Director.

The Manual draws from re cent contributions on the the

ory and practice of evaluation, with special reference to the 2030 Agenda for Sustainable Development, and provides guidance on how to adapt international standards, prac tices, and evaluation criteria in the context of rural devel opment, particularly when the end-clients of development interventions are smallholder farmers.

Please access Part I and Part II of the 2022 IFAD Evaluation Manual, together with the online training course and all related multimedia resources [here]

On 1 June, the use of Sense making and Participatory Nar rative Inquiry (PNI) took cen tre stage, as participants were invited to ask questions and to share their own experiences with using stories in eval uation. Because a large num ber of stories are collected and self-interpreted, it is pos

Fabrizio Felloni, Deputy Director, IOE

Profile

Dr. Kouessi Maximin Kodjo Lead Evaluation Officer, IOE

Profile

sible to conduct quantitative analysis of recurrent themes, perspectives and feelings emerging in the narratives. By combining elements of qualitative and quantitative research, these approaches help to make sense of com plex and evolving realities.

Moving from the premise that there is often a disconnect between decision making and M&E systems, the event held on 31 May explored innova tive approaches and changes that can attract and sustain the interest of more stake holders for M&E at country level, to adapt to the needs of decision makers in terms of time, format and modalities of communication. The event’s panel, composed of high-lev el representatives with roles at the interface between insti tutions in charge of M&E and government decision-makers, drew on direct experience from Benin, Morocco, Tunisia and Senegal. Points present ed by panellists and ques tions from participants show that, although each country context is specific, a com monality is the strong buy-in at a high governmental level that enables the adoption of innovative solutions or approaches for effective nation al M&E systems, which can provide timely evidence for decision-making.

GLOCAL Evaluation Week was launched in 2019. In the short time since gLOCAL was launched, organizing partners from around the world have hosted nearly 1000 M&E fo cused events across five con tinents in multiple languages.

Click to access

Profile

Government performance is a key enabler of IFAD-funded projects

The period 2010-2020 has seen a steep and continuous worsening of the performance of national governments in IFAD-supported operations. The latest Evaluations Synthe sis Report (ESR) of the Inde pendent Office of Evaluation of IFAD (IOE) reveals this neg ative trend, clarifying that the information currently availa ble through corporate moni toring and evaluation systems does not allow to explain why and how government performs in the context of IF AD-supported interventions. As a result, the reasons for this lagging government per formance are not well docu mented and understood, and there are significant knowl edge gaps with regard to the factors driving government performance. These and oth er related findings were dis cussed during the course of a public virtual learning event, on 3 June 2022.

“We all recognize that gov ernment is the key player in IFAD’s effectiveness. This means that we have to better understand how government performs, and what are the differences in performance in varying contexts. Government performance is not some thing that can be addressed through a top-down head quarters-based approach. This is something that needs to be addressed, first and fore most, at the country level. We need to emphasize the impor tance of having the right per sonnel on the ground, working in the local context and having a proper analysis of the insti tutional and political frame works”, stated Johanna Pennarz, Lead Evaluator, IOE.

Government is the key play er in IFAD’s development ef fectiveness. IFAD-support ed programmes are owned, managed and executed by governments and their agen cies in collaboration with other stakeholders. The data show, however, that gov ernment performance has been lagging for many years and that there are no signs of improvement. Between 2010-2020, the deteriorating government performance observed by the ESR can be linked to the increasing share of projects led by ministries of agriculture, which reflects IF AD’s closer focus on agricul tural and value chain projects.

At the same time, the perfor mance of local governments –the ‘traditional’ IFAD partners for local development pro jects – remained consistent, but their share in the overall portfolio decreased.

“This is a partnership and IFAD can only deliver so much of this partnership. Real suc cess happens when both sides of the partnership – IFAD and government – have full own ership and are fully involved in these projects. IFAD has only limited influence on some of the key challenges identi fied, such as ownership and in-country capacity. Despite this, we do all we can, and will continue to do so to support countries to build capacity and ensure the success of pro jects”, affirmed Donal Brown, Associate Vice President, Programme Management De partment, IFAD.

Organized and hosted by IOE, the on-line virtual workshop was a fully public event that drew interest from a broad va riety of attendants. Over one hundred participants joined the event, which was opened by Indran Naidoo, Director, IOE. This was followed by a presentation on the key find ings, lessons and conclusions from the ESR by Johanna Pennarz. Donal Brown provid ed the perspectives of IFAD Management on government performance. Thereafter, two

Dr. Johanna Pennarz Lead author, Evaluation Synethsis Report

discussion panels took place, on IFAD’s support on govern ment performance in fragile and conflict situations, and on IFAD’s approach to working in decentralized context. During the discussion, IFAD’s coun try programme staff shared context-specific perspectives on government performance.

evaluations as well as project performance evaluations, to gether with 46 project com pletion report validations and three impact evaluations cov ering 71 IOE-evaluated pro grammes or projects since 2010. The ESR offers a num ber of useful findings and lessons to enhance IFAD’s partnerships with government, and is complemented by a Learning Note on ‘Working in the context of govern ment decentraliza tion policies’, which was made available to participants prior to the event.

by instability, weak capacities and unfavourable policies, and institutional processes. The synthesis report identified a smaller number of countries that have shown consistently good performance, driven by strong government owner ship and leadership. For these countries, the institutional and policy contexts are very different.

The synthesis focused on the performance of governments in IFAD-supported opera tions. It covered the period 2010–2020, when government performance deteriorated. For this decade, performance data were available from 421 evaluations, including 57 country strategy and pro gramme evaluations and 364 project level evaluations. The synthesis selected 15 coun tries as case studies. These drew evidence from 38 coun try strategy and programme

“This synthesis takes a broad ap proach to review government per formance in the context of IFAD projects. It looks at government actions in terms of its insti tutional efficiency, prevailing enabling conditions, and the structures, capaci ties and processes involved. It identi fies the variables of govern ment performance, and the links between those variables. The focus is on the inner work ings of government action, to gether with the underlying dy namics and drivers”, explained Dr Naidoo.

Workshop participants recog nized that in most countries there were positive government performance drivers such as ownership, leadership and resources committed, but that these were often offset

On IFAD’s side, there were also positive and negative factors affecting government performance. On the positive side, there was good align ment with government prior ities. Long-term partnerships and continuous support – to gether with increasing coun try presence – have built sustained government trust and ownership over many years. On the negative side, workshop participants noted insufficient consideration of government capacities and institutional and policy frameworks, as well as lack of suita ble incentives to keep govern ment staff engaged.

Looking ahead, there is no panacea to reverse the trend at corporate level. IFAD has to build on its strength to iden tify and address drivers of government performance af ter careful analysis of institu tional and policy frameworks at country level. The organiza tion must become an enabling environment for country man agement by providing critical support for effective engage ment with government, such as technical advice, predictable resources and incentives for durable relationships.

IOE staff at the heart of international debate during European Evaluation Society conference

How to tackle the tradeoffs between continuity and renewal in large in ternational organizations’ re sults and evaluation systems and practices; how to ensure evaluations fully encompass unintended consequences of any intervention to the natural environment and to vulner able people; how to reflect the complexity of 21st century through the revision of evalu ation polices; and how to cre ate incentives for evaluators to adopt practices that can help organisations to achieve their transformative goals: these were the discussions that the Independent Office of Evaluation of IFAD (IOE) contributed to during the 14th European Evaluation Society (EES) biennial conference.

This year’s ESS biennial con ference saw the active in-

volvement of IOE staff mem bers in four events. On 8 June, Fabrizio Felloni, IOE Deputy Director, was a discussant in the session titled ‘Results and Evaluation Systems in Large International Organizations: Trade-offs between Continuity and Renewal’. The roundta ble’s objective was to identify barriers, opportunities, and solutions to implementing more transformative evalua tion practices in large organi zations. Discussions stemmed from the recognition that while the evaluation community is calling for evaluation to support the necessary trans formation to a more just and sustainable future, it is not easy for large international or ganizations to transform their results measurement and evaluation practices.

On 9 July, Dr S. Nanthikesan,

IOE Lead Evaluation Officer, was a panellist during the event titled ‘Mainstreaming the Environment into Eval uations: Perspectives from the UN’. He presented the pi oneering systemic approach pursued by IFAD to integrate environment considerations in all its evaluations. This, as participants recognized that evaluation must ensure that it focuses on whether inter ventions make a positive con tribution in the coupled hu man-natural systems. To do so, evaluation must expand its remit beyond the internal logic of interventions to as sess the relevance, impact and effectiveness and to fully encompass unintended con sequences of any intervention to the natural environment and to vulnerable people.

Later the same day, Fabrizio

Felloni participated as a dis cussant in the session titled ‘From Neutral Observers to Advocates, Truth Speakers, and Agents Provocateurs: What Role Should Evaluators Play? Reflections Around Evaluation Policies’. During the roundtable, representa tives of the evaluation offices of IFAD, FAO, WFP and CGIAR discussed the evolution of the evaluation function in re sponse to current global chal lenges; the role of organiza tional evaluation policies in facilitating interdisciplinary approaches through trans formative evaluations; the process of policy revision visà-vis the repositioning evalu ation functions on lines of ac tion that align to internal and external drivers; and the les

sons learnt from operational izing policies to maximize the use of evaluative evidence for decision-making.Ms. Johanna Pennarz, IOE Lead Evaluation Officer, co-coordinated the session on ‘A Transformative Change in Development Eval uation: o pportunities and Challenges for Changing Practices, Mindsets and Val ues in International Organisa tions’, which took place on 10 June. However, Ms. Pennarz could not participate in the session, and on her behalf Dr S. Nanthikesan participated as a discussant. During the session, participants reflect ed on ways to overcome the resistance to transformative evaluations and create incen tives for commissioners and evaluators to adopt evaluation

practices that can help organ isations to achieve their trans formative goals. In particular, panel members aimed to un cover positive approaches to initiate and sustain changes within their respective organisations.

The EES was initiated in 1992. The mandate of the EES is to stimulate, guide and promote the theory, practice and utili zation of evaluation in Europe and beyond. Held between 6 and 10 June 2022, in Copenhagen, the 14th EES biennial conference focused on insti tutional, identity, content and methodological shifts aimed at transforming evaluators, as well as evaluation systems and methodologies.

Fabrizio Felloni, Deputy Director, IOE

Dr. Johanna Pennarz, Lead Evaluation Officer, IOE

Dr. S. Nanthikesan Lead Evaluation Officer IOE

IOE advances international evaluation thought

through its membership to ECG,

New Evaluation Advisory Panel set to help IFAD make every investment count

“Ibelieve that the new Evaluation Advi sory Panel will play an important role in moving forward with the effective ness of the evaluation of our activities. At a time when we know that we are facing a looming crisis, with the shirking of fiscal space among our member states, we must make every investment count. This requires a rigorous evaluation function that can provide us with constructive critique and learning to ensure we continue to improve the quality of our delivery”. This, ac cording to Gilbert Houngbo, President of the In ternational Fund for Agricultural Development (IFAD), who delivered the opening address during the inaugural meeting of the Evaluation Advisory Panel (EAP) of the Independent Office of Evaluation of IFAD (IOE), on 12-14 July 2022.

Hosted in Rome, Italy, the hybrid meeting fea tured in-person sessions with active contri butions from participants joining on-line. The event included the high-level presence of members of the Evaluation Committee and the Executive Board of IFAD, in addition to the di rectors of the FAO, WFP and CGIAR evaluation offices, and a broad spectrum of IFAD senior management representatives.

“The establishment of the Advisory Panel is a milestone not only for IOE, but for IFAD as a whole, insofar as it will allow the organization to enhance its evaluation function by integrat ing external expertise into its existing independ ent and self-evaluation processes and products. We look forward to the Panel’s contribution to further promote development effectiveness in IFAD, which is more important than ever in such a critical moment”, reaffirmed Mei Hongyong, Deputy Permanent Representative of China to the UN Agencies for Food and Agriculture, ech oing Dr Houngbo’s opening remarks.

The five members of the EAP are Rob van den

Berg, Visiting Professor at King’s College Lon don; Donna Mertens, Emeritus Professor at Gallaudet University; Bagele Chilisa, Professor at the University of Botswana; Gonzalo Hernán dez Licona, Director of the Multidimensional Poverty Network at the University of Oxford; and Hans E. Lundgren, formerly Head of the Evaluation Unit at OECD.

During the course of the inaugural three-day event, the EAP reviewed IFAD’s corporate needs and demand for evaluative findings and recommendations, including by discuss ing topics, levels of analysis, decision-making processes, and the use of evaluation for ex-an te quality assurance on project and country programme design. The esteemed panel also provided feedback to IFAD and IOE on the evo lution of thinking in international development evaluation, gave advice on emerging priorities for evaluation, and participated in a session on the preparation of a Corporate-level Evaluation on Knowledge Management.

“IFAD is a very progressive organization be cause it emphasizes continuous improvement of methodologies, including data collection. This is essential, and points to the direction where we can have continuous improvement and help to ensure that evaluation findings are utilized. I come from the perspective of ‘no knowledge left behind’. This means that you have to be reflec tive all the time. For this to happen, you need to have an organizational structure that allows for bottom-up reflection. IFAD is doing very well in this regard, and it can do even better because it has that mindset to ensure that evaluation findings are utilized”, noted Bagele Chilisa, EAP member.

Members of the Evaluation Committee and the Executive Board of IFAD engaged prominent ly during the first day of the event, including

by providing feedback to the inputs received from independent evaluation, discussing how they could obtain more value from independ ent evaluation, and analysing how they could make better use of findings and recommenda tions from independent evaluation.

“IFAD has come a long way in terms of where it is compared to where it was in terms of evalua tion. It is probably the UN agency in Rome that has the most interest in evaluation, both internal and external. This is something that we, as mem ber states, need to bring into the other organiza tions”, observed Dr. Miguel Jorge García Wind er, Ambassador and Permanent Representative of Mexico to the Rome-based UN agencies.

IFAD senior management representatives were equally active, and illustrated key ongoing changes at IFAD, and underscored the ensuing demand for strategic and thematic evaluations. This included a brief presentation on the Up dated Development Effectiveness Framework, and ongoing work to strengthen the Fund’s self-evaluation function.

“IOE is helping to place IFAD as a learning or ganization in the overall global architecture. The products and processes that are being led by IOE have been really critical in helping us to think about how we can put IFAD at the centre of the learning and forward-thinking agenda”, highlighted Jyotsna Puri, Associate Vice-Pres ident, Strategy and Knowledge Department at IFAD.

The IOE multi-year evaluation strategy, ear ly insights on its implementation and ongo ing reforms of methodology and independent evaluative products were at the centre of the presentation that Fabrizio Felloni, IOE Depu ty Director delivered on behalf of the IOE di rectorate. In his presentation, Mr Felloni also touched upon tentative topics for future eval uations, and outlined the process that IOE has adopted to enrich engagement with IFAD man agement and collaboration with global evalua tion networks.

The three-day meeting also afforded EAP members the opportunity to engage in bilateral meetings with the directors of the evaluation offices of FAO and WFP and CGIAR.

Miguel Garcia Winder

Yaya Olaniran

Bjorg Skotnes

Haifa Aissami Madah

Caka Alverdi Awal

Mei Hongyong

Medi Moungui

Yves Guinard

TALK QUALITY ASSURANCE

utility through enhanced engagement

Quality endures, in all aspects of life. Evaluation is no exception. IOE takes quality assurance seriously – very, seriously. Only the very best will do. Rig or, precision and attention to detail are non-negotiable, and permeate the procedural, methodological and substantive robustness of the IOE approach.

In its pursuit of qualitative excellence, the IOE team draws from the contributions of a cadre of highly qual ified professionals, with academic distinctions from some of the world’s most reputable institutions. We come from thirteen different countries, master seven different languages, and have direct field experience in every continent. Our areas of expertise cut across the full spectrum of the evaluation rubric encom passing, inter alia, international development, research, teaching and capacity development, strategic planning and communications, and programme and knowledge management and administration.

Team members make substantive contributions not only to their own areas of work, but also provide valu able feedback to multiple evaluations through the newly revised peer review processes. These processes ensure that the highest qualitative standards are met, as every product draws from the widest possible array of insights.

Building on these solid foundations, in 2021 our Director, Indran A. Naidoo, established the IOE Evaluation Advisory Panel (EAP) to further our Office’s contribution to IFAD’s organizational and development effectiveness. During their first year, the members of the EAP provided comments on the IOE Multi-Year Evaluation Strategy, the 3rd edition of the IOE Evaluation Manual, and the Thematic Evaluation on Climate Change Adaptation. In addition, between 2021 and 2022, they delivered seminars on their respective are as of expertise. Henceforth, the EAP members will meet annually with the IOE team, as well as with IFAD Management and Governing Bodies representatives, as required, to provide insights and recommenda tions on the work undertaken and future strategic directions for IOE.

In the margins of the EAP’s first annual meeting, in the afternoon of Thursday, 14 July 2022, Independent Magazine sat down with the five members of the Panel, for a thought-provoking conversation on the present, past and future of IOE.

Good afternoon, esteemed colleagues.

Good afternoon, Alexander

Could you briefly describe the role and functions of the Evaluation Ad visory Panel and how these fit into IOE’s quality assurance process?

Donna

Our role is really to provide diverse per spective. We’ve engaged in some sub stantive reviews related to strategic planning. Being here has really given us the opportunity to understand the structure of the organization and how IOE supports and interacts with the oth er parts of the organization.

Donna Mertens Emeritus Professor Gallaudet University

Gonzalo

Evaluation is a never-ending story. You will never get the perfect evaluation system. That will never hap pen, ever. What happens is that you need to adapt, change and find new ways to use and produce evaluation because it has to do with exchanging with people and convincing people. Therefore, it is important to have various voices that can help you at various levels and in various circumstances to do the job in a better way. The good thing about the EAP is that we are a diverse group, and thus we can bring a variety of insights into the processes and work of IOE, including on how to improve current practices.

Rob

Our value added is at the strategic level, mostly. The overarching documents such as the evaluation man ual, the multi-year strategy, the evaluation policy, those are the ones that we have all provided inputs to. Timewise, we cannot comment on everything that is happening in the office. It’s more on an invitation basis. The structural support is, therefore, really at the strategic level. For this reason, it has been very important for us to understand what the other parts of IFAD are doing, and how evaluation relates to and interacts with them.

How do you see your individual areas of expertise contributing to strengthening the quality assurance process of IOE’s evaluation products?

Rob

I’m part of the second wave of transformational change. Donna established this for social justice, and from 2015 onwards the nexus between development and environment became extremely important be cause of climate change, biodiversity loss and a broad variety of eco-system related problems. This is now termed as the ‘sustainability crises’ of the world, because we cannot sustain the way that we are currently living. I bring this additional perspective to the group and to the evaluation office.

Bagele

I have written a lot on indigenous methodologies. This is about how we can make sure that those voices that have been left out, in terms of informing our evaluation process es from design to dissemination of findings, can be brought to the table. Currently, there is a movement on decoloniz ing evaluation, and I happen to be one of the activists in this regard. I bring this to IFAD, giving my own experi ence in terms of how the organization can engage with decolonizing evaluation so that evaluation is relevant, the methodologies are legitimate, and what we are told is working actually is working. Unfortunately, most of the time we are told that projects are doing well, the outcomes are good, while on the ground very little has changed. The international evaluation community is beginning to realize that the world is very complex. We need to integrate knowledge systems, and now we are at a point where we are asking “how”. My role is just to share my own experiences in this regard, and advise on what can be done.

Rob van den Berg

Visiting Professor

Donna

My work has been in the development of a transformative framework for evaluation. That work preceded the sustainable development goals (SDGs), which in turn preceded the use of transformation that we are hearing a lot about now with regard to the SDGs. I am delighted to see that this is a concept that has now come into prominence because when I do evaluations I want to design them in ways that are inclusive of a full range of stakeholders. Yes, we need to consider the funders and the programme people, but we also need to consider the people who have traditionally not had their voices heard, and not just in the provi sion of data, but also in the discussion around the important questions and the ways that we can collect the data so that it has validity within the context that we are working in. We need to be consciously aware of the power inequalities that are inherent in evaluation situations. How do we address those as evalua tors and bring in the voices of people with disabilities, women and all those that have been pushed to the margins? This is what most of our programmes are designed for, those are the intended beneficiaries. What motivates me is to work with people with the goal of affecting this kind of change.

Gonzalo

What I bring is the knowledge and the experience that I have had in terms of helping to put together mon itoring and evaluation systems. This is not only adding up evaluations, but it means looking at how you can produce and build a robust system. I have delivered this work at country and agency level. I think this is important because you may have a system, but within it there may be several areas for improvement. For instance, on the communications side, sometimes evaluators are not very clear in presenting results. That’s not very helpful for countries and policymakers. The way you communicate results, the way you address evaluation findings, is an area where evaluation offices, traditionally, can improve.

My area is around international good practice in terms of quality standards, criteria, doing joint evaluations. It’s important – as you see reflected in the new Evaluation Manual – that these agreements at the international level get appropriated, used, integrated and translated into the system that you have here. I can also contribute ideas around collaborative work, including with other providers and partner countries, which I think is an important way forward to share and build capacity.

What do you see as being the most important points of reflection and feedback within IOE’s current qual ity assurance process?

Bagele

The biggest mistake that I really zoom in is that of excluding other knowledge systems. I see myself as contributing to the quality of evaluation from that perspective. Right now, the mistake that evaluation is doing is to show that millions of dollars have been spent on development programmes, and while very little has changed, evaluations have been saying that projects have worked very well. This means that the discipline has been focusing

Bagele Chilisa Professor

University of

more on compliance and accountability. It is time that all UN organizations begin to reflect on this big mistake that is recurring. If the SDGs are going to really bring about change, then we need to look at this mistake and try to address it, otherwise the outcomes of the SDGs are going to be the same as their predecessors, the MDGs.

Donna

I would say that I’m a methodologist, that’s who I am, that’s what I do. Can I provide inputs from the be ginning to the end of a process? Of course, I can. However, my strength and experience lie in the meth odology. It is not just methodology, its methodology associated with social, economic and environmental justice. If we are going to talk about those as our goals, then I look at how do we integrate them into the decisions that we take about methodology.

Hans

Upward accountability is very important, and we need to look increasingly more into that, while not forgetting that we live in a system where many countries are funding UN organizations out of their tax payers’ money, and thus have an obligation to account for what is happening with this money. Therefore, we need both types of accountability to be able to balance interests in an equitable way and make insti tutions work.

Rob

There is also a micro/macro paradox in place, which is that at times there are very successful micro inter ventions that really succeed in bringing change, while the overall change in the country is not there, and even what is achieved is only temporary and does not continue beyond the lifecycle of the project. We see this especially in the environment-development nexus, where you try to address both and while some real transformations are possible, the overall trend is that the environment gets further degraded, we are going to have massive biodiversity loss, and we are running towards climate change. This, despite interventions that show how it could potentially be tackled. There is role for evaluation to point this out and not just be hap py to report that a certain project had a temporary and limited results, but to highlight what is going in the wrong direction in countries and internationally.

What do you think are the strengths of the Office and going forward, which are the areas that have opportunities for growth vis-à-vis emerging priorities for evaluation?

Hans

It’s really important to build on IOE’s innovative practices. We are in a situation where we have had a pandemic that has pushed millions of people into poverty. This is an enormous challenge for IFAD. Getting innovative solutions and scaling them up is critically important in these times.

Hans E. Lundgren

former Head

Unit, OECD

Gonzalo

We see strengths in the office, starting with the leadership from Indran that brings methods, technical knowledge and the focus on independence. There are areas for improvement whilst maintaining the independence. For instance, in the way we communicate and exchange ideas with others.

Donna

There’s a definitive strength in that evaluation is very integrally connected throughout IFAD. Every part of the organization that we talk to spoke about how they use and can use evaluation products, and some times that they wanted even more. For me, that speaks to the perceived value of evaluation, which is not always present in every organization. In terms of future opportunities, moving into a position where more data can be collected about the cultural barriers that are preventing the advancement of the people tar geted. We need to foster the advancement of everyone, especially the most marginalized and the poorest of the poor, who have not had access to resources.

Bagele

For me the strength is in the existing structures, such as the evaluation policy – which is very progressive and has transformational and decolonizational intentions –, the multi-year strategic plan, which is also very progressive, and the evaluation manual, which spells out the methodologies. This is very transpar ent, and points to direction and intention. Above all, the strength is in the open mindedness of the lead ership. In this regard, I can see how the EAP’s contributions are starting to influence and permeate the outputs of the office. Now, the challenge is in documenting innovative practices that come from what is being done. IOE has the strength and opportunity to come up with outputs such as an ‘indigenous evaluation framework’ that allows the marginalized to voice their concerns and be truly heard.

Gonzalo Hernández

THE IFAD EVALUATION ARCHITECTURE

Spotlight on quality enhancement processes, procedures and tools*

Independent evaluation and self-evaluation are the two pillars of the IFAD evaluation architecture. IOE is responsible for the for mer, while Management oversees the latter. Before delving into the details of the different approaches quality enhanced processes (QE), it is paramount to start by recognizing that the entire evaluation function at IFAD follows in ternationally recognized good standards and practices.

Pursuant to the publishing of the 2022 edition of the IFAD Evaluation Manual [here], common quality standards now apply to both self- and independent evaluation. It is this first time in the history of the Fund that methodological guidance and standards for evaluations are applied across the whole organization. This achievement enhances consistency between the two evaluative functions, ultimately foster ing a stronger results and evaluation culture at IFAD.

Both independent and self-evaluation func tions work at project, country or regional, and corporate or thematic levels. Furthermore, the majority of evaluations are based on contri bution analysis, which aims to measure inter ventions’ contribution to the overall change. To complement these analyses, impact assess ments or evaluations, conducted by RIA and IOE, respectively, and corporate impact assess ment reporting, conducted by RIA, are based on attribution analysis.

With these overarching quality standards and levels of analysis in place, differences nonethe less remain between the two pillars in terms of

quality enhancement processes, procedures and tools. IOE is structurally, functionally and behaviourally independent of Management. The organizational and functional independ ence of IOE is essential for ensuring sound, credible and transparent evidence consistent with international norms and standards and the principles set forth in the Revised IFAD Evaluation Policy [here], including data privacy and confidentiality.

The range of evaluation products is tailored to the different needs and potential uses to sup port accountability and learning within IFAD. They are interlinked and the delivery of one product contributes to others. The composition and design of IOE products address three dis tinct priorities: (i) providing the necessary ev idence for accountability at the project, coun try and corporate levels; (ii) contributing to learning by providing sufficient knowledge to IFAD Management and relevant stakeholders to strengthen organizational and development effectiveness; and (iii) strengthening the credi bility and quality of self-assessments and inde pendent evaluation products.

All evaluations undergo a thorough peer review process with the aim of enhancing the quality of the products. Different mechanisms of IOE peer review are needed for evaluations at corporate level, country level, and project level, as reflected throughout the four stages involved in all evaluations outlined below. Not all of the following steps are sequential, and some are overlapping or iterative; nonetheless, the broad stages provide the overall flow of tasks throughout the evaluation process.

Four stages of the IOE evaluation process

In the context of stage four of its QE process, and in-line with its Multi-Year Evaluation Strat egy [here], IOE is increasingly building evalua tion dialogues to support evaluation capacity and use within and outside IFAD. These efforts are ensuring that IOE retains and deepens its position as an internationally recognized lead er in the evaluation of rural development pro grammes, policies and strategies. Moreover, IOE is an active member of the United Nations Evaluation Group (UNEG) [here], the Evaluation Cooperation Group (ECG) [here], the Global Evaluation Initiative (GEI) [here], and the Inter national Development Evaluation Association (IDEAS) [here].

In addition, our staff members engage promi nently in multiple networks, associations and knowledge coalitions, including the Internation al Research Group for Policy and Programme Evaluation (IntEval), the European Evaluation Society [here], the African Evaluation Associ ation [here], the Asia Pacific Evaluation Asso ciation [here], the Wilton Park dialogue series [here], and the gLOCAL Evaluation Week [here].

With regard to the self-evaluation system, this is the responsibility of Management and is overseen by the Operational Policy and Results Division (OPR) and the Research and Impact Assessment Division (RIA). At the core of self-evaluation is the Development Effective ness Framework (DEF). Introduced in 2016, the DEF was developed to ensure that evidence is collected from projects and systematical ly used and to create the necessary structure to facilitate the collection and use of evidence in decisions on project design and implemen tation. Self-evaluation projects are designed to achieve the expected results of the DEF –namely, to strengthen accountability, enhance learning and ultimately, ensure that the deci sion-making process is based entirely on reli able evidence. An updated version of the DEF will be introduced in 2022.

Self-evaluation products are developed at three main levels: country, project and corporate. At the country strategic opportunities programme (COSOP) level, self-evaluation begins at design, when the results framework for the country strategy is reviewed by IFAD’s Quality Assurance Group, OPR and other members of

the Operational Strategy and Policy Guidance Committee, utilizing the development effec tiveness matrix for COSOPs. At completion, COSOPs undergo a completion review – i.e. a self-evaluation of their strategic objectives and IFAD’s performance in achieving them.

At the project level, self-evaluation is fully inte grated into the operation life cycle. At design, the development effectiveness matrix is used to review and enhance evaluability. During implementation, project teams prepare the annu al supervision report, describing the progress made and identifying the main challenges en countered during execution. At midterm, pro ject teams conduct a full review of the progress made and report it in the midterm review. At the end of the operation execution period, the relevant regional division prepares a project completion report (PCR). In addition to these common self-evaluation practices, which are applied to all projects, RIA conducts rigorous impact assessments of a representative sam ple of approximately 15 per cent of the projects closing in each replenishment period.

At the corporate level, the Report on IFAD’s De velopment Effectiveness presents the Fund’s annual operational and organizational performance by reporting on a set of 79 Results Management Framework (RMF) indicators agreed upon with Member States. IFAD also conducts thematic or cluster reviews on areas of specific interest to the Fund.

Looking ahead, IOE and Management will con tinue to maximise synergies and complemen tarities between independent and self-eval uations in two broad action pathways. First, collaboration between IOE and Management is pursued throughout the evaluation process, consistent with the independence of IOE. This includes the processes for selecting, planning and designing evaluation products, conducting evaluations and ensuring the identification and sharing of lessons learned and recommenda tions. Second, both IOE and Management work to ensure that the findings and lessons from each evaluation are communicated, absorbed and applied across institutions and shared with development partners and end clients in rural areas.

EVALUATORS CAUGHT FLAT-FOOTED BY COVID-19

Many evaluators ap pear to have been caught flat-footed by restrictions resulting from the pandemic environment, es pecially limits on travel and onsite work. Moving from this premise, the forthcoming book titled ‘Evaluation in the Era of COVID-19’ takes several institutional, national and dis ciplinary perspectives to ex plore both the shortcomings of evaluation but also the in novations and successes. The editors of the publication, Indran A. Naidoo, Director of the

Independent Office of Evalu ation of IFAD (IOE), Ray Rist, former president of the Inter national Development Eval uation Association (IDEAS), and Pearl Eliadis, Associate Professor at McGill University, discussed these issues during the annual meeting of the In ternational Research Group for Policy and Program Eval uation (INTEVAL), which took place on 23-25 May 2022, at Wilton Park.

“This book will make a significant contribution to the field

of evaluation, given that COV ID-19 has changed in many ways the manner in which evaluation has been working. Evaluation, with its focus on accountability and good gov ernance, should have been well-positioned to respond to the COVID-19 crisis. But, it is far from clear that evaluation systems actually rose to the challenge”, affirmed Dr Nai doo.

Hosted on the prestigious grounds of Wiston House, in the UK, the event was organ-

ized by Ida Lindkvist, Jos Vaessen, Ray Rist, Rob D. van den Berg, and by the UK Foreign, Commonwealth and Develop ment Office. Attended by over forty INTEVAL members, the meeting provid ed a space for distinguished authors to present recent and upcoming pub lications.

“This is the 37th INTEVAL an nual gathering, which has taken place uninterruptedly since 1986. When we gather each year, a signifi cant constella tion of evaluation leaders and authors emerge to help carry the group forward into the fu ture. To date, the group has published 31 books on various themes of evalu ation”, explained Dr Rist.

Presentations on ‘Evaluation in the Era of COV ID-19’ offered multidisciplinary perspec tives and critical assessments on how evaluation and evaluators responded to the pandemic. Presentations included Peter Wilkins (Australia), Ray Pawson (United Kingdom), and crosscutting insights into the impacts on the SDGs (Doro thy Lucks and Robery Lahey).

Jan-Eric Furubo presented his work on evaluation in times of turbulence, providing key in sights into how knowledge is selected by policymakers dur ing crises. Dr Naidoo analyz ed the UN’s socio-economic response plans, demonstrat ing how traditional oversight activities were suspended and unable to respond with sufficient agility.

“During the meeting, distin guished participants will be

taking stock of the latest books that have been prepared, in cluding the upcoming publi cation titled ‘Evaluation in the Era of COVID-19’”, Dr Naidoo noted.

Discussing the forthcom ing volume, Dr Naidoo, Dr Rist and Prof. Eliadis noted that the evaluation sec tor had evolved in environments that possessed a degree of stability; gov ernments had predictable and structured plan ning processes with clearly es tablished sets of users for reporting results. The COVID-19 crisis disrupt ed many of these systems and processes, as well as the connections among them. The demand for discrete studies based on sim ple attribution models declined while a move towards broad er knowledge streams, capable of respond ing to current crises, acceler ated.

“An original me ta-analysis of evaluation lit erature reiter ated points made by several authors: in the first half of the pandemic, evaluators were overwhelmingly focused on practical and operational mat

- Indran A. Naidoo, Director, IOE, IFAD

- Ray Rist, former President of IDEAS

- Pearl Eliadis, Associate Professor, McGill University

ters, while other knowledge actors rapidly occupied the field. In some cases, evalua tors were often absent in crit ical conversations about what the pandemic meant for evalu ation”, stated Prof. Eliadis.

Few evaluations were able to address the implications of the new democratic deficits nor the human rights vi olations that were revealed by exacerbated or emerging forms of inequality: these are not areas of comfort for tra ditional evaluation practice. Evaluators were being asked to move beyond methodolog ical problems and technical solutions to engage with sub stantive and high-level inquir ies that were content-driven, consistent with human rights norms, and informed by what has been learned during the pandemic.

“The book argues that evalu ators paid too little attention to the compelling need for substantive need for transfor mation in the profession as compellingly argued by Mi chael Quinn Patton. That is the context for extracting lessons to address the major systems transformations needed, with a focus on human rights as argued by Pearl Eliadis, and by Michael Quinn Patton who notes the need for that trans formation in health, climate, food, and social justice”, said Prof. Eliadis.

The session closed with re marks from discussant Dr Leslie Fierro, the Sydney Duder Professor of Evaluation at the Max Bell School of Public Policy (McGill University) who noted the implications of this

research for teaching, capacity building and the new gen eration of evaluators.

The INTEVAL gathering also included a seminar on ‘the future of evaluations’ hosted by the Independent Evalua tion Group of the World Bank (IEG), which co-founded the event.

“Looking ahead, during the meeting, we agreed to start work on a new book exam

ining the interrelations of evaluation with artificial In telligence. Three other books are presently in progress; in cluding evaluation and theo ry of change; evaluation and the post-truth environment; evaluation and sustainability”, highlighted Dr Rist.

In 2023, the INTEVAL group will meet at IFAD, in Rome, It aly.

Malawi and the challenge of sustaining productivity gains

IFAD-supported programmes in Malawi have initiated many positive practices that would need to be sustained and scaled up. Between 2016 and 2022, pro jects have achieved signifi cant increases in productivity, through provision of technol ogy, inputs and irrigation. In most cases, these gains were eroded soon after pro ject completion. For this rea son, the Independent Office of Evaluation of IFAD (IOE) is recommending the develop ment of a strategic approach for enhancing the impact and scale of successful practices and initiatives. These and other findings emerging from IOE’s Malawi Country Strate gy and Programme Evaluation (CSPE) were presented and discussed on 17 May 2022, during a virtual workshop.

“A lot of good things have started. We have a large num ber of initiatives and practices. We’ve seen a thousand flowers blooming in Malawi, but we want them to cover larger areas and have a more trans formative impact. Productivity gains are often eroded as soon as farmers stopped receiv ing inputs from the projects. We need to think beyond the project’s immediate support. For this, we need to have a more strategic approach to taking initiatives and practices to scale”, stated Ms Johanna Pennarz, Lead Evaluation Of ficer at IOE.

The IFAD programme initiat ed many positive practices in Malawi, which community level organisations have often not been able to sustain after project closure. The country programme invested heav ily into institution building. Nevertheless, sustainability met institutional and financial challenges, including insuffi cient funds and capacities at decentralised levels, low government ownership and insuf ficient integration of project activities into Government’s annual work plans and budg ets. Farmers groups were not formalised and empowered to engage with other value chain actors, in particular vendors, traders and processors.

Co-organized by the Gov ernment of Malawi and IOE, in collaboration with IFAD’s East and Southern Africa Division (ESA), the on-line virtual workshop brought together a wealth of high-level attend ants, including Eng Geoffrey Mamba, Director of Irrigation Services, Ministry of Agri culture; and Dr MacDonald Mafuta Mwale, Secretary to the Treasury, Ministry of Finance. Over 80 participants joined the event, representing the government of Malawi, development partners, civil society organizations, private sector partners, and IFAD senior management and staff.

The main objectives of the CSPE were to assess the re sults and performance of the ongoing Country strategic op portunities paper/programme (COSOP) 2016-2022; and to generate findings, conclusions and recommendations

Profile

Dr. Johanna Pennarz Lead author, Malawi CSPE

Dr. MacDonald Mafuta Mwale, Secretary to the Treasury, Ministry of Finance, Malawi

for the next COSOP in 2022.

“This is the first CSPE in Ma lawi, which presents a per fect opportunity to assess the results of IFAD projects and country strategy in Malawi. We expect that the findings and recommendations from this CSPE will inform the structure and strategic focus of the next COSOP. Therefore, the impor tance of this workshop cannot be overemphasized”, affirmed Dr MacDonald Mafuta Mwale.

IFAD’s financial allocations have almost doubled be tween 2016-2022. Projects be came larger and included an increasing number of stake holders and service providers to deliver the expected re sults. The country programme supported a large number of initiatives, innovations, pi lots and practices, many of them supported by additional grants. These efforts initiated many positive practices.

“IFAD projects have increased the capacity of our farmers to produce more even in the wake of increased calamities such as floods and droughts. We also need to ensure that, as a Ministry, we implement projects and programmes that are funded by IFAD by follow ing the right procedures and rules. For this reason, we are very excited to listen to IFAD colleagues during today’s workshop”, noted Eng Geof frey Mamba.

These achievements notwithstanding, the project had limited success in diver sifying production systems and securing reliable market access for smallholder farm

ers. Trade-offs exist between smallholders’ concerns about food self-sufficiency and the transition to market produc tion. Recent operations tried to address the multiple chal lenges through complemen tary designs, but in practice overlaps and synergies were too few to make a step change.

In light of the successes and constraints encountered, the IOE report recommends that IFAD’s next COSOP adopt an explicit approach to address ing chronic food insecurity and malnutrition through di versified and sustainable pro duction system, and include a clear strategy on how it will enhance the impact of suc cessful practices, support co herent roll out across districts and use the lessons learned to enhance the effectiveness and impact of upcoming initi atives and operations.

“The project had limited achievements in diversifying production systems and secur ing reliable market access for smallholder farmers. Food re mains the most important ex penditure item for smallholder farmers. Moving forward, IFAD would have to further enhance its support to sustainable and diversified production sys tems and take decisive steps to resolve the ongoing imple mentation challenges. To this aim, realistic implementation planning and effective over sight are of paramount impor tance”, highlighted Dr Indran

A. Naidoo, Director of IOE.

IFAD began operations in Ma lawi in 1981. Since then, it has provided USD 350.5 million lending..

2022 Blantyre Arts Festival goes green*

Bearing in mind that climate change is affecting the whole world — Malawi not being exceptional — this year’s Blan tyre Arts Festival (BAF) was celebrated under theme; ‘Climate Resilience Through Cultural Preservation’ held at Njamba Freedom Park.

At a press conference at Amaryllis Hotel in Blantyre, BAF’s founder and Executive Director, Thomas Chibambo, said they decided to go on the ‘green’ campaign because they want artists “to becomes a vehicle to spread messages and precaution measures to be taken to combat cli mate change’s negative effects”.

“BAF and its partners would like to become part of that environment conservation solution in Malawi,” he said, adding that climate change has triggered several natural disasters in Mala wi — the recent being Cyclone Ana in January this year.

After a two-year break due to CoVID-19, BAF was back in full swing with high-profile artists sucn as Kasama Arts from Zambia; Temp Trio from Hannover, Germany — to share the stage alongside Malawi’s great artists, the Black Missionaries; Ethel Kamwendo Banda; Gloria Manong’a; Agoroso and Skeffa Chimoto.

As usual, the festival was spiced by other per formances in poetry; comedy; cultural dances and visual arts. There were also training work shops in arts management; cultural music pro ductions; theatre productions; arts and cultural entrepreneurship and many more.

Past festivals’ foreign artists included Mutaba ruka from Jamaica, Salif Keita from Mali, and Rebecca Malope from South Africa.

Chibambo underscored the importance of the fact that the BAF 2022 was free for everyone for first time, “BAF believes that arts and culture is part of human rights consumption — therefore we would like to reach out to everyone to appreciate arts and culture without limitations”.

Held for two days, from October 8-9, BAF 2022 was made made possible with the support from UNESCO Malawi Commission; The Fed eral Republic of the Germany Embassy in Mala wi; Friends of Malawi Circles in Germany; Han nover City; First Capital Bank; Amaryllis Hotel; World Connect Malawi; Entertainers Promotions; aware&fair; Chibuku — amongst others.

WALKING THE TALK

GEI lays out the directions of its work

The first 18 months of implementation have set a solid foundation for the success of the Global Evaluation Initiative (GEI). Building on the accomplishments achieved thus far and addressing identified risks will be critical to maintaining GEI’s momentum. To discuss these, members of the GEI gathered in Paris, France, for this year’s GEI Partnership Council, on 3-4 May 2022. As a key partner of the GEI, the Independent Office of Evaluation of IFAD (IOE) was represented by its Director, Indran A. Naidoo.

Dr Naidoo was among the first the first to be

lieve in the value of the GEI. In his former role as Director of the Independent Evaluation Of fice of UNDP (IEO), Dr Naidoo signed a Mem orandum of Understanding with Alison Evans, World Bank Vice President and Independent Evaluation Group (IEG) Director General, in January 2020. This marked the first step towards establishing a global partnership to sup port evaluation capacity development. The col laboration foresaw that IEO and IEG would pool resources, share knowledge and expertise, and leverage the comparative advantages of each institution for scaling up initiatives and coordi nating global efforts on building M&E systems

and capacity. Ten months later, the GEI was officially born.

Much has happened since the GEI was launched in October of 2020. What started with two or ganizations signing an MoU, has grown to over 35 partners – organizations from all over the world, who are bringing their experience and expertise to deliver on a shared vision at GEI. Now, over eighteen months later, most implementing partners have received their financing and, across the partnership, there are many impressive activities underway. The GEI Glob al Team has delivered on nearly all activities proposed in 2021, successfully processed the majority of GEI grants, and established a strong brand identity for the program.

Achievements discussed during the 2022 Part nership Council include progress in the development of a culture of evidence-based decision making through the strengthening of country M&E systems, and ensuring that evaluative evidence is used to make better decisions. By virtue of a range of training and professional

development activities, the GEI is strengthening stakeholders’ capabilities for managing and conducting evaluations and for managing and using M&E to enhance evidence-informed decision-making, organizational learning, and accountability. Ultimately, it is envisaged that these efforts will contribute to developing more relevant and effective policies and programs that will help countries achieve their national goals and progress towards the SDGs.

Discussions also addressed key operational matters for GEI’s future success. Risks to the programme were discussed and mitigation strategies for each were proposed. These risks include partnership, political, institutional, and financial risk, all of which are expected as GEI continues to grow, expanding the scope of the program and adding more partners.

Key outcomes of the event include the endorse ment by GEI members of the overall direction of GEI’s work, its new fundraising strategy, and proposed evaluation methods for each of GEI’s business lines.

GEI website [access here]

Value chain dynamics pivotal for investments in horticulture

The evaluation highlighted the need to promote market access and vertical linkages between value chain actors to ensure the sustainability of benefits and the scale of impact, as the HSP made in sufficient efforts to link farm ers to processors and other value chain actors to ensure sustainable markets and fair pricing.

While Irrigation improvement activities reduced water losses, a number of factors affected the sustainability of the interventions. These include lack of clear operation and maintenance arrangements, limited collection of user fees, generally weak capacity, and frequent institutional changes related to water consumer as sociations.

Creating linkages and formalizing contractual agreements between producers, wholesale buyers and traders is essential to help enhance efficiencies in production and guaran tee demand for horticultural producers. The project per formance evaluation carried out by the Independent Office of Evaluation of IFAD (IOE) in Uzbekistan highlights the im portance of these value chain dynamics.

The IOE report presents the findings of the project perfor mance evaluation of the ‘The Horticultural Support Project (HSP)’ project in Uzbekistan. The HSP was the first IFAD-fi nanced project in Uzbekistan, and was implemented in nine districts of the region of Sur khandarya with a total budget of US$25.7 million.

While the targeting of poor er dehkan farmers in Sur khandarya was a relevant choice for piloting horticul tural support, the evaluation report found a tendency, in practice, to promote larger loan sizes. Thus, more support was provided to larger, non-dehkan farmers; this, as there was no monitoring of beneficiaries’ actual wealth status, and it is highly likely that poorer dehkan farmers, in particular, were excluded from accessing finance.

Women and youth were also scarcely involved. Cultural constraints made it difficult to encourage women to take loans and to receive training, while youth were not ade quately considered as a target group in the project’s design, and there was no monitoring of participants’ age.

Irrigation infrastructure im provement was another im portant aspect of the project.

Looking ahead, the report recommends that future in vestments in the horticulture subsector should focus more on regulatory aspects, pay greater attention to institu tional capacities and frame works, and ensure that IFAD maintains its comparative advantage by allocating suf ficient resources and focus to target poorer farmers, women and youth.

During the HSP, Uzbekistan’s economy was driven primarily by state-led investments and exports of natural gas, gold and cotton. The agriculture sector has traditionally been dominated by cotton and wheat. However, environmen tal and social issues associ ated with cotton production have spurred a shift from cot ton and wheat to horticulture, in order to export higher value-added goods rather than raw materials. The strongest growth in the last 15 years has come from horticulture.

Uzbekistan recognized as safest country in Central Asia*

Uzbekistan has been recognized as the safest country in Central Asia by the Global Peace Index rating. Uzbekistan ranked 86th in an annual Global Peace Index this year, finishing at 86 among 163 countries, while Iceland remained at the top position, according to a report by an international think tank.

The Global Peace Index (GPI) gauges ongoing domestic and international conflict, safety and security in society, and the degree of militarisation in 163 countries and territories by taking into account 23 indica tors.

Iceland remains the most peaceful country in the world, a position it has held since 2008, the Institute for Economics and Peace (IEP) said in a statement. It is joined at the top of the index by New Zealand and Ireland, with Afghanistan remaining the least peaceful country in the world.

The other countries of Central Asia: Tajikistan ranked 105th, Kyrgyzstan - 91, Turkmenistan - 92 and Ka zakhstan - 97.

APPROACH NEEDED FOR EVALUATION TO REMAIN RELEVANT

Evaluation needs to be transformative. For this to happen, one needs to adopt a systems approach, consider the environment and ensure downward accounta bility. There is no gold stand ard, and methods should feed the question. This is not easy given evaluation’s typical project-level focus, and its tendency towards upward accountability. Further, the notion of one expert evalua tor delivering results, over a multidisciplinary team means that issues of complexity are not addressed. Juha I. Uitto, Director, Independent Evaluation Office, Global Environment Facility, and co-editor of the book titled ‘Transforma tional change for people and the planet’, brought this issue to the forefront at the 2022 An nual Conference of the Czech Evaluation Society, during an on-line webinar.

“We can’t continue to evaluate projects as if they exist in a vacuum. We must take a sys tems approach to ensure that interventions actually make a difference”, stated Juha Uitto.

Indran A. Naidoo, Director of the Independent Office of Evaluation of IFAD (IOE), who wrote the introductory chap ter of the above-mentioned

volume, was among the panel members of the webinar which focused on the recent ly published book. In his in tervention, Dr. Naidoo sum marized highlights from his chapter which took an expe riential approach, and reflect ed on his experiences at three oversight units in the last 25 years, pointing to what was and was not achieved.

Drawing on national and international case studies, Dr. Naidoo examined the concept of transformation from a con textual perspective, noting the relativism in the concept which is context specific. He drove the point that intention does not mean results, and that critical thinking and independent functions are impor tant for talking truth to power. The panel presentation also featured Serge-Eric Yakeu Djiam, Vice-President, Inter national Development Evalua tion Association (IDEAS), and Susan Legro, Czech Evalua tion Society. Daniel Svoboda, IDEAS and Czech Evaluation Society, moderated the event.

“The book marks a significant contribution to evaluation in that it adds a dimension that is often not being spoken about, which is transformation. In my chapter, I talk about the rela tivism of transformation, for which there is no universal definition – what may appear transformative in one context may not be in another”, Dr. Nai doo explained.

“A key part of evaluation is to improve quality and make changes. Any changes can be considered transformative. The collection of essays in the book draw from a very diverse set of experiences, where people look at this topic from dif ferent angles”, Dr. Naidoo high lighted.

Published by Springer,, the book intends to provide an authoritative, interdisciplinary perspective on innovative and emerging evaluation knowl edge and practice related to environment, natural resourc es management, climate

change, and development. The book has its roots in the Third International Confer ence on Evaluating Environ ment and Development, held in Prague, Czechia, in October 2019. The conference brought together a large number of established and upcoming evaluators, researchers, and evaluation users from the Global North and South, rep resenting a wide variety of organizations, to discuss the frontiers of environment and development evaluation. Fol lowing the event, the organ izers identified and contact ed selected participants who made key contributions at the conference and asked them to develop their ideas and papers into full-fledged book chapters according to a co herent plan.

In addition to Dr. Naidoo’s opening chapter, the book also features substantive contributions from Dr S. Nanthikesan and Prashanth Kot turi, Lead Evaluation Officer and former Evaluation Officer at IOE, respectively. The book was also recently featured in the IOE Coffee Talk seminar series, at a special ‘meet the authors’ session. During the event, Juha Uitto was joined by Geeta Batra, co-author of the volume, for a joint pres entation.

Download open source, here. Purchase hard copy, here. Dr. Naidoo’s intervention, here.

Access IOE Coffee Talk ses sion on the book, here and here.

FURTHER READING

Participatory narrative methods and quantitative approaches: a possible combination to evaluate impact?

Fabrizio Felloni, IOE Deputy Director, recently co-au thored an article published in the eVALUation Matters journal. The study moves from the premise that there is a tradition of collecting narratives in social science research and evaluation, including stories of personal experiences from stakeholders of development programs, to make sense of complex situations. In this context, the article explores two examples of a structured approach to narratives, including elaboration through computer applications adopted in IFAD evaluations in Cameroon and Niger, in conjunction with oth er analysis methods, notably focus group discussions and impact survey data. The article presents the contribution of narratives to the epistemological (generating new and unexpected findings) and axiological (understanding end-users’ perspectives and value systems) capability of the evaluation exercises, and the key requirements for applying these ap proaches.

The study finds that participatory qualitative narrative in quiries with end-users of development interventions can complement quantitative methods for impact evaluation, and that narratives help fill evidence gaps and elicit from respondents their values and their own constructs of reality. In particular, the study discusses how through narratives, the evaluations in Cameroon and Niger obtained unexpected evidence on program impacts and revisited the assumptions made in their original theories of change, and explains that narratives helped open rich discussions between project end-users, project designers and managers, and promoted learning from experience at the grassroots level.

Felloni, F., E.Zucchini, M.Carbon and C.Bosch (2022): “Com bining participatory narrative methods with quantitative ap proaches to evaluate impact: Experiences in Cameroon and Niger”; eVALUation Matters Volume 1, 2022. Independent Development Evaluation, African Development Bank [here].

Once an evaluator, always an evaluator

The once Deputy Director of IOE, Ashwani Kaul Muthoo, came full circle when he became the first Director General (DG) of the Independent Evaluation Office of the BRICS New Development Bank (NDB), only a few months ago. Amidst these roles, a professional timeline that runs 30-years deep into global policies, country strategies, development projects and programmes.

Former Director of the Quality Assurance Group in the Office of the President and Vice President of IFAD, Ashwani’s evaluation expertise and knowledge of the agricultural and rural sectors lies deeply rooted in experiences covering more than 50 devel oping countries in Africa, Asia, Latin America and the Caribbean, and Near East and North Africa regions. Over the years, SouthSouth and Triangular Cooperation work have played a major part in Ashwani’s IFAD portfolio, especially in his roles as Director of the Global Engagement and Multilateral Relations Division, and as Director of the Global Engagement, Knowledge and Strategy Division.

On the eve of Ashwani’s departure from the Fund, Independent Magazine had the privilege to catch a glimpse of the soon-to-be NDB Evaluation DG’s perspectives.

Good morning, Ashwani.

Good morning, Alexander.

Could you tell us about your professional experiences in dealing with evaluations?

I started in IFAD’s evaluation office in 1990, and stayed until 2015. During those twenty-five years, I was involved in and spearheaded some significant and transformational evaluations that led to far-reaching reforms in IFAD. The first was in 2005, when I carried out a corporate level evaluation of the ‘direct supervision pilot programme’. Before that pilot, IFAD would entrust supervision of its projects and programmes to third parties, like international financial institutions, as well the UN office of project services. In that evaluation, we found that IFAD-led supervision was more effective compared to that performed by third parties. That evaluation laid the basis for IFAD’s first direct supervision policy. Another very important evaluation was the ‘field presence pilot programme’, in 2007. Again, we concluded that having perma nent presence in partner countries where we have operations leads to better results and sustainably of IFAD-funded programmes. I would like to single these out because I believe that they form the basis for IFAD’s recent decentralization. Equally important was the evaluation of IFAD’s first private sector strategy, in 2010. It was in that evaluation that we said that the IFAD should start doing non-sovereign operations, which it commenced over the last year. This means providing resources directly to private sector entities in developing countries, without sovereign guarantees.

Ashwani Kaul Muthoo Director General Independent Evaluation Office BRICS New Development Bank

In your opinion, is it important to have independent evaluation for IFAD to course correct and, if so, why?

In today’s context, I think that independent evaluation is more important than ever. IFAD has developed a self-evaluation system, over the years. This is fine, but self-evaluation involves people who are associated with design and implementation of operations. While I fully respect the professionality and capabilities of my colleagues who are doing those self-evaluations, there might be an element of bias. Thus, inde pendent evaluation brings in that perspective, that distance, which strengthens the credibility of IFAD’s overarching evaluation architecture. Independent evaluation has no stakes in what it looks at. The idea is to use robust methodologies to get accurate perspectives of what is happening on the ground in terms of results and lessons. A fundamental part of all this is to feed the findings back into the design of country strategies, projects and programmes. The two really have to go hand in hand. Self-evaluations are necessary, including since independent evaluation cannot cover the whole portfolio and do not normally intervene during implementation. This combination of independent and self-evaluation gives credibility to the organization as a whole, showing that IFAD takes accountability and learning very seriously.

What you see as being the main opportunities for evaluation to further permeate an evidence-based learning culture across IFAD?

First of all, IFAD is an organization that has put in place a lot of reforms in the last few years. The private sector non-sovereign operations are one example. There are a lot of transformations in IFAD’s business model as well as in its thematic priorities, such as greater attention to non-lending activities, including country-level policy dialogue. Against this backdrop, I think independent evaluation has a great oppor tunity to come in at the right time to assess the performance of these new initiatives and identify adjust ments that can be made to further strengthen results on the ground. We need to understand value for money, and having a strong IOE will allow us the opportunity to leverage the methodology and expertise to credibly report on performance and make the adjustments that need to be made moving forward.

Are there any final thoughts that you would like to contribute?

Having been on both sides of the institution in IFAD, management and independent evaluation, I can give credit to the whole organization for building an evaluation culture, a culture of results orientation, a culture of learning, and a culture of improvement. There is still further scope for the growth on this front. The exchange of personnel between management and IOE represents a great opportunity in this regard. In my case, after leaving IOE, I always leveraged the work of the Office, and instilled that kind of culture within my team, ensuring that there is attention to IOE’s lessons and to how they are being integrated in IFAD operations. This cross-fertilization is essential, and could be encouraged even further.

Looking ahead, I think that with Indran, the new Director, IOE has the required visionary leadership to take things to the next level. I really want to commend the whole of IOE for the excellent work delivered and for the future and look forward to engaging with the team in my new role.

MYTHS

On the surface, many evaluators feel that all that is needed is for people to follow their advice, and positive change will come about. To a certain extent, this may even be true. However, there is more than meets the eye, as evidence suggests that 70% of organ izational change methods fail. Why? To answer this question, the Independent Office of Evalua tion of IFAD (IOE) invited Dr Srini Pillay to speak at a special session of its Coffee Talk series, on 21 April. And special it certainly was, with rep resentatives of all three Rome-based UN agencies in eager attendance.

“

When you are doing an evaluation, you want something to change for the better. Focusing on mindset rather than changing of behaviour, actually gives you twice the likelihood of being successful in this regard”, Dr Pillay explained.

In his talk, Dr Pillay introduced participants to 5 myths related to performance appraisal and feedback, and explained how each of these op erates in the brain, and what alternatives peo ple can seek if they want to achieve more effec tive transformations through their evaluations. What followed was a lively discussion on how these myths could impact the structure and de livery of the performance evaluation process.

“The presumed goal of an evaluator is to share research-based findings in an objective manner that is received and acted upon in the service of the greater good. On the surface it sounds re ally easy. You evaluate, you decide what’s write and wrong, you communicate that, the person makes the changes and everything changes for the better – the problem is, things don’t actually work that way”, Dr Pillay stated.

By their very nature, evaluations are meant to shine a light on the strengths and challenges

APPRAISAL AND MYTHS DEBUNKED

that people face. While the goal may be to in spire change behaviour in the people who re ceive those evaluations, several bodies of re search from the biological sciences indicate that the approach taken is often misfocused. Emphasizing weaknesses, flaws, or other short comings, or even trying to “fix” the problem has an opposite effect. Moreover, this approach will likely activate the Negative Emotional Attractor (NEA), which causes people to defend them selves and, as a result, to close down. Not ex actly the desired outcome.

During the Talk, Dr Pillay also explained that people are more likely to reach a destination and appreciate the steps along the way if they are not just told what to do. Being told what to do creates weaker cognitive maps in the brain because there is no figuring out process that contributes to the plan. Furthermore, many studies have also demonstrated that strengthsbased performance appraisal motivates people to change and align with goals much more than one that looks at strengths and weaknesses or weaknesses alone. While this does not mean that evaluators should just emphasize strengths, it does suggest that the feedback should highlight the capacity to grow that strength.

“The reality is that you are delivering evaluation findings from a very complex brain to a very complex brain”, Dr Pillay concluded.

Dr Srini Pillay is a world-renowned keynote speaker, lecturer, author, consultant and Har vard trained psychiatrist. He is known for combining “head and heart” (figuratively and literally) in an approach to personal development and goal mastery that blends science, spirituality, and horns-grabbing joie de vivre to combat the stresses faced by ambitious and high-achiev ing people in academia, business, and life.

IOE

Learning, sharing and bringing the team together – the ‘CSPE way’ (which includes surviving a shipwreck)

Can you share knowledge, discuss substantive issues, and address technical matters all the while build ing a cohesive team spirit? In the case of IOE’s Country Strategy and Programme Evaluation (CSPE) Unit, the answer is “yes”. On 8 June 2022, the CSPE team gathered for a half-day mini-retreat, which proved to be fun, engaging and very informative. The informal setting and collegial environment favoured a dynamic flow of ideas, as interactive sessions, games, group discussions and presentations alternat ed throughout the morning. Substantive issues included similarities and differences between lessons learned, conclusions and recommendations; types of documents required for CSPEs; formatting of CSPE reports; strategic and operational recommendations; and challenges in recommendation formulation –all of which nicely tied into a group exercise on how to survive after a shipwreck, as a team, in the middle of the Atlantic Ocean. Rich in positive and useful takeaway messages (not least of which the fact that shark repellents actually exist), the mini-retreat turned out to be a success story in its own right; one which will no doubt be replicated in the future.

Getting the whole team back together: time for our second retreat

Going into the July 2022 IOE Retreat, expectations were running high. IOE staff and management alike were looking to build on the productive first retreat that was held under the tenure of IOE Director Indran A. Naidoo, back in November 2021. They were not disappointed, as expectations were met and – arguably – exceeded. An informal and jovial atmosphere set the tone for engaging discussions and rich sessions, with team building games and role play exercises always featuring prominently.

Sessions included an update and discussion on the revised Evaluation Manual; a discussion on improv ing IOE workflow; a review of the 5 Attributes of High Performing Teams; a discussion on effective team work; a conversation in individual teams discussing workload and recent feedback; a briefing on the retreat of the Evaluation Advisory Panel one week before; a discussion on leading smarter evaluations; and an interactive dialogue on ensuring impactful communication. Let’s not forget the cocktail making challenge, where the IOE team had fun making cocktails whilst developing active learning about behav ioural diversities and the impact of using them positively in groups.

Following the retreat, an action plan has been prepared to follow-up on the key items.