ONE SMALL STEP FOR THE U.N.

One giant leap for the audit and evaluation community

Audit and evaluation: worlds apart? Arguably, yes. Different paradigms, no doubt, different approaches to research and investigation, without question. The separation of the two functions ensures their independence. Therefore, said separation needs to be safeguarded – one might hastily conclude. But is working in silos really the only way? Maybe not. After all, both audit and evaluation are essential instruments for accountability, and both play crucial roles in the oversight of organizations such as IFAD. And not only that. If we look at some of the latest literature, research suggests that ‘enhanced oversight’ attained through collaboration among the two functions can promote a holistic three-dimensional view of performance, and create synergies between practitioners from different backgrounds. So where does this leave us? Are we to marry or forever divorce the two?

To answer this thorny question and tackle this important topic head-on, IOE hosted an event on 24 August to discuss the potential for crossover between the two functions, and their respective profes¬sional groups, as well as related incentives and potential barriers. IOE invited Helge S. Osttveiten, Director, Office of Audit and Oversight (OAI) of UNDP; Ana Rosa Soares, Chief of Section, Corporate Evaluation, Independent Evaluation Office (IEO) of UNDP; and Ray Rist, co-creator and co-director of the International Program for Development Evaluation Training (IPDET), and former president of the International Development Evaluation Association (IDEAS). Together with IOE Director, Indran –former UNDP IEO Director –, Helge, Ana and Ray plaid a prominent role in the ‘Joint Assessment of the Institutional Effectiveness of UNDP’ report. Published in 2017, this first-of-a-kind effort was conducted jointly by IEO and OAI – something that had never been done before.

Independent Magazine caught up with Helge, Ana and Ray to get an insider’s perspective into what proved to be a ground-breaking exercise, in more ways than one.

Good afternoon, esteemed colleagues.

Good afternoon, Alexander.

I have a feeling that this is going to be a long and multi-faceted story, with lots of twists and turns. So, let’s start from the very beginning. How did the idea for a joint exercise come about?

Ana

In 2013, UNDP started thinking about institutional effectiveness and how to measure it as a central focus of the organization’s strategic plan. In this context, in 2015, the Independent Evaluation Office, and the Office of Audit and Investigation agreed to embark in an approach to oversight that would be different. The objective of the exercise was to assess the extent to which policy and organizational measures enhanced UNDP’s ability to deliver higher quality programmes through results-based management and improve UNDP’s institutional effectiveness.

From a layman’s perspective, might I ask why was the joint nature of this exercise such a big deal?

Ana

Historically, both offices have worked according to their own plans. A joint assessment implied working across the traditional professional boundaries that we were very much attached to. It implied developing synergies of approaches, rationalizing the resources that are always scarce, and producing a coherent message to the UNDP management on this critical subject. It was not an easy task – but it proved to be a very rewarding one.

Before even opening the report, the first thing I noticed was that the title of the report refers neither to ‘audit’ nor ‘evaluation’. Instead, you use the term ‘assessment’. What was the thought process behind this choice?

Ana

What to call the exercise was the first challenges that generated very heated discussions. Every time we started to mention that we were going to have a joint audit and evaluation people’s eyes opened and they were really scared. In the end, Indran and Helge decided to call it a joint assessment. The time that it took to reach an agreement just on the name is the reflection of a much more significant challenge that needed to be addressed, which were decision rights, communication channels and protocols. That’s when Ray [Rist] had to come in and really help us come together as a team.

I guess that the phrasing of the title was not the only ‘wording issue’ you faced. From dealing with sensitive language to safeguarding independence, how far apart were the two offices when the time come to put pen to paper?

Ana

Writing the report and dealing with the sensitive language and the aspect of independence was a challenge. Audit reports to the UNDP Administrator, while evaluation reports to the Executive Board. As a result, the auditors’ language was different to the evaluators language, generating significant disagreement and severe delays in finalizing the product. We had to go back to the Directors to decide what and how would get communicated. The Directors were really very much hands-on in helping the team in finalizing the report so that we could say what we needed to say, but were able to do so in a way that did not attack but rather welcome contributions from the organization.

Helge

When it comes to audit and evaluation, it is too simplistic to say that they have different scope. It all depends on what kind of audit you do. A lot of the work we do is in the is in the performance auditing area and there are huge overlaps with evaluation, both in terms of scope – because we all address issues of results, efficiency and economy – and also in terms of methodology.

Ana

In this particular case, there were three pillars of the strategic plan that we were assessing, and results-based management (RBM) was one of those pillars. However, we could not assess that pillar on its own. That significantly influenced the way that we needed to scope the exercise and define questions for the assessment. Audit understood RBM much more aligned to monitoring and evaluation and reporting tools for compliance, while evaluation needed to assess the understanding of RBM in the organization as a strategy, not only to manage results but to promote learning to improve results and ensure institutional effectiveness. As a result, auditors wanted to focus on normative questions, while evaluators wanted to address descriptive and cause and effect questions.

If we put aside these challenges for a moment, one thing that both offices have in common is the pursuit of credible evidence. Was at least this something that brought you together and helped you to gel as a unified team?

Ana

To be quite honest, reaching a mutually agreed understanding of what constitutes credible evidence was in itself a challenge. This is not surprising though, given the different data collection methods and strategies for triangulation of the two offices. Here, the evaluators learned a lot from the auditors. Evaluators in our office usually favoured using surveys to close triangulation. However, having the surveys early in the process was extremely advantageous to help further limit the scope and to focus additional data collection efforts on what still needed clarity, allowing more time for the ‘why’, ‘what’ and ‘how’ questions. Nevertheless, the surveys could only be used to approach staff, and as evaluators we still needed to broaden the consultation and reach out to the field to engage with partners, governments and beneficiaries, which is something that auditors don’t do so often and broadly. That compromise was a challenge, but we arrived at a decision to go to the five regional hubs, bring staff from ten different countries from each region for an in-depth focus group, and then consult partners and beneficiaries in five countries where the focus groups took place. These interactions proved particularly relevant.

While you were busy grappling and coming to terms with so many challenges within the team, I wonder how people outside the team – such as the recipients of the exercise – felt about the initiative. How did UNDP staff look upon this effort, and what was the reaction of UNDP senior management?

Helge

It is correct to say that senior management was not super happy with this exercise. It’s also correct to say that in the beginning they actively pushed back on the whole thing. They felt in the beginning that we were ganging up against them and their response was that they needed to fight back. We had some interesting meetings with senior management along the way. They ignored us when we reported initially. However, I am almost certain that if you had asked them at the end of the process you would have received an honest “we were wrong and we see the benefits coming out of this exercise”. Maybe not all of them, but many of them saw that this was a good exercise for UNDP. It was a learning experience for senior management, a new thing for them.

The joint assessment caused significant fear and discomfort among managers, especially at the regional bureau and headquarters levels. Management first asked the exercise to be postponed very strongly. Then they lobbied to have the scope of the exercise reduced. The UNDP administrator refused to give an interview – she was running for the seat of UN secretary-general at the time and there were fears of the results of this assessment. The staff that we interviewed commented that they had been alerted to be careful of what was told to the assessment team as the report could come to affect the funding of the organization and therefore their jobs. Until the very last minute when the report was first discussed with the executive board, key management officials refused to attend the discussion of the report. So, when the report was formally presented to the executive board, in 2017, the management refused to present a management response to the recommendation.

After so much hard work, and underlying opposition from senior management, what was the final outcome?

Helge

It was a happy end. The response that they [senior management] gave us in addressing the recommendations, eventually, was very constructive. My point is that if we had not introduced it we would not have given them the opportunity to initially be a bit scared, but eventually see this as an opportunity for institutional learning.

Ana

Regardless of all that [opposition], the assessment was the richest in evidence to this day, highly acclaimed by the executive board and others, and the most read of all products we ever produced. Eventually, management produced a management response and implemented changes. The current administration re-stated its commitment to change and to implement the recommendations of the report. The report made a very strong statement for transparency, calling leadership to promote a results culture that encouraged critical reflection of success and failure as having value for organizational learning to improve effectiveness and stimulate innovation.

If I asked you to identify one defining learning element of this experience, what would be the single most important aspect that two offices took away from the joint assessment?

Ana

While there were challenges, the experience of conducting a joint assessment with elements from audit and evaluation was a unique learning opportunity for UNDP. It helped evaluators to look more closely at quantitative analysis and normative questions, and helped auditors to look at performance from the cause and effect angle to people and results. Audit and evaluations complementing each other can better feed the other side accountability and learning engines of an organization to ensure effectiveness.

Helge

There is a tendency in the audit community to rely very heavily on numbers and I think that is a great asset. What we learned is to use more effectively information that is coming through interviews and surveys. This synergistic approach created a product that was rock solid.

In the same vein, if I asked you to identify one defining key to success, what would be the one defining element that you think allowed the assessment to see the light of day?

Ana

The assessment was challenging at multiple stages, not least of which was the apprehension of UNDP management of the exercise. What made it possible was that both Directors from both offices, Indran and Helge, were convinced that the joint approach would produce a stronger report and generate much more relevant lessons for the organization.

Helge

It was a marriage. There were challenges along the way, but we were able to deal to them in a very constructive manner which meant that we were able to address all the concerns.

Ray

What is most important to me in this effort is that both groups learned: Indran’s group learned and Helge’s group learned. That’s a powerful statement to make. When we came together, it was a rough marriage for a while. But, in the end, they learned, and both are stronger for it now, having been through that.

Now that a couple of years have passed, what’s left of this tremendous effort? Has this experience affected the audit and evaluation communities, the UN system, and the two offices involved?

Ray

It was not an easy effort, but it was an important effort, one that has broken lots of new ground, bringing new perspectives in the UN system about the way that work can get done by different offices together. A dramatic effort at breaking down silos – which I think is one of the longest-term consequences that has come of this.

Ana

What followed the joint exercise was an avalanche of questions about the advantages and disadvantages of joint audit and evaluation exercises. There were not many conclusive answers, but we certainly need to continue this discussion. In fact, since this exercise, in 2015, both our offices try to conduct missions together, when possible, and try to exchange information to contribute to each other’s reports. A lot of the lessons we learned I am still using now as we work on other joint evaluations with other UN agencies in different types of exercises. This is useful not just for audit and evaluation, but also for any kind of joint exercise in the UN.

Any final thoughts or words of wisdom?

Ana

It is imperative that audit and evaluation work together in some ways to at least better coordinate workplans, ensure synergies and avoid duplication of efforts and wasting of resources. After all, audit and evaluation are part of a continuum. The autonomy of audit and evaluation functions is both historical in nature and institutional in form. It is clear that the nature of the responsibilities of international organizations like UNDP and IFAD are changing with the SDGs, and a recognition that the status quo of working in silos is no longer appropriate is obvious. The question, therefore, is not if but how to integrate: the marriage has to work.

Ray

What strikes me about the work that was done is that it was a tremendous effort at organizational learning that allowed both paradigms – audit and evaluation – to come away stronger and with new knowledge. This shows me an organization that is learning, that is vibrant. This is something that we often don’t see in many organizations, where the focus is on routine as opposed to creating new insights.

Thank you very much for your time.

You’re welcome.