Asian Development Bank Evaluation Profile - Evaluation Systems Review 2016

II. ASIAN DEVELOPMENT BANK

Asian Development Bank (ADB) Independent Evaluation Department (IED) Evaluation Mandate The Independent Evaluation Department (IED) of the Asian Development Bank (ADB) is mandated to evaluate sovereign and sovereign-guaranteed operations (public sector operations) and non-sovereign operations supported by the Bank; and the policies and strategies, practices, and procedures that govern them. The objective of evaluation in ADB is to assess development effectiveness and the long-term results of (i) ADB operations; (ii) country partnership strategies; and (iii) ADB policies, plans, practices, and procedures.

To respond to this objective, the IED is assigned the following functions: •

to assess issues of development effectiveness and the use of resources

•

to derive lessons and best practices to promote sustainability and the development effectiveness of ongoing and completed operations

•

to recommend appropriate measures for the design of future operations, country partnership strategies, and sector policy or strategy, as well as changes in ADB policies, practices, and procedures

•

to follow up on recommendations that are accepted by ADB.

Responsibility and scope of activities The first tier is self-evaluation of operations carried out within one or two years after the completion or other relevant stage of the project implementation. All ADB’s supported operations that are completed and have used ADB resources, need to be self-evaluated and the findings are made public on ADB’s website. This applies both to loan and grant supported operations and to ADB’s grant-based technical assistance activities. The responsibility for self-evaluation rests with operations departments and other support departments in ADB. The methodology for self-evaluations of operations is similar to that employed by IED for independent project evaluations. It follows OECD DAC guidelines on evaluations and good practice formulated by the group of multilateral development institutions in the Evaluation Cooperation Group (ECG). Every evaluation broadly assesses the operation’s relevance, effectiveness, efficiency, sustainability and arrives at an overall success rating.

The operation’s development impacts are separately rated, and also the performance of ADB and the borrower. Each evaluation identifies lessons and provides some recommendations and follow-up actions.

IED’s work falls into the second tier of ADB’s evaluation system. IED validates around 80% of the self-evaluated operations in terms of its success based on a random sample approach. It does not validate ADB’s short completion reports produced for its technical assistance activities, but it sometimes conducts special evaluation studies, and also conducts one to three cluster evaluations of technical assistance projects each year. IED produces 10-15 project performance evaluation reports a year for completed loan or grant based operations. These are independent evaluations of operations that are often purposely selected because of a special interest or because the results can feed into larger studies.

In addition, IED completes one to five project evaluations a year for ADB’s private sector operations.

EVALUATION SYSTEMS IN DEVELOPMENT CO-OPERATION: 2016 REVIEW © OECD 2016

81

II. ASIAN DEVELOPMENT BANK

Other responsibilities of IED are to: •

provide comments on newly proposed operations and selectively provide feedback on ongoing operations in terms of evaluation

•

assist in developing evaluation capacity in both operations departments in ADB and in developing member countries

•

disseminate the results and lessons of its evaluations.

Organisational Structure and Reporting Lines Central/main evaluation units Other units with evaluation functions

High level policy groups or ministries Reporting line

ADB Board of Directors

Development Effectiveness Committee (DEC)

ADB Management and Departments

Independent Evaluation Department (IED)

Lines of communication

IED reports to ADB’s Board of Directors through the Development Effectiveness Committee (DEC). The DEC consists of six members of the Board of Directors. The President appoints the members of the DEC, in consultation with the Board, and designates one of them as the Chairperson. The DEC provides the oversight function for ADB’s independent evaluation and plays a central role in the communication between the IED and the Board.

The DEC oversees the evaluation programme, reviews all IED reports, reports to the Board on important development effectiveness issues, and assists the Board in ensuring the achievement of the desired outcomes and the efficient use of resources. IED annually prepares and updates its rolling three-year evaluation work programme in consultation with both ADB Management, the DEC and the Board. The work programme is reviewed by the DEC and subsequently submitted to the Board for approval.

Types of Evaluation IED conducts not only project/programme-level evaluations, but also broad-based evaluations such as country assistance programme evaluations, validations of country partnership strategy final reports, impact evaluations, real-time evaluations, corporate evaluations, thematic/sector evaluations, and some topical papers on special subjects.

Some eight to ten of these broader reports are completed every year. IED produces one annual evaluation review each year, which is presented to ADB’s Board of Directors and discussed in a special meeting. IED operates in line with internationally accepted principles for the evaluation of development assistance, in particular the OECD DAC evaluation guiding principles and the good practice standards issued by the ECG.

82

II. ASIAN DEVELOPMENT BANK

Resources IED prepares an annual budget proposal based on the annual work programme. The DEC provides advice on the proposal and endorses it before it is submitted to the Board. For 2016, IED’s budget is USD 11.64 million, representing about 1.83% of the total ADB budget. In addition, IED applies for a technical assistance grant of around USD 2 million every year to finance some special evaluations.

Snapshot of evaluation resources ADB

DG / DDG / Advisor / Director / Lead

Professional evaluation staff

Fifty-one staff are engaged in the Administrative / Support staff implementation of evaluations in IED and based EUR 10 468 000 (USD 11 642 000) in headquarters, 21 of which are international 1.83% of the ADB budget staff, 11 are locally hired national officers and 14 staff providing administrative support. IED is led by a Director-General, a Deputy Director General, an Advisor, two Directors and a Lead Professional.

Principles of Evaluation Independence In the area of organisational independence, the Head of IED, the Director General (DG), is appointed by the Board upon recommendation of the DEC, in consultation with the ADB President. He/she is not permitted to work in other areas within ADB after completing the five-year, non-renewable term. During this period, the DG can only be removed by the Board on the grounds of inefficiency or misconduct. The DG is exempted from the formal annual performance review process; however, the Chair of DEC, in consultation with other DEC members, provides written annual feedback on his/her performance. The DG plays a major role in the selection of IED personnel but procedures are in accordance with ADB personnel selection and recruitment guidelines, as all IED staff except the DG are transferable to other units in ADB if they apply for positions.

To avoid conflict of interest, IED evaluators and management exclude themselves from evaluating or approving a report on any project/programme/activity/entity that they worked on, appraised, or had decision-making or approval responsibility for in a previous capacity, or when they expect to have such involvement in the future. Similar care is also taken in the selection of consultants.

Competence and capacity building Mandated with developing evaluation capacity, IED has provided regular training and workshops to stakeholders from ADB member countries on topics such as results-based monitoring and evaluation and evidence-based self-evaluation. Since 2009, IED has been undertaking internal workshop sessions within ADB and its country offices in preparing project and TA completion reports. In 2011, IED introduced a five-month on-the-job training programme in evaluation methods and approaches, with the aim of institutionalising results-based monitoring and evaluation. In 2014, IED started a two-year training programme for the executing agencies in order to strengthen their skills for evidence-based self-evaluation. A programme to support collaboration in project evaluations with staff of central evaluation units in ADB’s developing member countries was established in 2015.

EVALUATION SYSTEMS IN DEVELOPMENT CO-OPERATION: 2016 REVIEW © OECD 2016

83

II. ASIAN DEVELOPMENT BANK

Transparency All independent evaluation reports are disclosed on the ADB website in line with ADB’s Public Communications Policy. The IED also discloses its work programme, ADB management and IED responses to evaluation reports. IED also makes the validation reports for the self-evaluations available online. Project evaluations for ADB’s private sector operations are disclosed after removing commercially-sensitive information.

ADB’s Management responds to evaluation findings and recommendations. ADB Management’s responses and the DEC Chair’s Summaries of Discussions (if available) are included prominently in evaluation reports and also separately disclosed.

Knowledge management IED disseminates findings of its independent evaluation reports through the evaluation information system, called EVIS - a database of evaluation lessons - that provides users with an easily accessible source of searchable evaluation information and employs various parameters to refine a search. It is accessible publicly through the IED website. The website also promotes and disseminates outreach materials that emphasise learning from evaluations. It also produces evaluation derivate products such as: •

learning curves, which bring findings and recommendations of independent evaluations to a broader range of readers

•

learning lessons, which provide key lessons drawn from evaluations and may include contexts derived from literature review

•

case studies, designed to expose specialised material from evaluations

multimedia content such as photo essays, audio and video podcasts, all targeted to different types of audience.

The IED website is the main information and media platform but IED also maintains social media platforms for communicating and disseminating evaluation information for greater outreach. IED also maintains an enquiry desk, which responds to internal and external queries about evaluation products and services.

•

To monitor the progress of the implementation of recommendations defined by in the evaluation reports, the ADB has a management action record system. The evaluators enter recommendations produced by the report, and those accepted by the Management are tracked with the associated action plans. The DEC monitors the actions taken in response to recommendations and records the progress at least twice a year. This system is not public but new and completed actions are reported and reviewed in IED’s Annual Evaluation Review.

Co-ordination with donors IED co-ordinates evaluation practices and activities with other multilateral development banks through the ECG and its working groups. The ECG’s mandate focuses on evaluation principles, standards, and good practices. IED also participates in the evaluation activities of multilateral and bilateral agencies through the OECD DAC Evaluation Network and the United Nations Evaluation Group.

Note to reader: The section at the beginning of Part II entitled “Introduction and key for the member profiles” provides explanatory notes on the profiles.

84