Table 17. Case selection process

Selection of case studies

Table 17 presents how cases were selected for this evaluation.

Table 17. Case selection process

Steps Cases

1 41 assessments performed during the 2015-21 period on 36 MOs 2 Assessments for which we have claims for use by MOPAN members, staff or consultants; 2)and/or that have been assessed twice102 or should be assessed for a second time in 2021;103 3) allow some time for the uses to unfold; 4) allow for a balance among the different types of organisations assessed (UN organisations, MDBs, humanitarian organisations and normative organisations). Considering these criteria, we suggested in the inception note picking the 10 cases from among the following: AfDB, FAO, GFATM, ILO, IOM, OCHA, UNAIDS, UNDP, UNESCO, UN HABITAT, UNICEF, UNRWA and UN WOMEN.

3 The priority among these 13 potential case study organisations was having a “paper trail” of use. An in-depth examination of publicly available documents was conducted to identify cases in which MOPAN assessments were mentioned in reports and other official documents. A substantial paper trail was identified for IOM, ILO, FAO, UNESCO, UN WOMEN and UNWRA, with claimed/observed uses that fit several CCs, and to a lesser extent for UNICEF, UNDP, UN HABITAT.

4 MOs were asked whether they would agree to be part of this evaluation. One declined (AfDB) and none agreed: FAO, ILO, IOM, OCHA, UN WOMEN, UNDP UNESCO, UNICEF, UNRWA. GFATM was also solicited and agreed to participate. For SEAH and UNDS, no agreement was needed.

The final sample includes no IFIs for want of a paper trail suggesting that any made use of MOPAN assessments, except for IFAD.

Robustness of the case studies’ findings Robustness of findings is based on several layers of evidence:

Claims are collected through different channels (interviews with staff and member representatives, surveys of ILs, members and MO focal points). Each claim is about what has changed (or will) change, MOPAN’s role in these changes and the role of other factors in some cases.

Documentary evidence is then collected systematically to confirm whether the claimed changes occurred (e.g. the evaluation function was reformed) and whether there is some evidence that MOPAN contributed to them. This layer of evidence is even more necessary for seeing whether MOs have “adapted” to members’ requests. All our interviewees claimed (without providing any empirical evidence) that “MOPAN is useful” and that they were engaged in changes (mirroring the usual members’ demand for reform). Systematic tests of the claims include a chronology of changes (were they observed after the MOPAN report was made available?); alignment with “recommendations” (do they correspond to the report’s areas for improvement?); detailed management response with

102  5 organisations have been assessed twice: ILO, OCHA, UNDP, UNEP, UNICEF 103  6 MOs have been assessed once and should be assessed in 2021: AfDB, GFATM, IDB, IFAD, UNAIDS, WBG.