• World Bank Group staff sometimes face incentives that constrain the scope and relevance of IE(s). This includes incentives to assess interventions that are “easier” to evaluate, limited understanding of the tool, or fear of negative results. • At the World Bank, in the context of the flat budget environment, limited project funding for IEs, and their high relative cost, most evaluations initiated in recent years have come to rely heavily on trust funds. Many IEs rely on multiple trust fund sources. Although access to multiple sources of financing eases the budget constraint, this fragmentation adds to staff transaction costs. Trust fund resources are not necessarily detrimental to IE relevance, but they are less flexible than Bank budget in terms of where and how to allocate resources, as donor preferences must also be considered.
The Quality of World Bank Group Impact Evaluations To distinguish IEs that produce reliable findings (medium and high quality) from those that require significant additional analytical work to be credible (low quality), IEG used a quality assessment framework that was developed from a well-established literature on IE and independently validated by an IE specialist. Four key aspects of quality were assessed: data, data measurement, evaluation design, and robustness of findings. The framework was applied to 166 completed IEs (140 Bank and 26 IFC IEs). At the World Bank, 94 percent of IEs completed in 2000–10 meet medium or high quality standards. This prevalence of IEs meeting quality standards is consistent with the perception of senior management at the World Bank. Overall, these medium and high-quality IEs assessed well-defined and relevant outcomes, used reliable data, and applied some checks for selection bias. More specifically, the majority of IEs used baseline data, conducted their own surveys (including collecting longitudinal information), and relied less on retrospective data. The outcome indicators measured by these IEs were well defined, separate enough from the inputs of the projects, and achievable within the evaluated time frame. Most IEs tested the validity of their methods and conducted some form of robustness check; however, the scope and rigor of these analyses varied, particularly among those employing quasi-experimental methods and those of medium quality. For instance, 80 percent of IEs using quasi-experimental methods tested at least one identification assumption associated with the evaluation method, but less than one-third of them checked all the key identification assumptions. The technical quality of completed World Bank IEs has improved over time. For instance, 51 percent of completed IEs that were initiated in 2000–04 met high quality standards and 90 percent met at least medium quality standards. This compares with 71 percent of completed World Bank IEs initiated in 2005 or later that met high quality standards and 98 percent that met at least medium quality standards.
xx
World Bank Group Impact Evaluations: Relevance and Effectiveness