collection. This suggests that these IEs were planned as a formal part of the project design and its M&E framework.5 In contrast to completed IEs, for 49 percent of ongoing World Bank IEs whose project documents were reviewed, the appraisal document made some mention of plans for both an IE and baseline data collection.6 The statistically significant difference in the proportion of completed and ongoing IEs that were referenced in the project design documents (in terms of indications to conduct an IE and collect baseline data) is robust to alternative measures and suggestive of the increase in prospective IEs at the World Bank.7 This finding is also consistent with the reported push for such IEs by initiatives like DIME and SIEF. At the World Bank, recent IEs are also more likely to be used as an integral part of project M&E, partly as a result of the increase in prospective IEs. Based on survey results, IEs initiated in 2007–10 are more often reported to be an integral part of project M&E (49 percent) than projects initiated in preceding years (29 percent). This difference is statistically significant and robust to alternative measures and sources of data.8 Many questions that go beyond average short-term outcomes can add value to the operational relevance of IE, but have often received less attention. Answers to questions regarding long-term effects, the distribution of impacts, the differential effects of separate program designs, the channels of transmission, the external validity of the findings, and the efficiency of programs are also of interest to project administrators and policy makers and are likely to increase the use of IEs. Not only do they contextualize the findings on average impacts, but they also reveal if these impacts contribute to long-term objectives and are generalizable, what parts of the programs matter the most, which form of treatment is more beneficial, if the treatment is worth the cost, and how to better target the beneficiaries. • The majority of completed World Bank IEs assessed distribution of program impacts, but it was less prevalent in completed IFC IEs. Seventy-nine percent of completed World Bank IEs assessed the distribution of program impacts across different groups of beneficiaries based on such characteristics as age, gender, income, location, and duration of participation. In contrast, 35 percent of completed IFC IEs reported disaggregated program impacts. Understanding how program impacts vary for different beneficiary subgroups is relevant to operational decision making, as it can help policy makers decide if they need to expand or limit the treatment to certain groups in the population or pursue other alternatives to achieve intended outcomes. • A low number of completed World Bank Group IEs evaluated longerterm outcomes. Eleven percent of completed World Bank and 20 percent of completed IFC IEs made an effort to evaluate any medium or long-term outcomes, which suggests that in the World Bank Group, IEs measuring outcomes that develop more quickly are favored over IEs that assess longlasting effects, which are closer to the development objectives of programs.9 Evaluating longer-term outcomes can also help policy makers understand
Relevance of World Bank Group Impact Evaluations
47