Management believes that there are tractable, concrete ways that IEG could use to specify a proxy counterfactual. For example, in looking at the impact of IEs on project design and policy, the report could have compared it to the incorporation of evidence-based information in general in project design. The IEG report could compare, for instance, whether use or mention of any evidence that is based on actual data increases in Project Assessment Documents or Implementation Status and Results Reports when there is an IE in the sector or the project. Evidence from such an analysis would put the report’s findings about whether IEs are perceived as affecting projects in an appropriate context. A second way a counterfactual could have been established, particularly in the country case studies, is to look at policies that have been implemented and/or scaled up and assess whether IE had any role in either the implementation or the scale-up. Again, this would provide a useful counterfactual with which to assess the specific role of IEs, perhaps relative to other evidence. A third counterfactual could be established by looking at whether IEs complement or substitute for other forms of knowledge production. By looking at the overall production of knowledge (analytic and advisory activities and published papers from all sources) and seeing whether over time we are seeing an increase in the knowledge that is useful for policy would add considerably to the report. While all these ways to specify a proxy counterfactual have drawbacks, tackling the counterfactual question would appear to be more important than answering it perfectly “cleanly.” As long as the samples and results were clearly explained and the evaluation did not draw unwarranted conclusions, readers could judge for themselves.
Recommendations Bank management is in basic agreement with all of IEG’s recommendations for the Bank. With regard to a strategic approach to IE, as part of its IDA16 commitments, management has developed a strategic selection framework to increase the learning from IE and improve institutional accountability. Financing is a concern, and Bank management is exploring options, including the consolidation of external funding. However, management notes that the bulk of the costs are in country, particularly those associated with data collection, and countries themselves will fund part of these costs, often through project financing. With regard to the integration of IE into project design, many of the elements of this recommendation are relevant to all knowledge products and are major elements of the overall knowledge agenda under implementation. With regard to quality standards and enhancing the analytical elements of IE, management agrees, while noting that responsibility for quality control rests with the Regions. Current work on quality enhancement will address these issues. More detailed responses are set out below.
II. IFC Management Comments Management appreciates this report. While IFC is widely recognized to be in the forefront in development results measurement and reporting among private sector–oriented international financial institutions, we continue to seek
Management Response
xxix