Box 5.4
Influence of Impact Evaluations on Education Sector Strategy
Strategic priorities reflect a consensus built up from an earlier set of issues and one that is being constantly modified and challenged based on emerging findings from IEs and other forms of research. By the very nature of strategic priorities, which are based on institutional consensus that takes longer to form, the research and IE agenda will lead to filtering the findings into a broader formulation of priorities. The strategic priority of learning as an objective in the Education Sector Strategy of 2010–20 has been the result of the production of considerable evidence on what children are learning in school and several researchers highlighting that enrollment did not imply learning. It took almost a decade from the first papers on learning to the current strategic focus on learning to be articulated as a goal in the priorities of the World Bank. By then, a large body of evaluations had already built up (in addition to the 44 supported by the World Bank, there are close to 100 from outside the Bank) showing that a host of different interventions was producing small effects on learning (much smaller than what would be required) and these effects faded quite quickly. Meanwhile, researchers were documenting that enrollment itself had surprising effects on wages, fertility, and a host of other outcomes that were difficult to look at in short-term studies. Although the consensus is still being developed, there is increasing evidence that schooling produces non-cognitive skills that are important for later life outcomes and that keeping children in school longer helps. Research has moved in two parallel tracks: a recent book summarizes precisely the impact of programs looking at information, local accountability, pay for performance, and contracting (Bruns and others 2011), and another track has started looking at enrollment, long-term outcomes and non-cognitive skills. Source: IEG, based on discussions with World Bank management.
Bank country strategies that made a reference to IEs mostly correspond to countries in Latin America and the Caribbean and Africa Regions, where the IE agenda is active. For instance, Brazil is the one of the countries with the highest number of IEs; in fact, around one-fourth of all investment operations approved in 2009–10 in Brazil were subject to an IE. More recently, this IE effort has been highlighted and identified as one of the essential pillars of the Country Partnership Strategy for Brazil (FY12–FY15). The contribution of IEs to Brazil’s knowledge agenda, operational work, and policy dialogue is also reflected in the Country Management Unit’s decision to create a multisector group of regional IE experts, responsible for ensuring adequate technical quality of the activities, providing guidance to task team leaders, and supporting the operational and knowledge dialogue on the topic.
Influence of Impact Evaluations on Evaluation Capacity and Culture The direct contribution of conducting an IE to building local capacity has been limited because of the modest engagement of local researchers and government staff in the analytical stage of the evaluations. Thirty-two percent of completed World Bank IEs were considered by the surveyed team leaders and evaluators
102
World Bank Group Impact Evaluations: Relevance and Effectiveness